Cognitive task analysis: Think alouds & Difficulty Factors Assessment

14 Pages·2010·2.35 MB·English

by  Ken Koedinger

Unpacking & repacking
Cognitive Task Analysis: expertise: Chick sexing
Think Alouds and Difficulty
• Experts don’t know,
what they know
Factors Assessment
– 98% accurate after
years of on-the-job
training
Ken Koedinger • Interviews led to design of
“pictures in which critical
HCI & Psychology features of various types
were indicated”
CMU Director of Pittsburgh
• After just minutes of
instruction, novices
Science of Learning Center
brought to 84% accuracy!
Biederman & Shiffrar (1987). Sexing Day-Old Chicks: A Case Study and Expert Systems
Analysis of a Difficult Perceptual-Learning Task. JEP: Learning, Memory, & Cognition.
7/12/10 Pittsburgh Science of Learning Center 1 7/12/10 Pittsburgh Science of Learning Center 2
Overview Cognitive Task Analysis
• Cognitive Task Analysis • Techniques to specify cognitive
–What is it? Why do it? structures & processes associated
with task performance
• CTA methods
–Structured interviews of experts
–Difficulty Factors Assessment
–Think alouds of experts & novices
–Think Aloud
performing tasks
• Hands-on exercise
–Computer simulations of human
reasoning
7/12/10 Pittsburgh Science of Learning Center 3 7/12/10 Pittsburgh Science of Learning Center 4
Cognitive Task Analysis Isn’t knowledge analysis
Improves Instruction done for long-standing
Studies: Traditional instruction vs. CTA-based academic domains?
• Med school catheter insertion (Velmahos et al., 2004)
– Sig greater pre to post gain • Hasn't all this been worked out?
– Better with patients on all four measures used
– Example: Sig fewer needle insertion attempts!
• Other examples • Surely by now we understand the
– Radar system troubleshooting (Schaafstal et al., 2000) content of, say, Algebra?
– Spreadsheet use (Merrill, 2002)
• Meta-analysis, 7 studies: 1.7 effect size!
(Lee, 2004)
7/12/10 Pittsburgh Science of Learning Center 5 7/12/10 Pittsburgh Science of Learning Center 6
Need for a Knowledge
Overview
Decomposition Methodology
• Cognitive Task Analysis
• Good instruction targets the edge of students'
knowledge, what is "just-learnable"
–What is it? Why do it?
• Need a method for decomposing a topic into
• CTA methods knowledge components
– What components are learners’ missing?
–Difficulty Factors Assessment – What order do they acquire these components?
– Which components are particularly hard to acquire?
–Think Aloud
– What “hidden skills” must be acquired?
• Hands-on exercise
• Knowledge decomposition guides design of:
– problem solving activities, tutor interface, cognitive
model, hints and bug messages, problem sequence
7/12/10 Pittsburgh Science of Learning Center 7 7/12/10 Pittsburgh Science of Learning Center 8
Knowledge Decomposition through
Which problem is hardest for
Difficulty Factors Assessment (DFA)
beginning algebra students?
• Goal: Identify what is "just learnable" for
students at different levels of competence Story Problem
• The DFA methodology: As a waiter, Ted gets $6 per hour. One night he
1. Identify possible problem difficulty factors made $66 in tips and earned a total of $81.90.
- Use think aloud or analytic task analysis How many hours did Ted work?
2. Create test items & forms; Administer
Word Problem
3. Analyze results:
a. Main effects and interactions
Starting with some number, if I multiply it by 6 and
b. Strategies and errors
then add 66, I get 81.90. What number did I start
4. Create a cognitive model
with?
5. Create a “developmental model” or “learning
progressions” Equation
x * 6 + 66 = 81.90
7/12/10 Pittsburgh Science of Learning Center 9
Algebra Student Results: Part of a larger “Difficulty
Story Problems are Easier! Factors Assessment”
• Difficulty factors involved
80%
70%
–Presentation type
ct 61%
re 60% •Story, Word, vs. Equation
r
o 42% –Unknown position
C
40%
nt •Result-unknown vs. start-unknown
e
c –Number type
r 20%
e
P •Whole vs. decimal numbers
0% • Multiple quiz forms
Story Word Equation
• Detailed strategy & error analysis
Problem Representation
Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on
quantitative reasoning. In International Journal of the Learning Sciences.
7/12/10 Pittsburgh Science of Learning Center 12
More Common:
Formal, Translate & Solve Strategy
Informal Strategies
7/12/10 Pittsburgh Science of Learning Center 13
Algebra equations Expert Blind Spot
are like a foreign Algebra teachers worst at recognizing
algebra student difficulties
language -- takes
extensive 100
90
experience to
80
acquire % making 70
correct
60
ranking
(equations 50
hardest) 40
30
20
10
0
Elementary Middle High School
Teachers School Teachers
Teachers
Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early
algebra development. Journal of Mathematics Education Research, 31 (2), 168-190.
7/12/10 Pittsburgh Science of Learning Center 16
What’s behind expert blind
Using Cognitive Task Analysis to
spot? design better algebra instruction
• Self-reflection on current cognition, biased
memory of past learning • Inductive support strategy
• Aware of verbally-mediated reasoning – Help students generalize abstract math
– False inference: More words => more thinking from their own intuitive concrete solutions
• Not aware of implicit processing & learning – Similar to “progressive formalization” or
– Fluent algebra language processing requires “concreteness-fading” (Golstone & Son, 05)
extensive implicit learning
– Our minds are continually engaged in pattern • Test idea with an in vivo experiment
induction, analogy, chunking, strengthening …
7/12/10 Pittsburgh Science of Learning Center 17 7/12/10 Pittsburgh Science of Learning Center 18
Parametric Study: Textbook Parametric Study: Textbook
vs. Cognitively-Based Design vs. Cognitively-Based Design
5 5
Learning Due to Tutor Variants Learning Due to Tutor Variants
4 4
3 3
Pre to Post Pre to Post
Improvement Improvement
Score 2 Score 2
1 1
0 0
Textbook Inductive Support Textbook Inductive Support
(Symbolize first) (Solve & then symbolize) (Symbolize first) (Solve & then symbolize)
…Plumbing Co. charges $42 per 1. 35 + 42h = d 2. 35 + 42*3 = 161 …Plumbing Co. charges $42 per 1. 35 + 42h = d 2. 35 + 42*3 = 161
hour plus $35 for the service call 3. 35 + 42*4.5 = 224 hour plus $35 for the service call 3. 35 + 42*4.5 = 224
1. …. write an expression … 2. 35 + 42*3 = 161 1. …. write an expression … 2. 35 + 42*3 = 161
2. How much for a 3 hour call? 3. 35 + 42*4.5 = 224 1. 35 + 42h = d 2. How much for a 3 hour call? 3. 35 + 42*4.5 = 224 1. 35 + 42h = d
3. What will bill be for 4.5 hours? 3. What will bill be for 4.5 hours?
4. Find hours when bill is $140 4. 35 + 42h = 140 4. 35 + 42h = 140 4. Find hours when bill is $140 4. 35 + 42h = 140 4. 35 + 42h = 140
Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution
of a cognitive tutor for algebra symbolization. Interactive Learning Environments. of a cognitive tutor for algebra symbolization. Interactive Learning Environments.
7/12/10 Pittsburgh Science of Learning Center 19 7/12/10 Pittsburgh Science of Learning Center 20
Overview What is a
Think-Aloud Study?
• Cognitive Task Analysis
Basically, ask a users to “think aloud” as
–What is it? Why do it?
they work...
• CTA methods
...on a task you want to study
–Difficulty Factors Assessment
...while you observe & audio or videotape
–Think Aloud ...either in context (school) or in lab
...possibly using paper/storyboard/interface
• Hands-on exercise
you are interested in improving
7/12/10 Pittsburgh Science of Learning Center 21 7/12/10 Pittsburgh Science of Learning Center 22
The Roots of Think-Aloud The Cognitive Psychology Theory behind
Think-Aloud Protocols
Protocols
• People can easily verbalize the linguistic
• Allen Newell and Herb Simon created
contents of Working Memory (WM)
the technique in 1970s
• People cannot directly verbalize:
– Applied in ‘72 book: “Human Problem – The processes performed on the contents of WM
Solving” • Procedural knowledge, which drives what we do, is outside our
conscious awareness, it is “tacit”, “implicit” knowledge.
• People articulate better external states & some internal goals,
• Anders Ericsson & Herb Simon’s book not good at articulating operations & reasons for choice
– Non-linguistic contents of WM, like visual images
– “Protocol Analysis: Verbal Reports as Data”
• People can attempt to verbalize procedural or
1984, 1993
non-linguistic knowledge, however, doing so:
– Explained & validated technique
– May alter the thinking process (for better or worse)
– May interfere with the task at hand, slowing
performance
7/12/10 Pittsburgh Science of Learning Center 23 7/12/10 Pittsburgh Science of Learning Center 24
Example: Think Alouds in
How to Collect Data in a
Statistics Tutor Development
Think-Aloud Study
(Gomoll, 1990, is a good guide)
• Task: Exploratory Data Analysis
1. Set up observation 5. Explain that you will not – Given problem description and data set
provide help
– write tasks – Inspect data to generate summaries & conclusions
6. Describe tasks
– recruit students – Evaluate the level of support for conclusions
7. Ask for questions before
2. Describe general purpose
you start; then begin • Example Problem
of observation observation
In men’s golf, professional players compete in either the regular
3. Tell student that it’s OK – say “please keep tour (if they’re under 51 years old) or in the senior tour (if they
to quit at any time talking” if the
are 51 or older). Your friend wants to know if there is a
4. Explain how to “think participant falls silent difference in the amount of prize money won by the players in
aloud” for 5 seconds or more the 2 tours. This friend has recorded the prize money of the top
– give a demonstration – be sensitive to a severe 30 players in each tour. The variable money contains the money
– give an unrelated desire to quit won by each of the players last year. The variable tour indicates
practice task, e.g., 8. Conclude the observation which tour the player competed in, 1=regular, 2=senior. The
add digits variable rank indicates player rank, 1=top in the tour.
Thanks to Marsha Lovett!
7/12/10 Pittsburgh Science of Learning Center 25 7/12/10 Pittsburgh Science of Learning Center 26
Rational Task 1) Translate question into Sample Transcript
statistical terms
Analysis of Major
L# Participants words & actions Annotation
2) Identify relevant
Goals variables 60% 1 Oh, okay.
2 So we need to, he wants to know whether there is a Goal 1
3 difference in the amount of prize money, the amount of
• Inspired by ACT-R
3) Characterize problem 0% 4 money won by players in the two tours.
theory (variable types, etc.) 5 So, I think this is the prize money, uh, money contains the Goal 2
6 prize money won by each of these players.
• Break down task:
4) Select appropriate 100% 7 Tour indicates which tour the player competes in.
– 7 major goals analysis 8 Well, you don't really need rank, in order to solve this, right?
9 Cause like, well, I don't know.
– Each goal has involves
multiple steps or 5) Conduct analysis 100% 10 Um... I'm gonna do a boxplot... ... Goal 4
11 [Subject uses statistics package to make a boxplot] Goal 5
subgoals to perform
12 oh, cool (laugh)- I did it.
– Key productions react
6) Interpret results of 100% 13 All right, uh, so just looking at the average. Goal 6
to major goals & set analysis 14 It looks like the people in the senior tour get less money.
subgoals 15 Um, and there's a lot less variation in the amount of money
7) Evaluate evidence with 60% 16 that, like all the prizes.
respect to question 17 A couple little outliers in each which means like, I don't
18 know, like some people won, like a lot of money at a time…
Key observations about this
Comparing Think 1) Translate question into 20%
statistical terms
verbal report
Aloud Results with
2) Identify relevant
Rational Task Analysis variables 60%
• No evidence for goal 3 “characterize the
problem” • Percentages to the right 3) Characterize problem 0%
(variable types, etc.)
of each step represent
– Line 10: student simply jumps to selecting the percentage of
a data representation (goal 4) without students in the think- 4) Select appropriate 100%
analysis
thinking about why. aloud study who showed
explicit evidence of
• No evidence for goal 7 “evaluate engaging in that step. 5) Conduct analysis 100%
evidence” • Step 3 is totally absent!
– A tutor can help students 6) Interpret results of 100%
to do & remember to do analysis
step 3
7) Evaluate evidence with 60%
respect to question
7/12/10 Pittsburgh Science of Learning Center 29
Inspiration for Production Rules
(Knowledge Components)
Statistics Tutor
• Missing production (to set goal 3):
Design: Explicitly
Characterize problem
If goal is to do an exploratory data analysis prompts students to
& relevant variables have been identified
engage in critical
then
set a subgoal to identify variable types
subgoals
• Buggy production (skipping from goal 2 to 4):
Select any data representation
If goal is to do an exploratory data analysis
& relevant variables have been identified
then
set a subgoal to conduct an analysis by picking any data
representation
7/12/10 Pittsburgh Science of Learning Center 31 7/12/10 Pittsburgh Science of Learning Center 32
Overview
• Cognitive Task Analysis
Students are prompted to –What is it? Why do it?
complete critical subgoal
• CTA methods
#3 from CTA
–Difficulty Factors Assessment
–Think Aloud
• Hands-on exercise
7/12/10 Pittsburgh Science of Learning Center 33 7/12/10 Pittsburgh Science of Learning Center 34
Cognitive Task Analysis
Try this ...
Exercise
• One person think aloud while solving
• Use Think Aloud to design a Difficulty this problem. You can use paper.
Other person is experimenter.
Factors Assessment
Experimenter: Remember to say “keep
• Find someone next to you to work with talking” whenever participant is silent
– I will give two problems
– Take turns giving a think-aloud solving • Ready ...
these next two problems
• What is 5 ÷ 3/4 = ?
7/12/10 Pittsburgh Science of Learning Center 35 7/12/10 Pittsburgh Science of Learning Center 36
Think about student thinking ...
Now this ...
• Switch roles: • Which will be easier?
– Other person think aloud
– What’s written on paper is part of TA • Why?
– Did the experimenter say “keep talking”?
• Strategy & error analysis:
• Ready …
– What strategies will students use?
– Will there be differences in strategy selection
• If 5 yards of ribbon are cut into pieces that are between problem types?
each 3/4 yard long to make bows, how many – What errors might account for observed differences?
bows can be made?
7/12/10 Pittsburgh Science of Learning Center 37 7/12/10 Pittsburgh Science of Learning Center 38
How could you design a DFA
“Latin Square” Design
to test your hypotheses?
5 ÷ 3/4 = ? 7 ÷ 2/3 = ?
• Can you put these two problems on
the same quiz form?
No Context Form 1 Form 2
– Why not? What can you do instead?
• What other factors might be involved? Context Form 2 Form 1
– Size of the numbers--big nums discourage
informal strategy
• Don’t give problems with same answer on same form
– “Tempting” nums like 6 ÷ 3/5
• Can give problems with both values of a difficulty factor
– Order: context first vs. context second • Example above
– Students using either Form 1 or Form 2 will get both a No-
Context & a Context problem
– But, two forms “counter balance” the number types
7/12/10 Pittsburgh Science of Learning Center 39 7/12/10 Pittsburgh Science of Learning Center 40