The think-aloud controversy in second language research

185 Pages·2010·1.19 MB·English

by  Melissa A. Bowles

The Think-Aloud Controversy
in Second Language Research
The Think-Aloud Controversy in Second Language Research aims
to answer key questions about the validity and uses of think-alouds,
which are verbal reports completed by research participants while
they perform a task. It offers an overview of how think-alouds have
been used in language research and presents a quantitative meta-
analysis of findings from studies involving verbal tasks and think-
alouds. The book begins by presenting the theoretical background
and empirical research that has examined the reactivity of think-
alouds, then offers guidance regarding the practical issues of data
collection and analysis, and concludes by providing implications for
the use of think-alouds in language research. With its focus on a
much-discussed and somewhat controversial data elicitation method
in language research, this timely work is relevant to students and
researchers from all theoretical perspectives who collect first or sec-
ond language data. It serves as a valuable guide for any language
researcher who is considering using think-alouds.
Melissa A. Bowles is Assistant Professor of Spanish, Linguistics, and
SLATE (Second Language Acquisition and Teacher Education) at
the University of Illinois at Urbana-Champaign. Her main research
interests are instructed second/heritage language acquisition and
research methods.
Second Language Acquisition Research Series:
Theoretical and Methodological Issues
Susan M. Gass and Alison Mackey, Editors
Monographs on Theoretical Issues:
Schachter/Gass Second Language Classroom Research: Issues and
Opportunities (1996)
Birdsong Second Language Acquisition and the Critical Period
Hypotheses (1999)
Ohta Second Language Acquisition Processes in the Classroom:
Learning Japanese (2001)
Major Foreign Accent: Ontogeny and Phylogeny of Second
Language Phonology (2001)
VanPatten Processing Instruction: Theory, Research, and
Commentary (2003)
VanPatten/Williams/Rott/Overstreet Form-Meaning Connections
in Second Language Acquisition (2004)
Bardovi-Harlig/Hartford Interlanguage Pragmatics: Exploring
Institutional Talk (2005)
Dörnyei The Psychology of the Language Learner: Individual
Differences in Second Language Acquisition (2005)
Long Problems in SLA (2007)
VanPatten/Williams Theories in Second Language Acquisition
(2007)
Ortega/Byrnes The Longitudinal Study of Advanced L2 Capacities
(2008)
Liceras/Zobl/Goodluck The Role of Formal Features in Second
Language Acquisition (2008)
Monographs on Research Methodology:
Tarone/Gass/Cohen Research Methodology in Second Language
Acquisition (1994)
Yule Referential Communication Tasks (1997)
Gass/Mackey Stimulated Recall Methodology in Second Language
Research (2000)
Markee Conversation Analysis (2000)
Gass/Mackey Data Elicitation for Second and Foreign Language
Research (2007)
Duff Case Study Research in Applied Linguistics (2007)
McDonough/Trofimovich Using Priming Methods in Second
Language Research (2008)
Larson-Hall A Guide to Doing Statistics in Second Language
Research Using SPSS (2009)
Dörnyei/Taguchi Questionnaires in Second Language Research:
Construction, Administration, and Processing, Second Edition
(2009)
Bowles The Think-Aloud Controversy in Second Language
Research (2010)
Of Related Interest:
Gass Input, Interaction, and the Second Language Learner (1997)
Gass/Sorace/Selinker Second Language Learning Data Analysis,
Second Edition (1998)
Mackey/Gass Second Language Research: Methodology and
Design (2005)
Gass/Selinker Second Language Acquisition: An Introductory
Course, Third Edition (2008)

The Think-Aloud
Controversy in Second
Language Research
Melissa A. Bowles
First published 2010
by Routledge
270 Madison Avenue, New York, NY 10016
Simultaneously published in the UK
by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an
informa business
This edition published in the Taylor & Francis e-Library, 2010.
To purchase your own copy of this or any of Taylor & Francis or Routledge’s
collection of thousands of eBooks please go to www.eBookstore.tandf.co.uk.
© 2010 Taylor & Francis
All rights reserved. No part of this book may be reprinted or
reproduced or utilized in any form or by any electronic,
mechanical, or other means, now known or hereafter
invented, including photocopying and recording, or in any
information storage or retrieval system, without permission in
writing from the publishers.
Trademark Notice: Product or corporate names may be
trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Bowles, Melissa A.
The think-aloud controversy in second language research /
Melissa A. Bowles.
p. cm. — (Second language acquisition research series: theoretical and method-
ological issues)
Includes bibliographical references.
[etc.]
1. Second language acquisition—Research. 2. Language and languages—
Research. 3. Thought and thinking—Study and teaching.
4. English language—Study and teaching—Foreign speakers. I. Title.
P118.2.B69 2010
418.007'2—dc22
2009045982
ISBN 0-203-85633-3 Master e-book ISBN
ISBN13: 978–0–415–99483–5 (hbk)
ISBN13: 978–0–415–99484–2 (pbk)
ISBN13: 978–0–203–85633–8 (ebk)
Contents
Acknowledgments xi
Introduction 1
1 The Use of Verbal Reports in Language Research 5
History of Data Collection with Verbal Reports 5
In Non-SLA Fields 5
In L1 and L2 Research 6
L1 Research 6
Reading 6
Writing 6
Language Testing 7
L2 Research 7
Reading 8
Writing 9
Translation 9
Interlanguage Pragmatics 9
Conversational Interaction Research 10
Attention and Awareness 11
Implicit and Explicit L2 Knowledge 12
2 Controversy Over the Use of Think-Alouds: Reactivity 13
Classifying Verbal Reports: Ericsson and Simon’s Model 13
Challenges to the Validity of Verbal Reports:
Veridicality and Reactivity 13
Main Predictions of Ericsson and Simon’s Model 14
Responding to Challenges: Investigations of Validity
in Cognitive Psychology 15
Studies Comparing Non-Metacognitive Protocols
and Silent Controls 15
Reactivity for Latency but not Accuracy 16
viii Contents
Reactivity for Both Accuracy and Latency 19
Reactivity for Neither Accuracy Nor Latency 21
Studies Investigating the Reactivity of
Metacognitive Protocols 24
Studies Comparing Metacognitive Protocols
and Silent Controls 25
Reactivity for Both Accuracy and Latency 25
Improved Task Performance 25
Hindered Task Performance 29
Reactivity for Accuracy but not for Latency 29
Improved Task Performace 29
Hindered Task Performance 36
Outliers 37
Reactivity for Latency but not for Accuracy 39
Reactivity for Neither Latency nor Accuracy 43
Studies Comparing Metacognitive and Non-
Metacognitive Protocols 46
Reactivity for Accuracy 46
Improved Task Performance 46
Hindered Task Performance 50
Non-Reactivity for Accuracy 52
Mixed Results for Reactivity 54
Conclusions 56
Synthesis: What Makes a Task Amenable to
Thinking Aloud 59
Same Task, Same Results 59
Same Task, Different Results 61
Similar Tasks, Different Results 62
Different Tasks, Different Results 63
Summary 64
3 Features that Make a Task Amenable to Think-Aloud:
A Meta-Analysis of Studies Investigating the Validity
of Think-Alouds on Verbal Tasks 67
Reactivity Studies in SLA 67
Leow and Morgan-Short (2004) 67
Bowles and Leow (2005) 67
Sachs and Polio (2007) 68
Sachs and Suh (2007) 69
Rossomondo (2007) 70
Bowles (2008) 70
Yoshida (2008) 71
Sanz et al. (2009) 72
Contents ix
Polio and Wang (in review) 73
Synthesizing Research on Reactivity 76
Meta-Analysis 78
Identification of Studies 79
Selection Criteria 80
Inclusion Criteria 80
Exclusion Criteria 81
Coding 81
Calculation of Effect Sizes 82
Combining Effect Sizes 83
Analysis of the Homogeneity of Effect Sizes 84
Results 86
Publication Characteristics 86
Learner Characteristics 87
Effects of Type of Report on Post-Test Performance 90
Effect of Type of Report on Text
Comprehension 93
Effect of Type of Report on Receptive
Form Learning 94
Effect of Type of Report on Productive
Form Learning 95
Effect of Type of Report on Latency 96
Effects of Language of Report on Post-Test
Performance 96
Effects of Language of Report on Text
Comprehension 96
Effects of Language of Report on Receptive
Form Learning 97
Effects of Language of Report on Productive
Form Learning 98
Effects of Language of Report on Latency 98
Effects of Language of Task on Post-Test
Performance 99
Effects of Language of Task on Text
Comprehension 99
Effects of Language of Task on Receptive
Form Learning 99
Effects of Language of Task on Productive
Form Learning 100
Effects of Language of Task on Latency 101
Effects of Type of Task on Post-Test Performance 102
Effects of Type of Task on Text
Comprehension 102