Utilizing Think-Aloud Protocols to Assess the Usability of a Test for Ethical Sensitivity in Construction

16 Pages·2017·0.46 MB·English

PaperID#9827
Utilizing Think-Aloud Protocols to Assess the Usability of a Test for Ethical
Sensitivity in Construction
Mr. KennethStaffordSandsII,VirginiaTech
KennethS.SandsIIisadoctoralcandidateandgraduateassistantinEnvironmentalDesignandPlanning
atVirginiaTech. Hisresearchfocusisonprofessionalethicsanditspedagogy.
Dr. DeniseRutledgeSimmons,VirginiaTech
Denise R. Simmons, Ph.D., is an assistant professor in the Myers-Lawson School of Construction &
CivilandEnvironmentalEngineeringatVirginiaPolytechnicInstituteandStateUniversity. Sheholdsa
B.S., M.S., andPh.D.incivilengineeringandagraduatecertificateinengineeringeducation–allfrom
ClemsonUniversity.Until2012,shewasthedirectoroftheSavannahRiverEnvironmentalSciencesField
Station.Dr.Simmonshasnearlyfourteenyearsofengineeringandprojectmanagementexperiencework-
ing with public utility companies, a project management consulting company, and a software company.
Sheisaregisteredprofessionalengineer,projectmanagementprofessionalandLEEDaccreditedprofes-
sional.. Her research interests are in investigating students’ development of leadership skills and other
professionalcompetenciesandinstudentinvolvementinco-curricularactivities. Dr. SimmonsisaNSF
CAREERawardwinnerforherresearchentitled, ”InvestigatingCo-CurricularParticipationofStudents
UnderrepresentedinEngineering.”
P
a
g
e

2
4
.
1
3
5
5
(cid:13)cAmericanSocietyforEngineeringEducation,2014 .1
Utilizing Think-Aloud Protocols to Assess the Usability of a
Test for Ethical Sensitivity in Construction

Abstract

Unethical conduct in the construction industry has been so prevalent that research has revealed it
is one of the most corrupt industries in international business. Additionally, accredited
construction and construction engineering programs are expected to provide professional ethics
instruction to its students to meet the standards established by the profession. However, no
assessment tool measuring a student’s ethical sensitivity to professional issues in construction
exists to determine whether this instruction improves student recognition of ethical issues of the
construction industry. The purpose of this paper is to outline the use of think-aloud protocols in
assessing the usability of a test instrument, the Test for Ethical Sensitivity in Construction
(TESC). This paper details the process taken to develop and administer think alouds, the results
of the think-alouds, and how the results assisted refinement of the TESC. The think-aloud
sessions helped researchers improve the usability of the TESC, making revisions that accounted
for mechanical or structural and cognitive issues of the instrument. It was found that think-aloud
sessions improved the TESC by helping to improve terminology, remove extraneous
information, understand the length of time required to complete the TESC, and improve the
scoring rubric.

Introduction and Motivation

Unethical conduct in the construction industry is so prevalent that Transparency International, a
non-governmental organization that monitors and publicizes corporate and political corruption in
international development, reveals that the construction industry is more corrupt than any other
sector of international economy1. The FMI Corp. performed a survey for the Construction
Management Association of America (CMMA) and reports, “the construction industry, in
general, is tainted by prevalent acts that are considered unethical,” adding that it is “tainted by
illegal acts”2.

As a part of the effort to curb unethical behavior, the mandate of construction related accrediting
bodies have instituted requirements for literacy of ethics in the curriculum. The American
Council for Construction Education (ACCE) requires ethics integration in construction
curriculum (at least 1 semester hour). The ACCE also states:

In addition, oral presentation, business writing, and ethics must be integrated
throughout the construction-specific curriculum. Example courses in this division
include: Human relations, psychology, sociology, social science, literature,
history, philosophy, art, language, political science, and other appropriate
courses.4

Additionally, the Accreditation Board for Engineering and Technology (ABET) has ethics
education requirements for construction programs. In the 2012-2013 ABET criteria for
P
accrediting engineering programs, two students’ outcomes of the list labeled (a) through (k) a
g
relate directly to ethics education and states that students should possess: e

2
4
.
1
3
5
5
.
2
(c) an ability to design a system, component, or process to meet desired needs
within realistic constraints such as economic, environmental, social, political,
ethical, health and safety, manufacturability, and sustainability and
(f) an understanding of professional and ethical responsibility.5

Construction programs must include some form of ethical component to satisfy both the
construction industry and accreditation requirements; however, no benchmark has been
identified in construction education literature that addresses whether ethics instruction in
construction education actually produces more ethically sensitive professionals. Furthermore,
there is no standard to teaching ethics in construction education and construction programs have
a multitude of options to satisfy the construction industry and accreditation requirements. In
order to address this, we must investigate ethical competencies of construction students. In order
to assess the competencies of construction students, an instrument was developed to measure the
minimum competency in ethical conduct, i.e. the ethical sensitivity of construction students.

Ethical sensitivity is derived from Rest’s four component model of cognitive moral development
which includes: moral sensitivity, moral judgment, moral intent, and moral behavior. Component
one, moral sensitivity, involves “imaginatively constructing possible scenarios, and knowing
cause-consequence chain of events in the real world; it involves empathy and role-taking skills”6.
For the purpose of this paper, ethics are associated with the rules of conduct recognized in a
particular profession or area of human life and morals are associated with personal values that
drive ethical behavior7. Adopting Rest’s moral sensitivity component and placing it in an ethical
context specifically for the construction industry, the term ethical sensitivity shall refer to the
ability of an individual to recognize ethical issues of the construction industry.

Operationalizing Ethical Sensitivity to Test Student Ethical Sensitivity

To investigate the ethical sensitivity of construction engineering students, a test was developed
called the Test for Ethical Sensitivity in Construction (TESC). This test is modeled after
Clarkeburn’s Test for Ethical Sensitivity in Science (TESS), which takes an open-ended
questionnaire response approach to investigate the ethical sensitivity of university science
students8. Such a test allows for qualitative spontaneous recognition of ethical issues that cannot
be observed with check box methods of ethical sensitivity investigation.

The TESC is an eight vignette qualitative response test for engineering students in the
construction industry with an original intended time of completion of 20 minutes. The TESC
situates students in a mental avatar with the following original scenario:

You recently graduated and have been hired as a project engineer for Solid
Construction, an established medium-sized commercial construction company in
the US. As an entry-level employee, you must participate in a rotation program so
that you are involved in various company operations.

The original eight-vignette test was developed from ethical concepts that were found to be major P
a
issues of the construction industry. The test embedded 14 issues of the construction industry into g
e

2
4
.
1
3
5
5
.
3
the eight vignettes (i.e. 14 pilot test items). The 14 issues were: claims games, collusion, bid
shopping, bid peddling, theft, abuse of client resources, unfair labor allocation through overtime,
labor issues, frontloading, payment games, low competence of work performance, improper
client relations, use of joint ventures to increase satisfaction of prequalification requirements, and
bid rigging.

Response generation of the TESC requires students to read and reflect on the vignette and
provide 3 statements or questions regarding each vignette. It is expected that within 3 statements
or questions, a student would be able to identify the issue, or write statements or questions that
showed a level of discomfort with the issues embedded in the vignette. Student response rating is
based on a three level scoring rubric, indicating the level of student ability to recognize an issue:

1) Score of 0 = Inability to recognize issue
2) Score of 1 = Unable to name issue but demonstrates unfavorable response to issue
3) Score of 2 = Ability to recognize issue by name or thoroughly demonstrates
unfavorable response to issue with explanation

Problem Statement/Aim

As a new instrument, the TESC requires refinement, review, and pilot testing. Certain unknowns
of a new instrument need addressing. Is the TESC clear? How long will it take to administer the
TESC? Essentially, instrument developers need to know the usability of the TESC prior to pilot
testing. The aim of this paper is to answer these questions using the verbal report (i.e. think-
aloud protocols) and to discuss benefits of testing the usability of an instrument.

Literature Review

Conceptualizing the Think-Aloud
The think-aloud is a type of verbal report method where useful research data is obtained via
elicitation of test taker vocalization of self-generated ‘symbols’ while he/she performs a given
task. The data elicitation method is referred to as ‘thinking aloud’ or ‘concurrent verbalization.’
This vocalization occurs at three different levels, vocalization of covert articulatory or oral
encodings, description/explication of the thought content, and explanation of the thought
processes or thoughts. The written transcripts of the verbalizations are called think-aloud
protocols10,11.

Adding to this, the think-aloud is an approach to cognitive assessment of a test taker, whereby
test administrators are able to observe the thought processes of the test taker whilst performing a
task (i.e. completing a test). It has been posed as a qualitative research tool designed to highlight
test taker’s thought processes during the administration of a test. Additionally, think-alouds can
yield valuable insights concerning perception, interpretation, and responses to test items12 (p.
274).

P
a
g
e

2
4
.
1
3
5
5
.
4
Usability Testing and the Think-Aloud of the TESC
Assurance of instrument rigor should involve tests of usability. The most widely accepted
definition of usability is from the International Organization for Standardization (ISO 9241-11)
which defines it as:

The extent to which a product can be used by specified users to achieve specified
goals with effectiveness, efficiency, and satisfaction in a specified context of use13
(p.1).

By another definition, usability is the ability of a user of a product to do what he or she wants to
do with the instrument the way he or she expects to do it without hindrance, hesitation, or
questions14 (p.4). For this paper, usability is the ability of test takers to complete the TESC as
intended by the researchers with ease and efficiency. Satisfaction of the components of usability
is important to rigorous data collection and results of actual TESC administration post piloting
the TESC. Think-alouds are widely used for usability testing in various fields15 (p. 339) and are
an essential element in achieving good results from small changes16. Barnum situates think-
alouds for usability testing and identifies the key elements of a think-aloud protocol in usability
tests as having the following:

1) Defining the user profile
2) Creating task-based scenarios
3) Using think-aloud processes
4) Making changes and testing again16

To be usable, an instrument must possess the ability of being useful, efficient, effective,
learnable and satisfying 14,16,17. For the TESC, relevant components of usability require
satisfaction prior to piloting. Table 1 summarizes each component of usability in general and
how it relates to the TESC. As related to the TESC, each of these key elements will be discussed
in another section.

Table 1.

Usability Components Relative to the TESC14

Component of Usability In General The TESC
Usefulness Degree to which a product The ease in which students
enables a user to achieve their complete the TESC
goal/willingness to use the
product/ease of use
Efficiency The quickness with which the The time in which the
user can accomplish their goal students complete the TESC
Effectiveness The extent to which the The construct validity of the
product behaves in the way TESC
users expect it to
Learnability As part of effectiveness, refers Essentially, it is expected P
a
to the user’s ability to operate that training involves ethics g
e

2
4
.
1
3
5
5
.
5
Component of Usability In General The TESC
the system at some level of education and thus the
competence after training TESC should not require
additional education to
complete since the TESC is
structured as a short answer
test students should be
accustomed to
Satisfaction User’s perception, feelings, The follow-up survey post
and opinions of the product, TESC think-aloud provides
usually captured through both satisfaction data
written and oral questions

Operationalizing the Think-Aloud in Research
There are different procedures of the think-aloud which uses various protocols to gain certain
results; however, they tend to follow a general approach12 (p. 274). This general approach has
been used in a variety of disciplines for a variety of purposes. The think-aloud has been used in
adjustment studies 20,21, clinical intervention 23,24, translational process research25,11, cognitive
psychology 10,12, and library studies15 as well as a host of other research studies.

Exemplar Think-Alouds in Engineering Education
In engineering education, various studies used think-aloud processes. A study by Atman and
Bursic utilized the think-aloud protocol as a tool to document the processes engineering students
use to approach design problems. The study was administered to two freshman engineering
students and differences in approach to completing a design problem were documented. The
study concluded that verbal protocol analysis (i.e. think-alouds) is a powerful tool for design
process assessment.26

Additionally, a study by Trenor and colleagues argues that the advantage of the think-aloud is to
provide evidence that survey items are interpreted by participants as the test administrator
intended, i.e. cognitive validity or construct validity. This study posed a cognitive validation
model for use by engineering education researchers in survey design. Operationalizing the
model, this study used ten participants across three iterations of think-alouds to see how they
respond to taking a web-based survey identifying and quantifying engineering undergraduates’
access to and activation of social capital in making decisions about entering and persisting in
engineering. It was found that issues of the survey ranged from grammatical errors to “serious
cognitive mismatches,” which caused issues with interpretation and responses.9

The Think-Aloud Protocol
Think-alouds range in type based on its intended use. There is no archetype protocol for
conducting think-aloud sessions. Literature provides general points of consideration for hosting a
think-aloud session which can keep the session in line with the intended purpose. Some guiding
principles include, telling the participant to voice confusion and assuring that the participant
knows that the instrument is being evaluated and not the participant. Additionally, literature
suggests that if a participant struggles with the test, the researcher should probe with particular P
a
questions to help fully understand the thought processes of the participant9,27,28. However, the g
e

2
4
.
1
3
5
5
.
6
protocol used for this study is minimally intrusive and does not probe participants with questions
during the think-aloud sessions.

Adding to these key principles, Van Someren and colleagues outlines the practical procedures in
obtaining think-aloud protocol data as summarized in Table 229.

Table 2.

Summary of Practical Procedures in Obtaining Think-Aloud Protocols29
Procedure Description
Setting 1) Ensure subject is at ease, room is quiet, water is available
2) Focus should be on tasks with limited interference by
experimenter for influence
3) Explain the purpose of research, protection/confidentiality of
data, and that the interest is in the way problems are completed
(the instrument) and not the ability of the test taker
Instructions 1) Give instructions about the task, i.e. perform the task and say out
loud what comes to your mind
2) Avoid elicitation of opinions, i.e. asking the participant to tell
you what they think
3) Avoid making the instructions too long
Warming Up 1) Allow participant time to practice thinking aloud
 Assists in getting participant to verbalize thoughts and
get comfortable with the task
Behavior of the 1) Experimenter should be restrained
Experimenter 2) Interfere ONLY when participant stops talking
Prompting 3) If interference is required, experimenter should at most say:
‘Keep on Talking’
Recording 1) Think-alouds are typically recorded
2) Check and recheck your instrument prior to starting session
3) Inconspicuously check your instrument periodically to ensure
that it works
Transcription of the 1) Transcribe the session as verbatim as possible and code from
Protocol transcription, not directly from audio tape
2) Everything in the session should be typed out (even interruptions
as they influence results)
3) Note recognizable pauses and unusual silences in transcription
by using dote, i.e. ‘I guess…the answer’
4) Avoid unwarranted interpretation
Review 1) When possible, reviewing protocol with participant can provide
very useful information, especially when protocols are difficult
to interpret
 If review will be done, do it as soon as possible after the
P
think-aloud session a
g
e

2
4
.
1
3
5
5
.
7
Methodology

Situating and Using the Think-Aloud Protocol
The creation of a test is the “product of the thoughtful and sound application of established
principles of test development”12 and the test development process involves five fundamental
stages as outlined in Figure 1.

Test Conceptualization Test ConstructionTest TryoutAnalysis Revision

Figure 1: The Test Development Process12

Additionally, Trenor and colleagues summarize literature and presents a model for the
instrument development process that highlights the concept of survey instrument cognitive
validation. This model, in conjunction with the test development process, inspires the TESC
instrument development process employed by this study (see Figure 2).

TESC Conceptualization
Participants (Theoretical grounding, construct deconstruction and
For the think-aloud sessions, three construct mapping)

student participants were
purposively selected. The small
representative sample size was
TESC Construction with Expert Review and
selected for the think-alouds are Refinement
similar to other qualitative methods,
seeking rich, in-depth data from a
small sample30 (p.432). Additionally,
research has found that the first three
participants detect the most severe
Perform Think-Aloud Revise TESC
problems of usability, additional Sessions of the TESC Instrument
participants are less likely to detect
additional issues, and running fewer
sessions saves time and money9,31. Saturation
Pilot Test
Point
TESC
Recruitment
Student participants were initially
recruited without promise of
Final Review of
compensation; however, gift cards TESC
for lunch were provided at the end of
the session for the participants’ time
Launch
and contribution to the study ($15
TESC
Panera Bread Gift Cards). The main
Figure 2: Think-Aloud Protocol in Context to the
consideration for recruitment was
Development of the TESC
the student enrollment and class
(Derived from Trenor and colleagues9)
standing in a
construction/construction P
a
engineering program. Special consideration was made to ensure two students were seniors, g
e

2
4
.
1
3
5
5
.
8
because the intended population of interest for the TESC is senior students who are expected to
have thorough knowledge of construction ethics based on program requirements (see Table 3 for
participant demographics).

Table 3.

Participant Demographics

Descriptor Participant 1 (P1) Participant 2 (P2) Participant 3 (P2)
Program of Study Construction Building Construction Building
Engineering Construction
Student Standing Senior Junior (4th Year) Senior
Age 22 22 22
Sex Male Female Male

Administration of the TESC Think-Aloud Sessions
Following the practical procedures of the model as provided by Van Someren and colleagues,
with slight modifications, the following sections describe and discuss the TESC think-aloud
procedures:

Setting(s)
The physical settings for hosting the think-aloud sessions were quiet spaces for recording with
conditions conducive for test taking. To ensure participants’ comfort, statements of purpose,
their rights and administrator behavior were explained. Participants were advised that this was a
test of the instrument and not of their ability. They were also advised that if they felt
uncomfortable, they could discontinue the session at any time. The participants were encouraged
to ask questions during the think-aloud, but warned that the administrator will not answer
questions.

Additionally, participants were warned that the administrator will be checking the time
frequently and this should not be misconstrued as being rude or otherwise, this is simply to
collect data. Lastly, participants were warned about prompts when there is a lull in the session.
Instead of verbally prompting participants to keep talking, participants were advised that taps on
the desk (with example tapping) would be an indicator for them to keep talking.

Instruction(s)
Participants were verbally instructed to say aloud whatever comes to their mind during the
process of performing the TESC. Due to the nature of the TESC, the participants were asked to
express any questions or concerns about the instrument during the session but were told that the
administrator would not be allowed to answer them. In addition, participants were told that the
administrator will not tell them when the session is complete or when to stop providing
responses to the TESC, and that they must determine this on their own.

P
a
g
e

2
4
.
1
3
5
5
.
9
Warming Up
In order to warm up, a simple exercise for thinking aloud was used. Participants were asked to
spend a minute describing the room where the think-aloud session was held. The method
employed is different from the task and was intentional, as the administrator did not want to hint
at the type of task the participant would be performing.

Behavior of Observer
During the think-aloud, the administrator of the TESC was minimally intrusive, only prompting
participants to continue talking via desk taps. The observer also avoided looking at the
participant and attempted to take limited handwritten notes to avoid distracting the participant.

Recording
The think-aloud session was recorded with an audio recording device.

Transcription of the protocol
The protocol was immediately transcribed from the audio recording with as many audio cues
transcribed as possible.

Revision and Feedback
After each session, the TESC was revised to account for issues discovered during the session.
Issues were resolved and changes were made after the first think-aloud and prior to
administrating the next think-aloud session (see Table 3). After completion of three iterations of
think-aloud sessions and prior to pilot testing the instrument, participant 1 was asked to review
the final version of the TESC and provide feedback.

Results

The outcomes of the think-aloud sessions revealed various usability issues with the TESC (see
Table 4 for examples issues revealed). Issues participants revealed during the think-aloud
sessions were mechanical/structural issues (S) (typos, grammar) and/or cognitive (C) (relative
information not recalled) issues, with the cognitive issues being more severe 9,27. Cognitive
issues may impact a student’s ability to correctly respond to the vignettes of the TESC. For the
TESC, cognitive issues were more prevalent than structural issues. Seven cognitive issues were
found with the TESC (see Table 4).

Think-aloud sessions also revealed attributes of the TESC that were simply indeterminable
during initial survey construction. The attributes includes time to complete, determination if
participants answer as intended, and elicitation of affective reasoning in processing responses.

Table 4.

Example Usability Issues Revealed by Think-Alouds

Participant Revealed Issues Correction Procedure
P
P1 (C) Project specifications too Changed specific project from airport a
g
specific, distracts intent of renovation to simply ‘renovation’ for e

2
4
.
1
3
5
5
.
1
0