Dynamic Quality Management for Cloud Labor Services: Methods and Applications for Gaining Reliable Work Results with an On-Demand Workforce

219 Pages·2014·7.589 MB·English

by  Robert Kern (auth.)

Robert Kern
Dynamic Quality
Management for
2
9
1

P Cloud Labor Services
I
B
N
L
Methods and Applications for Gaining Reliable Work Results
with an On-Demand Workforce
123
Lecture Notes
in Business Information Processing 192
Series Editors
Wil van der Aalst
Eindhoven Technical University, Eindhoven, The Netherlands
John Mylopoulos
University of Trento, Povo, Italy
Michael Rosemann
Queensland University of Technology, Brisbane, QLD, Australia
Michael J. Shaw
University of Illinois, Urbana-Champaign, IL, USA
Clemens Szyperski
Microsoft Research, Redmond, WA, USA
More information about this series at http://www.springer.com/series/7911
Robert Kern
Dynamic Quality
Management for
Cloud Labor Services
Methods and Applications for Gaining Reliable
Work Results with an On-Demand Workforce
123
RobertKern
Karlsruhe Service Research Institute
Karlsruhe Instituteof Technology
Karlsruhe
Germany
ISSN 1865-1348 ISSN 1865-1356 (electronic)
ISBN 978-3-319-09775-6 ISBN 978-3-319-09776-3 (eBook)
DOI 10.1007/978-3-319-09776-3
LibraryofCongressControlNumber:2014946606
SpringerChamHeidelbergNewYorkDordrechtLondon
©SpringerInternationalPublishingSwitzerland2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartofthe
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodologynow
knownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnectionwith
reviewsorscholarlyanalysisormaterialsuppliedspecificallyforthepurposeofbeingenteredandexecuted
onacomputersystem,forexclusiveusebythepurchaserofthework.Duplicationofthispublicationor
partsthereofispermittedonlyundertheprovisionsoftheCopyrightLawofthePublisher’slocation,inits
currentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Permissionsforusemaybe
obtainedthroughRightsLinkattheCopyrightClearanceCenter.Violationsareliabletoprosecutionunder
therespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpublication,
neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforanyerrorsor
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
materialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Foreword
Not least due to the rapid advances in theinformation and communication technology
and the resulting dynamics, organizations are required to adapt quickly to changing
customer requirements and demand. The ability to keep business processes both
flexible and scalable has become a particularly critical factor for success requiring an
“on-demand” increase or decrease of the quantity of resources used in business pro-
cesses. For IT resources, the current increasingly popular cloud computing concepts
impressively demonstrate and capture the potential. Despite all the rationalization and
automation efforts of the past decades, however, many typical business processes are
stilldependentontheuseofhumanlaborandintelligence.Thus,onlyacomplementary
conceptof“cloudlabor”wouldleadtobusinessprocessesthataretrulyfullyscalable.
First offerings like Amazon’s Mturk platform or service providers that allow single
micro-taskstobecarriedoutbyalargenumberofpeopleoutsideanorganizationhave
emergedoverthelastdecade. Unfortunately,noneofthese crowdsourcing approaches
comprise efficient mechanisms to guarantee a required level of quality of the work
results.However,exactlythiswouldbetheprerequisitetobeabletomorewidelyapply
suchconceptsinbusiness-criticalprocessesandtoactuallyexploitthefullpotentialfor
commercial enterprises as well as for nonprofit purposes.
Robert Kern’s work makes a considerable contribution toward solving the quality
problem for scalable human work. Based on a comprehensive framework of cloud
labor,hedevelopsasetofmethods toconceptuallymeasureandaggregatethequality
of human work results, implements a platform to put those methods to work, and
evaluates their application in a number of very compelling, real-world scenarios suc-
cessfully combining concepts from statistics, information technology, and manage-
ment. Reading this book will be beneficial to novices in cloud labor services looking
for orientation in this new field as well as to advanced researchers and practitioners
developingcloudqualityconcepts–inordertotakescalabilityofbusinessinformation
processesontothenextlevel.Theworkshouldalsocontributetowardrationalizingthe
larger societal debate on the future of human work.
Iwishtheaudienceanenjoyableandinspiringreadingandtrustthat thisworkwill
draw wide attention and reception in both academia and industry.
Karlsruhe, Germany
May 2014 Gerhard Satzger
Director IBM Business Performance Services Europe/
Karlsruhe Service Research Institute (KSRI)
Acknowledgments
This book was originally written as a PhD thesis with the Karlsruhe Service Research
Institute (KSRI) of the Karlsruhe Institute of Technology (KIT) in Germany. It would
not have been possible without the support and guidance of a large number of col-
leagues, industry partners, students, and friends.
First and foremost, I would like to thank my supervisor Prof. Dr. Gerhard Satzger
forhiscontinuousencouragementandforhistimelyandcomprehensivesupportinall
stages of my studies. Moreover, I am indebted to IBM for providing me with the
opportunitytopursuethisresearchandforsupportingmyresearchprojectwithanIBM
Faculty Award grant. Likewise, I express my sincere gratitude to Prof. Dr. Christof
Weinhardt for co-advising this thesis and providing me with constructive comments.
I am grateful to my colleagues from the KSRI and specifically from the Service
Innovation and Management group for providing an enjoyable and motivating work
environment and supporting me with regular feedback that has been essential for the
successofmythesis.SpecialthankstoDr.ChristianZirpins,whohelpedmefocusmy
research in an early phase of the dissertation and accompanied my work with con-
tinuous feedback, encouragement, and cooperation throughout the years.
I would also like to thank Ajith Kumar Parlikad and Alexander Borek from the
UniversityofCambridgeforoursuccessfuljointresearchactivities.Inaddition,Iwish
to acknowledge the valuable contributions of the diploma, master, and bachelor stu-
dentsandstudentassistantsHansThies,CordulaBauer,PascalWichmann,JanMeller,
David Bermbach, and Sandra Rath.
The best theoretical models are worth nothing without an exhaustive validation in
real-life scenarios. Therefore, I thank all my contacts and supporters at buw Group,
bitworxx GmbH, Humangrid GmbH, IBM, Semfinder AG and at the health insurance
company who supported the medical coding case study. In particular, I would like to
thank Roland Peisl, Andreas Pasing, Ronald Fritz, Sönke Volquartz, Harald Kiehle,
GerhardPfau,AlexanderSchmid,HansRudolfStraub,MaurusDuelli,FabianDimski,
Christian Rozsenich, Sacha Moufarrege, Volker Pigors, and Antje Guenther for the
fruitful collaboration.
Likewise, I want to thank all who supported me in the final stage of my work, in
particular Dr. Simon Caton for his general feedback and Stefan Momma for his gen-
erous offer and commitment to proofread several versions of my thesis.
Finally and above all, I wish to express my deepest gratitude to my dear wife,
Bernadett, for her continuous support, her love, and her patience.
Karlsruhe, Germany
May 2014 Robert Kern
Contents
Introduction
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Structure of This Book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 History of This Book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Fundamentals of Cloud Labor Services
2 Cloud Labor Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.2 Concept. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Related Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 Paid Crowdsourcing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.3 Human Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.4 Social Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.5 Open Source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.6 Collective Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.7 Human Tasks inBusiness Process Management and SOA. . . . 21
2.2.8 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Application Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Platform Challenges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.3 Workforce Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Application Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.1 Existing Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.2 Identification of Relevant Tasks . . . . . . . . . . . . . . . . . . . . 31
2.4.3 Workflows and Task Granularity. . . . . . . . . . . . . . . . . . . . 33
2.4.4 Privacy, Copyright and Compliance. . . . . . . . . . . . . . . . . . 35
2.5 Platform Perspective. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.5.1 Existing Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.5.2 Technical Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.5.3 Worker-to-Task Matching and Allocation. . . . . . . . . . . . . . 42
2.5.4 Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6 Workforce Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.1 Worker Demographics. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.2 Motivation and Incentives . . . . . . . . . . . . . . . . . . . . . . . . 45
2.6.3 Education and Feedback. . . . . . . . . . . . . . . . . . . . . . . . . . 47
X Contents
2.6.4 Task Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.6.5 Work Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3 Quality Management for Cloud Labor Services. . . . . . . . . . . . . . . . . 51
3.1 Quality of Cloud Labor Services . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1.1 Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1.2 Relevant Quality Dimensions . . . . . . . . . . . . . . . . . . . . . . 52
3.1.3 Influencing Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2 Quality Management Approaches. . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2.1 Qualification Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2.2 Output-Based Quality Management . . . . . . . . . . . . . . . . . . 58
3.2.3 Execution Process Monitoring. . . . . . . . . . . . . . . . . . . . . . 58
3.2.4 Response Time Management . . . . . . . . . . . . . . . . . . . . . . 59
3.3 Patterns for Output-Based Quality Management. . . . . . . . . . . . . . . 60
3.3.1 Relevant Task Characteristics . . . . . . . . . . . . . . . . . . . . . . 60
3.3.2 Gold Pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.3.3 Voting Pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.3.4 Validation Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.3.5 Iteration Pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3.6 Comparison Pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4 Comparison of Output-Based Approaches. . . . . . . . . . . . . . . . . . . 71
3.4.1 Decision Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.4.2 Gap Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Dynamic Quality Management for Cloud Labor Services
4 Statistical Quality Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.1 Quality Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.2 Statistical Quality Control. . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2 Areas of Statistical Quality Control . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.1 Acceptance Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.2 Statistical Process Control . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 Design of Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3 Acceptance Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.2 Single-Sampling Plans for Attributes . . . . . . . . . . . . . . . . . 84
4.4 Continuous Sampling Plans. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.4.1 The Continuous Sampling Plan 1 (CSP-1) . . . . . . . . . . . . . 90
4.4.2 DeterminationofClearanceNumberandSamplingFraction... 91
5 Core Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.1.1 Application of Statistical Quality Control. . . . . . . . . . . . . . 94
5.1.2 Quality Management Patterns for Sample Inspection . . . . . . 95
Contents XI
5.2 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.2.1 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.2.2 Process Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.2.3 Rationale for Using the CSP-1 . . . . . . . . . . . . . . . . . . . . . 100
5.3 Dynamic Voting Mechanism (DVM) . . . . . . . . . . . . . . . . . . . . . . 101
5.3.1 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3.2 Statistical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.3 Calculation of the Conditional Probabilities . . . . . . . . . . . . 103
5.4 Completion Time Management . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4.2 Maximum Throughput. . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4.3 Fixed Deadline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.5 Model Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.5.1 Configuration of the CSP-1 and the DVM . . . . . . . . . . . . . 107
5.5.2 Initialization of Worker Failure Rates. . . . . . . . . . . . . . . . . 108
5.5.3 Detailed Process Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.5.4 Selection of Completion Time Management Option. . . . . . . 110
5.6 Small Worker Pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.6.2 Prevention of Delayed Task Inspection . . . . . . . . . . . . . . . 111
6 Model Variations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.1 Multi-labeling Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.1.1 Types of Classification Scenarios . . . . . . . . . . . . . . . . . . . 114
6.1.2 Assumptions and Definitions . . . . . . . . . . . . . . . . . . . . . . 114
6.1.3 Application of the DVM . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.1.4 Process Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.1.5 Posterior Probability per Label . . . . . . . . . . . . . . . . . . . . . 118
6.1.6 Identification of Best Suitable Labeling Decision. . . . . . . . . 119
6.1.7 Updating the Worker’s Sensitivity and Specificity. . . . . . . . 122
6.2 Non-deterministic Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.2.2 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.2.3 Process Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.2.4 Sampling Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.2.5 Inspection Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.2.6 Number of Reviewers to Be Used. . . . . . . . . . . . . . . . . . . 131
Evaluation and Case Studies
7 Toolkit Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.2 Live Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.3 Simulation Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139