Data Collection: Planning for and Collecting All Types of Data

190 Pages·2008·3.79 MB·English

by  Cathy Stawarski

JWSF012-Philip November27,2007 11:45
About This Book
Why This Book Is Important
ThissecondbookintheM&ESeriesbeginswithabriefintroduction
to the ROI process model and the Twelve Guiding Principles. It
goes on to explain, in detail, one of the most critical activities in
the evaluation process: data collection. Data are collected during
every step of the ROI Methodology, and how and when these data
are collected are crucial to successful program evaluation. This book
providesdetailedinformationaboutdatacollectionmethodsandhow
andwhentocollectdataateachevaluationlevel.
What This Book Achieves
This guide details how to develop or use each of the following data
collectiontools:
(cid:1)
Surveys
(cid:1)
Questionnaires
(cid:1)
Tests
(cid:1)
Simulations
(cid:1)
Interviews
(cid:1)
Focusgroups
(cid:1)
Directobservation
(cid:1)
Performancemonitoring
i
JWSF012-Philip November27,2007 11:45
ii AboutThisBook
(cid:1)
Actionplans
(cid:1)
Performancecontracts
Inaddition,thisbookexplainshowtoselecttheappropriatedata
collectionmethodforanysituationandanylevelofevaluation.
How This Book Is Organized
This book begins by introducing and describing each data collection
method and ends by discussing the methods that are best for each
evaluation level. Chapter One details how to design effective ques-
tionnaires and surveys in order to maximize response rates. It also
providestipsforpresentingquestionnairestoparticipantsinwaysthat
willencouragethemtoprovidevaluabledata.ChapterTwodiscusses
themajortypesoftests,includinghowtodevelopandadministerthem.
Whenandhowtousesimulationstocollectdataarealsoexplained.
Chapter Three explores how to conduct interviews and focus
groupsforoptimalresults,includingquestiondevelopmentandinter-
viewtechniques.Guidelinesforeffectiveobservationarealsoprovided.
Chapter Four explains how to use business performance monitoring,
actionplanning,andperformancecontractstocollectdata.
Chapter Five focuses on measuring reaction and planned action
and the importance of gathering participant feedback. Chapter Six
discusses the reasons for measuring learning data and then describes
howtousethedata.Measurementandadministrativeissuesinvolved
indatacollectionatthislevelarealsoexplored.
Chapter Seven explains the importance of collecting application
and implementation data. The challenges and issues of collecting
applicationdataareexplored,andappropriatedatacollectionmethods
are also discussed. Chapter Eight explores the critical reasons for
collecting impact data. Effective impact measures, appropriate data
collectionmeasures,andlinkingthemeasurestobusinessneedsareall
discussedinthischapter.
Finally,ChapterNineillustrateshowtoselecttheappropriatedata
collectionmethodforeachprogramandeachlevelofevaluation.
JWSF012-Philip November27,2007 11:45
The Measurement and Evaluation Series
Editors
Patricia Pulliam Phillips, Ph.D.
Jack J. Phillips, Ph.D.
JWSF012-Philip November27,2007 11:45
JWSF012-Philip November27,2007 11:45
Introduction to the Measurement and
Evaluation Series
The ROI Six Pack provides detailed information on developing
ROIevaluations,implementingtheROIMethodology,andshow-
ingthevalueofavarietyoffunctionsandprocesses.Withdetailed
examples,tools,templates,shortcuts,andchecklists,thisserieswill
beavaluablereferenceforindividualsinterestedinusingtheROI
Methodology to show the impact of their projects, programs, and
processes.
The Need
Although financial ROI has been measured for over one hundred
yearstoquantifythevalueofplants,equipment,andcompanies,the
concept has only recently been applied to evaluate the impact of
learning and development, human resources, technology, quality,
marketing,andothersupportfunctions.Inthelearninganddevel-
opment field alone, the use of ROI has become routine in many
organizations. In the past decade, hundreds of organizations have
embraced the ROI process to show the impact of many different
projectsandprograms.
Alongtheway,professionalsandpractitionersneedhelp.They
need tools, templates, and tips, along with explanations, exam-
ples, and details, to make this process work. Without this help,
using the ROI Methodology to show the value of projects and
v
JWSF012-Philip November27,2007 11:45
vi IntroductiontotheMeasurementandEvaluationSeries
programs is difficult. In short, practitioners need shortcuts and
proven techniques to minimize the resources required to use this
process. Practitioners’ needs have created the need for this se-
ries. This series will provide the detail necessary to make the ROI
Methodologysuccessfulwithinanorganization.Foreasyreference
anduse,thebooksarelogicallyarrangedtoalignwiththestepsof
theROIMethodology.
Audience
The principal audience for these books is individuals who plan to
use the ROI Methodology to show the value of their projects and
programs.Suchindividualsarespecialistsormanagerschargedwith
provingthevalueoftheirparticularprojectorprogram.Theyneed
detailedinformation,know-how,andconfidence.
AsecondaudienceisthosewhohaveusedtheROIMethodol-
ogy for some time but want a quick reference with tips and tech-
niques to make ROI implementation more successful within their
organization.Thisseries,whichexplainstheevaluationprocessin
detail,willbeavaluablereferencesetfortheseindividuals,regard-
lessofotherROIpublicationsowned.
A third audience is consultants and re-searchers who want to
know how to address specific evaluation issues. Three important
challengesfaceindividualsastheymeasureROIandconductROI
evaluations:(1)collectingpost-programdata,(2)isolatingtheef-
fectsoftheprogram,and(3)convertingdatatomonetaryvalues.A
bookisdevotedtoeachofthesecriticalissues,allowingresearchers
andconsultantstoeasilyfinddetailsoneachissue.
A fourth audience is those who are curious about the ROI
Methodologyanditsuse.Thefirstbookinthisseriesfocusesspecif-
ically on ROI, its use, and how to determine whether it is appro-
priateforanorganization.Wheninterestispiqued,theremaining
booksprovidemoredetail.
JWSF012-Philip November27,2007 11:45
IntroductiontotheMeasurementandEvaluationSeries vii
Flow of the Books
The six books are presented in a logical sequence, mirroring the
ROI process model. Book one, ROI Fundamentals: Why and When
to Measure ROI, presents the basic ROI Methodology and makes
thebusinesscaseformeasuringROIasitexploresthebenefitsand
barriers to implementation. It also examines the type of organi-
zation best suited for the ROI Methodology and the best time to
implement it. Planning for an ROI evaluation is also explored in
thisbook.
Booktwo,DataCollection:PlanningForandCollectingAllTypes
of Data, details data collection by examining the different tech-
niques, processes, and issues involved in this process, with an em-
phasis on collecting post-program data. It examines the different
data collection methods: questionnaires, interviews, focus groups,
observation, action plans, performance contracts, and monitoring
records.
Bookthree,IsolationofResults:DefiningtheImpactoftheProgram,
focuses on the most valuable part of the ROI Methodology and
theessentialstepforensuringcredibility.Recognizingthatfactors
other than the program being measured can influence results, this
bookshowsavarietyofwaysinwhichtheeffectsofaprogramcan
be isolated from other influences. Techniques include comparison
analysis using a control group, trend line analysis and forecasting
methods,andexpertinputfromavarietyofsources.
Book four, Data Conversion: Calculating the Monetary Benefits,
covers perhaps the second toughest challenge of ROI evaluation:
placingmonetaryvalueonprogrambenefits.TocalculatetheROI,
datamustbeconvertedtomoney,andDataConversionshowshow
this conversion has been accomplished in a variety of organiza-
tions.Thegoodnewsisthatstandardvaluesareavailableformany
items.Whentheyarenot,thebookshowsdifferenttechniquesfor
converting them, ranging from calculating the value from records
to seeking experts and searching databases. When data cannot be
JWSF012-Philip November27,2007 11:45
viii IntroductiontotheMeasurementandEvaluationSeries
convertedtomoneycrediblyandwithminimumresources,theyare
considered intangible. This book explores the range of intangible
benefitsandthenecessarytechniquesforcollecting,analyzing,and
recordingthem.
Bookfive,CostsandROI:EvaluatingattheUltimateLevel,focuses
oncostsandROI.Thisbookshowsthatallcostsmustbecaptured
in order to create a fully loaded cost profile. All the costs must be
included in order to be conservative and to give the analysis ad-
ditionalcredibility.Next,theactualROIcalculationispresented,
showingthevariousassumptionsandissuesthatmustbeaddressed
when calculating the ROI. Three different calculations are pre-
sented:thebenefit-costratio,theROIpercentage,andthepayback
period. The book concludes with several cautions and concerns
abouttheuseofROIanditsmeaning.
Booksix,CommunicationandImplementation:SustainingthePrac-
tice, explores two important issues. The first issue is reporting the
resultsofanevaluation.ThisisthefinalpartoftheROIMethodol-
ogyandisnecessarytoensurethataudienceshavetheinformation
they need so that improvement processes can be implemented. A
range of techniques is available, including face-to-face meetings,
briefreports,one-pagesummaries,routinecommunications,mass-
audience techniques, and electronic media. All are available for
reporting evaluation results. The final part of the book focuses on
how to sustain the ROI evaluation process: how to use it, keep
it going, and make it work in the long term to add value to the
organizationand,often,toshowthevalueofalltheprogramsand
projectswithinafunctionordepartment.
Terminology: Programs, Projects, Solutions
In this series the terms program and project are used to describe
manyprocessesthatcanbeevaluatedusingtheROIMethodology.
Thisisanimportantissuebecausereadersmayvarywidelyintheir
perspectives. Individualsinvolvedin technology applicationsmay
JWSF012-Philip November27,2007 11:45
IntroductiontotheMeasurementandEvaluationSeries ix
Table I.1 Terms and Applications
Term Example
Program Leadershipdevelopmentskillsenhancementfor
seniorexecutives
Project Areengineeringschemeforaplasticsdivision
System Afullyinterconnectednetworkforallbranchesofa
bank
Initiative Afaith-basedefforttoreducerecidivism
Policy Anewpreschoolplanfordisadvantagedcitizens
Procedure Anewschedulingarrangementfortruckdrivers
Event Agolfoutingforcustomers
Meeting AU.S.CoastGuardconferenceoninnovations
Process Qualitysampling
People Staffadditionsinthecustomercarecenter
Tool Anewmeansofselectinghotelstaff
use the terms system and technology rather than program or project.
Inpublicpolicy,incontrast,thewordprogramisprominent.Fora
professionalmeetingsandeventsplanner,thewordprogrammaynot
bepertinent,butinhumanresources,programisoftenused.Finding
one term for all these situations would be difficult. Consequently,
the terms program and project are used interchangeably. Table I.1
liststheseandothertermsthatmaybeusedinothercontexts.
Features
Each book in the series takes a straightforward approach to make
it understandable, practical, and useful. Checklists are provided,
chartsareincluded,templatesarepresented,andexamplesareex-
plored.AllareintendedtoshowhowtheROIMethodologyworks.
Thefocusofthesebooksisimplementingtheprocessandmakingit
successfulwithinanorganization.Themethodologyisbasedonthe
workofhundredsofindividualswhohavemadetheROIMethod-
ologyasuccessfulevaluationprocesswithintheirorganizations.