Table Of ContentNumber145
Spring2015 NewDirectionsforEvaluation
PaulR.Brandon
Editor-in-Chief
Accreditation, Certification,
and Credentialing: Relevant
Concerns for U.S. Evaluators
James W. Altschuld
Molly Engle
Editors
ACCREDITATION,CERTIFICATION,ANDCREDENTIALING:RELEVANTCONCERNS
FORU.S.EVALUATORS
JamesW.Altschuld,MollyEngle(eds.)
NewDirectionsforEvaluation,no.145
PaulR.Brandon,Editor-in-Chief
Copyright© 2015 Wiley Periodicals, Inc., A Wiley Company, and the
American Evaluation Association. All rights reserved. No part of this
publication may be reproduced in any form or by any means, except
as permitted under sections 107 and 108 of the 1976 United States
Copyright Act, without either the prior written permission of the
publisher or authorization through the Copyright Clearance Center,
222 Rosewood Drive, Danvers, MA 01923; (978) 750-8400; fax (978)
646-8600. The copyright notice appearing at the bottom of the first
pageofachapterinthisjournalindicatesthecopyrightholder’sconsent
that copies may be made for personal or internal use, or for personal
orinternaluseofspecificclients,ontheconditionthatthecopierpay
forcopyingbeyondthatpermittedbylaw.Thisconsentdoesnotextend
tootherkindsofcopying,suchascopyingforgeneraldistribution,for
advertisingorpromotionalpurposes,forcreatingcollectiveworks,orfor
resale.Suchpermissionrequestsandotherpermissioninquiriesshould
be addressed to the Permissions Department, c/o John Wiley © Sons,
Inc.,111RiverStreet,Hoboken,NJ07030;(201)748-6011,fax(201)
748-6008,www.wiley.com/go/permissions.
Microfilmcopiesofissuesandarticlesareavailablein16mmand35mm,
aswellasmicrofichein105mm,throughUniversityMicrofilmsInc.,300
NorthZeebRoad,AnnArbor,MI48106-1346.
NewDirectionsforEvaluationisindexedinAcademicSearchAlumniEdi-
tion(EBSCOPublishing),EducationResearchComplete(EBSCOPub-
lishing),HigherEducationAbstracts(ClaremontGraduateUniversity),
SCOPUS (Elsevier), Social Services Abstracts (ProQuest), Sociological
Abstracts(ProQuest),WorldwidePoliticalScienceAbstracts(ProQuest).
NEWDIRECTIONSFOREVALUATION(ISSN1097-6736,electronicISSN1534-
875X)ispartofTheJossey-BassEducationSeriesandispublishedquar-
terlybyWileySubscriptionServices,Inc.,AWileyCompany,atJossey-
Bass, One Montgomery Street, Suite 1200, San Francisco, CA 94104-
4594.
SUBSCRIPTIONSforindividualscost$89forU.S./Canada/Mexico/interna-
tional. For institutions, $358 U.S.; $398 Canada/Mexico; $432 inter-
national. Electronic only: $89 for individuals all regions; $358 for
institutionsallregions.Printandelectronic:$98forindividualsinthe
U.S.,Canada,andMexico;$122forindividualsfortherestoftheworld;
$430 for institutions in the U.S.; $470 for institutions in Canada and
Mexico;$504forinstitutionsfortherestoftheworld.
Allissuesareproposedbyguesteditors.Forproposalsubmissionguide-
lines, go to http://www.eval.org/p/cm/ld/fid=48. Editorial correspon-
dence should be addressed to the Editor-in-Chief, Paul R. Brandon,
UniversityofHawai‘iatMa¯noa,1776UniversityAvenue,CastleMemorial
HallRm118,Honolulu,HI96822-2463.
www.josseybass.com
Coverphotographby©iStock.com/Smithore
NEW DIRECTIONS FOR EVALUATION
SponsoredbytheAmericanEvaluationAssociation
EDITOR-IN-CHIEF
PaulR.Brandon UniversityofHawai‘iatMa¯noa
ASSOCIATEEDITORS
J.BradleyCousins UniversityofOttawa
Lois-ellinDatta DattaAnalysis
EDITORIALADVISORYBOARD
AnnaAhSam UniversityofHawai‘iatMa¯noa
MichaelBamberger Independentconsultant
GailBarrington BarringtonResearchGroup,Inc.
FredCarden InternationalDevelopmentResearchCentre
ThomasChapel CentersforDiseaseControlandPrevention
LeslieCooksy SierraHealthFoundation
FionaCram KatoaLtd.
PeterDahler-Larsen UniversityofSouthernDenmark
E.JaneDavidson RealEvaluationLtd.
StewartDonaldson ClaremontGraduateUniversity
JodyFitzpatrick UniversityofColoradoDenver
JenniferGreene UniversityofIllinoisatUrbana-Champaign
MelvinHall NorthernArizonaUniversity
GeorgeM.Harrison UniversityofHawai‘iatMa¯noa
GaryHenry VanderbiltUniversity
RodneyHopson GeorgeMasonUniversity
GeorgeJulnes UniversityofBaltimore
JeanKing UniversityofMinnesota
SavilleKushner UniversityofAuckland
RobertLahey RELSolutionsInc.
MiriLevin-Rozalis BenGurionUniversityoftheNegevandDavidsonInstituteat
theWeizmannInstituteofScience
LauraLeviton RobertWoodJohnsonFoundation
MelvinMark PennsylvaniaStateUniversity
SandraMathison UniversityofBritishColumbia
RobinLinMiller MichiganStateUniversity
MichaelMorris UniversityofNewHaven
DebraRog WestatandtheRockvilleInstitute
PatriciaRogers RoyalMelbourneInstituteofTechnology
MaryAnnScheirer ScheirerConsulting
RobertSchwarz UniversityofToronto
LynShulha Queen’sUniversity
NickL.Smith SyracuseUniversity
SanjeevSridharan UniversityofToronto
MonicaStitt-Bergh UniversityofHawai‘iatMa¯noa
Editorial Policy and Procedures
NewDirectionsforEvaluation,aquarterlysourcebook,isanofficialpublicationofthe
AmericanEvaluationAssociation.Thejournalpublishesworksonallaspectsofevalua-
tion,withanemphasisonpresentingtimelyandthoughtfulreflectionsonleading-edge
issues of evaluation theory, practice, methods, the profession, and the organizational,
cultural,andsocietalcontextwithinwhichevaluationoccurs.Eachissueofthejournal
isdevotedtoasingletopic,withcontributionssolicited,organized,reviewed,andedited
byoneormoreguesteditors.
Theeditor-in-chiefisseekingproposalsforjournalissuesfromaroundtheglobeabout
topics new to the journal (although topics discussed in the past can be revisited). A
diversityofperspectivesandcreativebridgesbetweenevaluationandotherdisciplines,
aswellaschaptersreportingoriginalempiricalresearchonevaluation,areencouraged.
Awiderangeoftopicsandsubstantivedomainsisappropriateforpublication,including
evaluativeendeavorsotherthanprogramevaluation;however,theproposedtopicmust
beofinteresttoabroadevaluationaudience.
Journalissuesmaytakeanyofseveralforms.Typicallytheyarepresentedasaseriesof
relatedchapters,buttheymightalsobepresentedasadebate;anaccount,withcritique
andcommentary,ofanexemplaryevaluation;afeature-lengtharticlefollowedbybrief
criticalcommentaries;orperhapsanotherformproposedbyguesteditors.
Submitted proposals must follow the format found via the Association’s website at
http://www.eval.org/Publications/NDE.asp.Proposalsaresenttomembersofthejour-
nal’sEditorialAdvisoryBoardandtorelevantsubstantiveexpertsforsingle-blindpeer
review.Theprocessmayresultinacceptance,arecommendationtoreviseandresubmit,
orrejection.Thejournaldoesnotconsiderorpublishunsolicitedsinglemanuscripts.
Beforesubmittingproposals,allpartiesareaskedtocontacttheeditor-in-chief,whois
committedtoworkingconstructivelywithpotentialguesteditorstohelpthemdevelop
acceptableproposals.Foradditionalinformationaboutthejournal,seethe“Statement
oftheEditor-in-Chief”intheSpring2013issue(No.137).
PaulR.Brandon,Editor-in-Chief
UniversityofHawai‘iatMa¯noa
CollegeofEducation
1776UniversityAvenue
CastleMemorialHall,Rm.118
Honolulu,HI968222463
e-mail:[email protected]
C
ONTENTS
EDITORS’NOTES 1
JamesW.Altschuld,MollyEngle
1. TheInexorableHistoricalPressoftheDeveloping 5
EvaluationProfession
JamesW.Altschuld,MollyEngle
Theauthorsprovideanoverviewofthedevelopmentofthefieldof
evaluationduringthepast40–50yearswithimplicationspertinentto
accreditation,certification,andcredentialing.
2. CompetenciesforProgramEvaluatorsinLightofAdaptive 21
Action:What?SoWhat?NowWhat?
JeanA.King,LaurieStevahn
Thischaptercontainsadescriptionofkeytermsrelatedto
competencies,currentworkgoingoninstudyingthem,issuesfor
consideration,anduseoftheskillsintheevaluationfield.
3. TheStateofPreparingEvaluators 39
JohnM.LaVelle,StewartI.Donaldson
Inthischapterareareviewoftrendsinevaluationeducation
programs,recentadvancesintheprofessionalpreparationof
evaluators,andadiscussionoffutureresearchonkeytopicsin
evaluationeducation.
4. HowDoesAccreditationFitIntothePicture? 53
JamesC.McDavid,IreneHuse
Usingbusinessmanagement,accounting,andhumanresource
developmentasexamples,theaccreditationofinstitutionsor
programsofferingevaluationeducationisthoughtfullyexamined.
5. CredentialedEvaluatorDesignationProgram,theCanadian 71
Experience
KeikoKuji-Shikatani
TherationalefortheCanadiansystemandhowitcameintobeing
(theforcesthatshapedit,whatitlookslike,howitfunctions,issues,
andotheraspects)arethemaintopicscovered.
6. EvaluatorCertificationandCredentialingRevisited:A 87
SurveyofAmericanEvaluationAssociationMembersinthe
UnitedStates
MichelleBaronSeidling
ThischapteranalyzestheresultsofasurveyofAEAmemberswithin
theUnitedStatesregardingthefeasibility,need,andpotentialfor
certificationorcredentialinginthefuture.
7. Accreditation,Certification,Credentialing:DoesItHelp? 103
GeneShackman
Thischapterisasynthesisoftheprioroneswithacarefullookatthe
evidenceaboutwhetheraccreditation,certification,orcredentialing
improvestheevaluationofprogramsorprojects.
INDEX 115
E ’ N
DITORS OTES
A
s charter members of the American Evaluation Association (AEA)
and being active in its predecessor organizations, we have an ex-
tensive history of discussing, reviewing, debating, and wondering
about accreditation, certification, and credentialing (ACC). We are long-
term collaborators (over 34 years) who have been involved continuously
in presenting and writing about the preparation of evaluators and cer-
tification. In the early 1990s as an AEA board member, Engle recruited
(cajoled, actually) Altschuld to lead a new directory of evaluation prepa-
rationprograms.
This effort led to a New Directions in Program Evaluation (now New
DirectionsforEvaluation[NDE])issue(No.62)publishedin1994.Wenote
this earlier result because it relates to the current issue. The first coedi-
tor was communicating with the NDE Editor-in-Chief about an issue on
needsassessment(Engleisacontributortoit).Inthatconversation,itwas
mentioned that she and I were recommended for an update to the 1994
publicationfocusingontheinterconnectedideasofACC.
Wow! What an opportunity to reconsider a topic that was done two
decades ago, especially given the vastly changing context of the practice
ofevaluation,thetheoreticalunderstandingsofit,andwhathashappened
andishappeningwithprofessionalorganizations.Weweredelightedtobe
thoughtofforthetaskandtherestofthestoryis,astheysay,“history.”
Thisissueisnotanupdateofthe1994effort,norisitintendedtobe:its
focusisoninterconnectinganddissectingideasrelativetoACC.Wewanted
to take a fresh look at them and we wanted our chapter authors to do the
samething.Tothatend,weassembledadiversegroupofindividualswith
uniqueviewpointstobeourwriters.
Wedebatedhowtostartthecurrentjourneyandfeltthatlookinginto
ACCnecessitatedplacingthethreecomponentsinthehistoricalcontextof
how evaluation has evolved in the United States since the 1960s. We are
notadvocatingforACC;rather,weilluminatewhattheirnatureisandhow
past and current conditions have influenced thinking about them. We see
the three parts of ACC as interrelated, and as critically important to any
discussionofourfield.
We start the discussion of the historical context by creating a time-
linefromthe1960stothepresent(althoughwearesurethatsomeevents
are omitted, making this a personal expression of history, not an actual
one). This is the core content in Chapter 1 (“The Inexorable Historical
Press of the Developing Evaluation Profession”) written by us. In what
waysdothepastandthepresentinformandguideusrelativetoACC?The
NAsEsWocDiaIRtiEoCnT.IONSPFuObRlisEhVeAdLUoAnTlIiOnNe,innoW.1il4e5y,OSpnrliinnge2L0ib1r5ary©(wi2l0ey1o5nWlinileelyibPrearriyo.cdoicma)ls,•InDc.O,aI:n1d0t.h1e00A2m/eevr.2ic0a1n0E7valuation 1
2 ACCREDITATION,CERTIFICATION,ANDCREDENTIALING
chapter also contains definitions of terms and how changes in evaluation
haveaffectedACC.
Having set the stage, we challenged our authors to go beyond simply
providinganupdateofwhatisgoingonatpresent;weaskedthemtodelve
intothesubtleandproblematicconcernsthatarepartofallthecomponents
of ACC. They were up to the responsibility, and we are grateful for their
opennessandwillingnesstoacceptthischargewithoutflinching.
The focus of Chapter 2 (“Competencies for Program Evaluators in
LightofAdaptiveAction:What?SoWhat?NowWhat?”),byJeanA.King
andLaurieStevahn,addressesthecompetenciesthatdefineevaluators.This
is a tricky consideration, and an ACC issue without it would certainly
miss the mark. These two authors, along with others, have been study-
ing competencies (and competent practice) for nearly 15 years and are
known throughout the world for their publications and in-depth research
and thinking about this area. They have done an excellent job of dealing
withconcepts,examiningsalientresearch,andthenraisingquestionsthat
force evaluators to be more thoughtful in regard to specifying what they
bringtothetableviatheirservicestoprogramsandprojects.
Withthisbackdropofcompetencies,itwouldseemasimplematterto
describe the educational environment necessary for evaluators. Put sim-
ply, “Where do individuals acquire the skills and culture of the profes-
sionalevaluator?”Weknewtheanswerwasn’tstraightforwardandsought
out John M. LaVelle and Stewart I. Donaldson for preparing Chapter 3
(“TheStateofPreparingEvaluators”).Inthepastfewyearstheyhavebeen
conductingresearchaboutthenatureofprogramsatuniversitiesandjustre-
centlycompletedanotherstudy.Butlikeeverythingelseinthefield,prepa-
ration is now extensively offered in many venues other than academia.
What does the educational picture look like and what might it become in
the near future? The landscape they draw for us opens vistas to a quite
complexscene.
What role does or should accreditation play in the ACC scenario was
the question posed to James C. McDavid and Irene Huse for Chapter 4
(“How Does Accreditation Fit Into the Picture?”). McDavid and Huse are
Canadianswhoauthoredamajortextonevaluationandwhowereinstru-
mental in writing the 2006 literature review that was the underpinning of
the2009CanadianEvaluationSociety’sCredentialedEvaluatorDesignation
system for program evaluators. With expanding educational and employ-
ment options, the field must be concerned with accreditation of evaluator
preparation programs. How should evaluation as a budding profession go
aboutensuringtheirvalidityandquality?Thisconsiderationwasraisedby
AEA in the 1990s and is as meaningful now as it was then. McDavid and
Huse’svastknowledgeofthefieldgetsusclosertoaresponse.
Asbrieflymentionedabove,Canadainitiatedasystemthatisbecoming
known to many evaluators in the United States and in the world. It could
NEWDIRECTIONSFOREVALUATION•DOI:10.1002/ev
