Barriers to and Facilitators of the Evaluation of Integrated Community-Wide Overweight Intervention

19 Pages·2016·3.77 MB·English

International Journal of
Environmental Research
and Public Health
Article
Barriers to and Facilitators of the Evaluation of
Integrated Community-Wide Overweight
Intervention Approaches: A Qualitative Case Study
in Two Dutch Municipalities
TessaM.vanKoperen1,*,AnjadeKruif2,LisavanAntwerpen3,Anna-MarieHendriks4,
JacobC.Seidell5,AlbertineJ.Schuit6andCarryM.Renders1
1 DepartmentofHealthSciences,FacultyofEarthandLifeSciences,SectionPreventionandPublicHealthand
EMGO+InstituteforHealthandCareResearch,VUUniversityAmsterdam,DeBoelelaan1085,
Amsterdam1081HV,TheNetherlands;[email protected]
2 DepartmentofHealthSciences,FacultyofEarthandLifeSciences,SectionMethodsandApplied
Biostatistics,QualitativeResearch,VUUniversityAmsterdam,DeBoelelaan1085,Amsterdam1081HV,
TheNetherlands;[email protected]
3 YouthonHealthWeight,Causariestraat5,DenHaag2511VB,TheNetherlands;[email protected]
4 DepartmentofHealthPromotion,FacultyofHealth,MedicineandLifeSciences,Maastricht,
MaastrichtUniversity,Universiteitssingel60,Maastricht6229ER,TheNetherlands;
[email protected]
5 DepartmentofHealthSciences,FacultyofEarthandLifeSciencesandtheEMGO+InstituteforHealthand
CareResearch,VUUniversityAmsterdam,DeBoelelaan1085,Amsterdam1081HV,TheNetherlands;
[email protected]
6 NationalInstituteforPublicHealthandtheEnvironment,POBox1,Bilthoven3720BA,TheNetherlands;
[email protected]
* Correspondence:[email protected];Tel.:[phone]
AcademicEditors:SaraKirk,Jessie-LeeMclsaacandTarraPenney
Received:5February2016;Accepted:28March2016;Published:31March2016
Abstract: Topreventoverweightandobesitytheimplementationofanintegratedcommunity-wide
interventionapproach(ICIA)isoftenadvocated. Evaluationcanenhanceimplementationofsuch
an approach and demonstrate the extent of effectiveness. To be able to support professionals in
the evaluation of ICIAs we studied barriers to and facilitators of ICIA evaluation. In this study
tenprofessionalsoftwoDutchmunicipalitiesinvolvedintheevaluationofanICIAparticipated.
Weconductedsemi-structuredinterviews(n=12),observedprogrammemeetings(n=4)andcarried
outdocumentanalysis. Datawereanalyzedusingathematiccontentapproach. Welearnedthat
evaluationishamperedwhenitisperceivedasunfeasibleduetolimitedtimeandbudget,alackof
evaluationknowledgeoranegativeevaluationattitude. Otherbarriersareapoorunderstandingof
theevaluationprocessanditsaddedvaluetooptimizingtheprogramme. Sufficientcommunication
betweeninvolvedprofessionalsonevaluationcanfacilitateevaluation,asdoessupportforevaluation
ofICIAstogetherwithstakeholdersatastrategicandtacticallevel. Tostimulatetheevaluationof
ICIAs,werecommendsupportingprofessionalsinsecuringevaluationresources,providingtailored
training and tools to enhance evaluation competences and stimulating strategic communication
onevaluation.
Keywords: evaluation;barriers;facilitators;perceptions;casestudy;community;overweight
Int.J.Environ.Res.PublicHealth2016,13,390;doi:10.3390/ijerph13040390 www.mdpi.com/journal/ijerph
Int.J.Environ.Res.PublicHealth2016,13,390 2of19
1. Introduction
Childhoodobesityisaglobalpublichealthproblem[1,2]. In2013,23.8%oftheboysand22.6%of
thegirlsindevelopedcountrieswereaffectedbyoverweightorobesityandindevelopingcountries
thesepercentageswere12.9%and13.4%respectively[3]. Althoughtheincreaseintheprevalenceof
childhoodobesityappearstobelevellingoffinsomecountries[4,5],inmostcountriestheincrease
continues[6]. Obesityisassociatedwithanincreasedriskofseriouscomplicationssuchastype2
diabetesandcardiovasculardiseasesbutalsowithpsychosocialproblemsandareducedqualityof
life[7,8]. Moreover,childhoodobesityoftenpersistsintoadulthood[9,10].
Governmentsandcommunitiesworldwidetrytopreventobesity,startinginchildhoodoreven
beforebirth[11]. Traditionally,suchpreventionapproachesfocusedonindividualhealthdeterminants
suchasthemotivationofchildrentobephysicallyactiveonadailybasis,orthelackofknowledge
about healthy eating [12]. Unfortunately, these individually targeted prevention efforts showed
disappointingoutcomes[13,14]. Thisseemedrelatedtothefactthattheseindividualdeterminantsof
overweightandobesityweretargeted,whilethesocialandphysicalenvironmentinwhichchildren
growupremainedthesameandthereforedidnotstimulateahealthierlifestyle[15]. Expertsarenow
advocatingtheimplementationofanintegratedcommunity-wideinterventionapproach(ICIA),in
whichbothpersonalaswellasenvironmentaldeterminantsaretargeted. Suchanapproachshould
includeamixofinterventionsworkinginconjunctionwitheachotherinthesettingswherechildren
live, learn, and play (early care and education, home, school, community, health care), should be
directedtowardsmultipletargetgroupsinthecommunity,andshouldsharealong-termhealthrelated
goal. Subsequently,itshouldbeimplementedbysectorswithinandoutsidethehealthdomain[16–20].
AsuccessfulexampleofanICIAistheEPODEapproach(EPODEstandsfor“EnsemblePrévenons
l’ObésitéDesEnfants”or“TogetherLet’sPreventChildhoodObesity”),whichwasimplementedin
15countriesincludingtheNetherlands,whereitwasnamed“YouthonHealthyWeight”(InDutch:
“Jongeren op Gezond Gewicht” or the JOGG-approach) [21–24]. The strength of this originally
Frenchapproachliesinfivecriticalelements: (1)broadpoliticalcommitment(i.e.,fromhealthand
non-health sectors) and integrated (public health) policies; (2) commitment of and collaboration
betweenpublicandprivatestakeholders;(3)useofsocialmarketingtechniquesininterventiondesign;
(4)scientificevaluationand(5)integratedpathwaysofpreventionandcare. Between2009and2014,
75municipalitiesadoptedthisapproachintheNetherlands,andthisnumbercontinuestorise.Toassist
localprofessionalsinvolvedintheJOGG-approachwithdesigning,implementing,evaluatingand
theadvocacyoftheapproachtowardsstakeholdersalogicmodelwasdeveloped,baseduponthe
EPODElogicmodel[25](seeFigure1). OtherinstrumentsofferedbytheNationalCoordinationOffice
oftheJOGG-approach(JOGG-office)tosupportprofessionalsintheevaluationoftheirICIAwere
(1)anevaluationtrainingforprogrammemanagersandepidemiologists;(2)anevaluationmanual
includingmultipleinstrumentsandtoolstosupporttheevaluationprocessanddatacollectionand
(3)thepossibilityofinvolvinganevaluationexpert.
Unfortunately,earlyexperienceswiththeJOGG-approachshowthatthecriticalelement“scientific
evaluation”isoftenneglectedwithintheJOGG-approachesinmunicipalities,despitethefactthat
evaluationisrecognizedasbeingimportantincontinuouslyimprovingtheICIAandinachievinga
sustainableandmoreeffectiveprogramme[26,27]. EvaluationinthecontextofanICIAisbroaderthan
simplyasystematicassessmentoftheworthormeritofsomething.Itis“thesystematicacquisitionand
assessmentofinformationtoprovideusefulfeedbackaboutsomeobject”[28]. Hence,itrelatestothe
totalprocessofformativeandsummativeevaluation,whichalreadycommenceswhenanintervention
isinitiatedandcanbeusedtoimprovetheinterventionduringitscourse. InICIAsanevaluation
processtakesplacewithinapoliticalandorganizationalcontext,itrequiresgroupskillsandcapacities,
managementability,politicalleverageandsensitivitytotheneedsofmultiplestakeholders. Evaluation
isthereforedifferentfromresearchwhichisgroundedinexperimentalmethodsandwhosegoalitisto
createnewscientificknowledge. Aclassicalwaytodescribethisdifferencebetweenevaluationand
researchisgivenbyPatton: “researchseekstoprove,evaluationseekstoimprove... ”[29].
Int.J.Environ.Res.PublicHealth2016,13,390 3of19
Int. J. Environ. Res. Public Health 2016, 13, 390 3 of 19
INPUT OUTPUT OUTCOME
Contextual
factors
Political Support Intersectoral Changes in the
Collaboration Environment
Home
Schools
Care
n Neighbourhoods
ordination al organisatio Public Private Partnership CapCaocmitym Buuniitldy ing CHPhehiaylsdlthi cBya lNe Ahuctartiivvtioiitoyn u ↑↑r ChInilHcdWrreeeeaanilgst hwheyt i%t h a
ntral CoOffice Loc Social Marketing ChIanndgiveisd uina lthe
OGG Ce PInrteevgerantteiodn Paantdh wCaayres ANcPuAtihtvcryiitttisiiveoiicsntay ol& n AKAwntoSatiwtrkueileldnlsdeegsses
J
Scientific Guidance and Evaluation

Figure 1. JOGG model.
Figure1.JOGGmodel.
Previous studies suggest that a possible barrier to the evaluation of complex community-wide
Paprepvroioacuhsess tius dthiee sdisfuficgugletys tint hsealtecatipnog sasni balpepbroaprrriiaetre tdoestihgen eavndal muaettihoondoolofgcyo mforp elevxalucoatmiomn [u30n–i3ty3]-.w ide
Other studies indicate that the evaluation of these initiatives is hampered because evaluation is
approachesisthedifficultyinselectinganappropriatedesignandmethodologyforevaluation[30–33].
perceived differently by the various stakeholders involved [34,35] and performing such an evaluation
Other studies indicate that the evaluation of these initiatives is hampered because evaluation is
needs to compete with other priorities for scarce available resources (e.g., time, budget) [25,36,37].
perceiveddifferentlybythevariousstakeholdersinvolved[34,35]andperformingsuchanevaluation
A limited number of studies have explored barriers to and facilitators of evaluation. However, to the
needstocompetewithotherprioritiesforscarceavailableresources(e.g., time, budget)[25,36,37].
best of the authors’ knowledge no studies have involved professionals engaged in an ongoing
A limited number of studies have explored barriers to and facilitators of evaluation. However, to
ICIA [38–40].
thebestoIfnt ohredearu ttoh iomrsp’rokvneo swuplepdogrte ton othsotsue dimiepslehmavenetiinngv IoClvIAeds apcrroosfse scasisoesn, ainls-deenpgtha guenddeirnstaanndoinngg oing
ICIAa[b3o8u–4t 0b]a.rriers to and facilitators of evaluation of ICIAs is needed. This understanding can then
Iinmoprrodveer ptolainmnipnrgo vaneds uexpepcourtitotno otfh eovsaeluimatpiolne mofe tnhteisneg cIoCmIpArsehaecnrosisvse caapsperso,aicnh-edse apnthd uhenndceer mstaayn ding
aboutinbfoarrmri eprusbtloic apnodlicyfa acnildit aptroarcsticoef oenv oaplutiamtiiozantioonf IoCf ItAhes uissen oefe IdCeIAd.s. TThheisreufonrde,e wrset aenxpdlionrgedc athne then
perceptions of professionals involved in an ICIA with regards to barriers to and facilitators of the
improveplanningandexecutionofevaluationofthesecomprehensiveapproachesandhencemay
evaluation of an ongoing ICIA.
informpublicpolicyandpracticeonoptimizationoftheuseofICIAs. Therefore,weexploredthe
perceptions of professionals involved in an ICIA with regards to barriers to and facilitators of the
2. Materials and Methods
evaluationofanongoingICIA.
2.1. Design
2. MaterialsandMethods
A qualitative case study design was used to collect data in two Dutch municipalities (our cases)
2.1. Dthesaitg inmplemented the JOGG-approach. Since our goal was to explore a wide range of barriers and
facilitators we chose to combine semi-structured interviews with observations and a document
Aanqaluyaslisit aatmivoengca as ehsettuerdoygedneesoiugsn swamaspules eodf mtoucnoiclliepcatlidtiaest.a Tinhitsw doesDigunt cshupmpuorntisc itphae lietxiepslo(roautirvec ases)
thatinmaptulreem oef nthteisd stthuedyJ O[4G1]G. T-ahpe pprrooapcohsa.lS winacse soubumrigttoeadl two athset omeexdpiclaolr eethaicwali dcoemramnigtteeeo offb tahrer iVeUrs and
facilitmateodriscawl ececnhtorsee, wtohciochm jbuidngeesde mthia-ts tar uwctauivreedr oinf tmerevdieicwals ewthitihcaol baspeprrvoavtaiol nwsaasn adpapldicoacbulem feonrt tahnisa lysis
amonegxpalohreatteivroe gsteundeyo uons psaemrceppletioonfsm ofu pnriocfiepsasliiotnieasls. aTbhoiustd eevsailgunatsiounp opfo IrCtsIAths.e explorativenatureofthis
study[41]. TheproposalwassubmittedtothemedicalethicalcommitteeoftheVUmedicalcentre,
2.2. Sampling
whichjudgedthatawaiverofmedicalethicalapprovalwasapplicableforthisexplorativestudyon
perceptionTso orefcpruroitf eosusri otwnoa lcsaasebso, uwtee evmalauilaetdio an stoufdIyC pIAarst.icipation request to six eligible municipalities
that had implemented the JOGG-approach in February 2014. Inclusion criteria were: one large and
2.2. Soanmep lsimngall municipality; with preferably different initiators of the JOGG-approach, which began
implementation at least two years ago; affiliation with different Regional Public Health Services
Torecruitourtwocases,weemailedastudyparticipationrequesttosixeligiblemunicipalities
(RPHS); and no participation in another external research programme. Based on these criteria, we
thathsaeldecitmedp tlwemo menutneidcipthaelitJiOesG (hGer-eaapfpterro caacllhedin CFaseeb Aru aanrdy C20as1e4 B.)I.n clusioncriteriawere: onelargeand
one small municipality; with preferably different initiators of the JOGG-approach, which began

implementation at least two years ago; affiliation with different Regional Public Health Services
(RPHS);andnoparticipationinanotherexternalresearchprogramme. Basedonthesecriteria,we
selectedtwomunicipalities(hereaftercalledCaseAandCaseB).
Int.J.Environ.Res.PublicHealth2016,13,390 4of19
Fromthetwocases,weusedemergentsamplingtoselectaheterogeneousgroupofrespondents
basedonposition(e.g.,programmemanagers,RPHSepidemiologists,representatives,policyadvisors)
anddegreeofinvolvementwithintheJOGG-approach(e.g.,involvedindailyoperations,operating
atstrategicorattacticallevel). Emergentsamplingimpliesthatourstudypopulationemergedand
unfoldedwhilethestudyprogressed,ratherthanbeingconstructedpriortothestudy. Westarted
with purposive sampling for initial response and because it requires the knowledge of insiders to
locaterespondentsforthestudyweusedsnowballsamplingtoidentifyotherrespondents. Snowball
sampling means that we asked the first purposively sampled respondents for additional relevant
contactswhowouldbeabletoprovidedifferentorconfirmingperspectives[42,43]. Allrespondents
employedbythemunicipalitywerefromtheDepartmentofHealth.
In Case A, no prior ICIAs had been undertaken to prevent children getting affected by
overweightandobesity. Amultinationalcorporationthatheadquarteredinthemunicipalityinitiated
the JOGG-approach by convincing the municipality to implement the JOGG-approach. Case A
was a medium sized urban municipality in the Netherlands of approximately 25,000 inhabitants.
TheJOGG-approachwasimplementedinthewholemunicipality. InCaseBthemunicipalityinitiated
the implementation of the JOGG-approach as a follow-up to an existing integrated approach to
the prevention of lifestyle problems, including overweight. Case B was a somewhat larger urban
municipality in the Netherlands with approximately 55,500 inhabitants. In this municipality, a
community approach was chosen in which two neighbourhoods were involved of approximately
7400 people in total. In contrast to Case A, no other municipality that had implemented the
JOGG-approachwasaffiliatedwiththeRPHSinCaseB.
2.3. DataCollection
2.3.1. Interviews
Interviewswereconductedwithfiverespondentsineachcase(LvA).Respondentshadpositions
withinthemunicipalityandwithintheRPHSatstrategic(oradministrative),tactical(ormanagerial),
andoperational(orexecutive)level. Thedurationoftheinterviewsrangedfrom43minto67min.
AspresentedinTable1,thetotalamountofhoursavailablefortheJOGG-approachwashigherin
CaseBthanCaseA,respectively33.20hand10.45haweek.
Table1.Individualcharacteristicsoftherespondents(n=10).
WorkingTime
Age Levelof Yearsof
Respondent* Function Gender Organization forJOGG
(Years) Education Service
(Hours)**
Programme
A1 F 23 BS Municipality 2 8.00weekly
manager
A2 Policyadvisor F 28 MS RPHS 2 1.25weekly
A3 Epidemiologist F 43 PhD RPHS 4 0.24weekly
Health
A4 promotion F 27 MS RPHS 4 0.96weekly
professional
A5 Alderman M 62 - Municipality 4 -
Programme Municipality&
B1 F 48 BS 2&13 20.00weekly
manager RPHS
B2 Policyadvisor F 55 MS Municipality 14 8.00weekly
B3 Epidemiologist F 38 MS RPHS 12 4.00weekly
Epidemiologist
B4 &Policyadvisor F 37 MS RPHS 11 1.20weekly
(RPHS-employee)
B5 Alderman M 55 - Municipality 4 -
*=LetterArepresentsCaseA,letterBrepresentsCaseB;**=Calculatedonthebasisof208workabledays
(40-hourworkingweek).Abbreviationsused:BS=BachelorofScience,MS=MasterofScience,PhD=Doctor
ofPhilosophy,RPHS=RegionalPublicHealthService.
Int.J.Environ.Res.PublicHealth2016,13,390 5of19
Furthermore, the programme manager and policy advisor for Case B were on average ten
yearsolderthantheonesforCaseA.RespondentfourofCaseAisindicatedas“HealthPromotion
Professional”andrespondentfourofCaseBisindicatedas“RPHS-employee”(seeTable1). Priorto
theinterview,allrespondentssignedaninformedconsent. Interviewswereconductedwithatopic
list. This topic list was based on literature and the model of Preskill and Boyle [44]. Additionally,
threeresearchers(LvA;AdK;MvK)developedthetopiclistcollaborativelyandcarriedoutapilot
test. Thetopiclistevolvedovertimeastheinterviewprocesscontinued. Topicsfocusedontheroleof
arespondentinevaluation,evaluationcompetence(knowledge,skills,andattitude),perceptionof
theprogressofevaluation,resourcesforevaluation,organizationalstructureofthemunicipality,key
personsinevaluation,andperceivedcollaboration.
Firstly, programme managers and policy advisors of Case A and Case B were interviewed.
InCaseA,LvAalsoattendedanintroductorymeetingattacticallevelwithanewaccountmanager
employed by the JOGG-office. During this meeting the account manager asked questions of the
programmanagerandpolicyadvisortooutlinethecurrentsituationoftheJOGG-approachinthe
municipality. The responses from the interviews from Case A were supplemented with shared
information. InsightwasgainedintoorganisationalstructureandthestatusoftheJOGG-approachin
themunicipalities. Basedontheresultsofthefirstfourinterviews,complementaryinterviewswere
performedwithotherstakeholdersatCaseAandCaseB.Astheinterviewprocessprogressed,the
programmemanagerswereinterviewedagaintofurtherelaborateontheevaluationofactivitiesand
responsibilityforevaluation. Aftereachinterview,LvAwroteareporttorecordthefirstimpressions
andabriefdescriptionoftherespondentandinterviewlocation,thecourseoftheinterview,summary,
andkeywords. Therecordingsweretranscribedverbatim. Respondentsreceivedasummaryofthe
interviewinordertocheckthecredibilityofthefindings(membercheck)[45].
2.3.2. Observations
Observations were conducted during meetings at strategic programme level (e.g., steering
committee)andoperationalprogrammelevel(e.g.,workinggroups)ofboththeJOGG-approaches.
In Case A, the researcher (LvA) conducted an observation at operational project level. In Case B
two observations were conducted at the strategic project level and one at the operational project
level. Notesregardingthesetting,actions,mutualpowerrelationships,modeofinteraction,content,
decision-making, and non-verbal cues of all stakeholders were made, and the extent to which
stakeholders participated during the meetings. Afterwards, LvA continued informal discussions
with stakeholders where possible (e.g., during a car ride). In this way, she was able to strengthen
therelationshipsshehadestablishedinearlierinterviewsandwasabletoaskquestionsinamore
informalway. Aftereachdayofobservation,LvAmadecomprehensivefieldnotesofherobservations
andinformaldiscussions. Thesedetailednotesgeneratedinsightandbetterunderstandingofthedata
collectedduringtheinterviews.
2.3.3. DocumentAnalysis
Allinterviewedrespondentswereaskedtosharedocumentsoutliningtheprojectstructureand
currentsituationofthe(evaluationofthe)JOGG-approachanddocumentscontaininginformation
abouttheembeddingoftheJOGG-approachinpolitics. Avarietyofdocumentswerereviewedin
eachcase: programmeplans,minutesofmeetings(atstrategicandoperationallevel),overviewsof
resolutionsoftheMunicipalBoard,MemorandumsofPublicHealth,newsletters,andorganization
charts. Theinformationextractedfromthedocumentationwasusedintwoways: (1)asinputfor
interviewsortohighlightsituationsthatneededtobeobserved;and(2)tocross-validatedatagathered
duringtheinterviewsandobservations[46].
Int.J.Environ.Res.PublicHealth2016,13,390 6of19
2.4. DataAnalysis
Conceptsfromthemulti-disciplinarymodelofevaluationcapacitybuilding(ECB)ofPreskill
andBoyle[44]wereusedinthedevelopmentofthecodingscheme. Althoughourresearchdoesnot
focusonECBitself,exploringthebarrierstoandfacilitatorsofperformingevaluationmayincrease
insightintowhatisneededtobuildevaluationcapacityamongprofessionalsinICIAs. Thisconceptual
frameworksuggestsmultipleindividualandorganizationalresourcesthatshouldbeinplaceinorder
tobuildlocalevaluationcapacityandestablishasustainableevaluationpractice[44]. Whenthese
resourcesarenotinplaceasustainableevaluationpracticeisnotfeasible.
Theinterviewtranscriptsandobservationswereanalysedbytworesearchers(LvA,AdK)using
thematiccontentanalysis[47]. Thetranscriptswerereadthroughseveraltimes. Thetextsweredivided
intofragments,andcodes(labels)wereassignedtothesefragments(opencoding). Thesecodeswere
allorganizedintoamindmap. ThepreliminaryconclusionswerethoroughlydiscussedwithMvK.
Thelastphaseoftheanalysiswasselectivecoding: theessenceofwhateachthemewasaboutwas
identified,searchedforrelationsthroughconstantcomparisonacrosscasesandanalysedvariation
withinandbetweencases. Finally,thedifferentthemeswerefittedintothebroaderpictureoftheentire
data. Duringtheentireanalyticalprocess,memoswereusedtorecordthoughtsabouttheresultsand
todistinguishbetweentheresearchers’interpretationsandrespondents’ownviews. Toensurevalidity
andreliabilitythoughtsandanalysiswerediscussedwithintheresearchteam.
3. Results
3.1. CaseDescription
Both cases had adopted the JOGG-approach in the same year, 2012, but with different
initiatorsandpointofdeparture. InCaseA,amultinationalcorporationthatheadquarteredinthe
municipalitybroughttheJOGG-approachtotheattentionofthealderman,whointurnsupportedthe
implementationinordertomaintainagoodrelationshipwiththisprivatecompany. Nopreparations
hadbeenundertakenpriortotheJOGG-approach: themunicipalitystartedfromscratch,asreflected
bytheprogrammemanager’sperceptionofherfunction:
“Sometimes(Ihave)thefeelingof: ‘wheretostart?’ and(itfeelslike)dropletsintheocean”(A1).
InCaseB,theinitiatoroftheJOGG-approachwasthemunicipality. Aprogrammesimilartothe
JOGG-approach,calledprogrammeXinthisstudy,hadalreadybeenimplemented. ProgrammeX
wasanintegratedapproachtothepreventionoflifestyleproblems,includingoverweight,andwas
accompaniedbyfinancialandsubstantivesupport. Therefore,acertainknowledgeandexperience
basewasalreadyestablished. AccordingtotheprogrammemanagertheJOGG-approachwasusedto
amplifythepreviousprogramme.
Inbothcasesasteeringgroupandmultipleworkinggroupswereestablished. Atstrategiclevel
bothJOGG-approacheswerecoordinatedbyanaldermanandthedepartmentheadofthehealthsector
ofthemunicipality. Inbothcasesa“steeringgroup”functionedmoreontheoperationallevelthanon
thestrategiclevel. ArespondentfromCaseAdescribedthesteeringgroupashavingalackof:
“... realvision,policy,andmakingdecisions(atstrategiclevel)”(A1).
MorepeoplewerepartoftheprojectstructureinCaseBthaninCaseA.ForexampleinCaseA
eight people were part of the working group “Interventions” and in Case B there were twelve.
ThesteeringgroupofCaseAhadsevenmembersandinCaseBeleven. CaseAfocuseditsprogramme
structurearound“Interventions”and“Communication”.AndCaseBfocusseditsprogrammestructure
around“Interventions”,“Careandprevention”,and“Evaluation”.
TheinterviewedprofessionalsinCaseAwereyoungerandhadnotbeenintheemploymentof
theirorganisationsforaslongasthoseinCaseBandwerelessexperiencedwithevaluatingpublic
healthinitiatives. Moreover,workingtime(hours)fortheJOGG-approachwashigherinCaseBandin
Int.J.Environ.Res.PublicHealth2016,13,390 7of19
thiscasemoreprofessionalsfrominandoutsidethelocalgovernmentandatdifferentlevels(strategic,
tacticalandoperational)wereinvolved.
Inbothcases,noevaluationteamwasestablishedeventhoughthiswasrecommendedinthe
JOGG evaluation manual and in the first (of four) evaluation training session of the JOGG-office.
The programme manager and the epidemiologist in both cases attended this first training session.
TheprogrammemanagerofCaseBexplainedtheydidnotestablishaseparateevaluationteam,but
madeevaluationpartofexistingworkinggroups.Itappearsthiswasanon-formalisedagreementsince
allrespondentsofCaseBwereinvolvedinevaluationandhadtheirownstrategicandoperational
tasksandroles. TheprogrammemanagerofCaseAsaidtheydoubtedtheneedforsuchateam,saida
clearmandateforsuchateamhadnotbeengivenbythesteeringgroup,andtherewerenoresources
forevaluation.
3.2. EvaluationBarriersandFacilitators
IntheupcomingparagraphswedescribeourmergedresultsfromCaseAandCaseB.Perceived
barriers to and facilitators of evaluation in each case are presented using direct quotations from
interviews (indicated by italic letters and enclosed in quotation marks), field notes, observations,
and documents (enclosed in quotation marks). A total of six themes derived as key-issues after
data-collection: motivatingfactorstoevaluate,perceivedfeasibilityofevaluation,knowledgeand
attitudesofevaluation,communicationandinvolvementwithevaluation,evaluationresources,and
supportfromdecision-makers. AnoverviewoftheresultscanbefoundinTable2.
Table2.BarrierstoandfacilitatorsoftheevaluationoftheJOGG-approach.
Barriers(´)/
Themes Subthemes Examples
Facilitators(+)
Theneedforanevaluationmotive
Anevaluationexpertwhoprovidesexpertiseandsupportto
Persontomotivate +
Motivating starttheevaluationprocess
evaluation
factorsto
´ Lackofapersonwhostimulatesperformanceofevaluation
evaluate
Demandfor + Externalfunderoraldermantoaskforevaluationresults
evaluation
Theprogrammemanagerdoesnotgiveacommandtostart
´
theevaluationprocess
Evaluationfeasibility
Assumptionson + Existenceofarealisticperceptionofevaluation
Perceived feasibilityof
feasibilityof evaluation ´ Negativeperceptionsonfeasibilityofevaluationaspresented
evaluation intheoryandevaluationmodels
Capabilitiesof + Trustininterpretationoftasks
thoseinvolved
Lackoftrustincapabilitiesofthosethatshouldbeinvolvedin
´
theevaluation:programmemanagerandepidemiologist.
Positiveattitude
+ Evaluationisregardedasimportant
towardsevaluation
Knowledgeand ´ DoubtaboutpossibilitiestoshoweffectsofICIA
attitudeson
evaluation Knowledgeon Availabilityofapersonwithsufficientknowledgeonwhatthe
+ (processof)programmeevaluationimpliesandhowto
evaluation
conductsuchanevaluation
Lackofevaluationknowledge(i.e.,oralinformalprocess
´
evaluationhasnoscientificvalue)
Perceptionofown
+ Highself-efficacytoconductevaluation
capabilities
´ Negativeperceptionofone’sowncapabilitiestoevaluate
Int.J.Environ.Res.PublicHealth2016,13,390 8of19
Table2.Cont.
Barriers(´)/
Themes Subthemes Examples
Facilitators(+)
Evaluationfeasibility
Financialresources + Allocatedfinancialresourcesforevaluationprocess
´ Limitedresourcestohirepersonnel,resultinginlimitedtime
Evaluation
resources Time + Allocatedhoursforinvolvementofepidemiologistor
evaluationexpert
Lackofpersonnelfordatacollection;Lackoftimefor
´
evaluationeducation
Availabilityof
Availabilityofgenericsuitableevaluationinstruments
evaluation +
(i.e.,questionnaires,logicmodel)
instruments
´ Non-functioningorincompleteevaluationinstruments
Commitmentofevaluationstakeholders
Regularandhighdegreeofcommunicationbetween
Communicationon +
Communication programmemanagerandepidemiologistonevaluation
evaluation
andinvolvement
Lowdegreeofcommunicationonevaluationbetween
withevaluation
´ programmemanagerandepidemiologist;Lowdegreeof
communicationbetweenprogrammemanagerandalderman
Activeinvolvementofstakeholdershelpstoseetheadded
Involvementin +
valueofevaluation
evaluation
Involvementofmembersoftheprogrammestructureat
´
strategicaswellastacticalandoperationallevel
Evaluationisconsideredimportantbyalderman
Politicalsupport +
andcitycouncil
(tactical)
Supportform
Competingthemesthatreduceattentionandmakefewer
decision-makers
´ resourcesavailable;Politiciansonlyinterestedinlong-term
atmultiplelevels
goalsandnotinmid-termorprocessevaluations
Apro-activeattitudeofdepartmentmanagementtogenerate
Managerial + resourcesandsupport;aclearpolicyvisionofRPHS
support(strategic) supportivetotheapproachandevaluation
Atimeconsumingpolicyprocesstogenerate
´
extrafinancialresources
Supportfrom + Stakeholdershaveacommongoal
implementers
Limitedinterestinevaluationfromthosewho
(operational) ´
implementtheICIA
3.2.1. MotivatingFactorstoEvaluate
Analysis of data showed that a trigger or motivating reason could give rise to an evaluation
(CaseB).Ontheotherhand,thelackofitcouldimpedeevaluation(CaseA).Allintervieweesasked
saidaspecificpersonwasneededtomotivateevaluation. Inparticulartheprogrammemanagerwas
seenashavinganidealpositiontomobiliseandmotivateprofessionals:
“Theprogrammemanagerisanenthusiasticpersonwhocantransferherenergyproperly. Sothat
worksverywell”(B4).
The programme manager mentioned (Case B) also seemed to understand her role to trigger
evaluationprocessessincesheaskedforevaluation-relatedactivities:
“Myrole(asprogrammemanager)istoquestion(..) them(thestakeholders)aboutthebenefitof
theiractivities. Sointhatwaywetrytocollectdata(aboutevaluation)”(B1).
InCaseB,thealdermanalsoprovidedsuchatriggerbyaskingforaneffectevaluation. Moreover,
theprogrammemanagerandpolicyadvisorinCaseBsaidanexternalpublicfunderofprogrammeX
providedanadditionaltriggerbyrequestingaprocessevaluationinafinalreportforaccountability
reasons. AlsotheevaluationexpertwhowasmadeavailabletomunicipalitiesbytheJOGG-office
Int.J.Environ.Res.PublicHealth2016,13,390 9of19
was a catalyst in the design and execution of evaluation and helped others to understand the
evaluationprocess.
“( ... )itwasgoodtohavetheevaluationexpertfromtheJOGG-officevisitusanddiscussthe
evaluationpossibilitiesandeffectstobeshownatlocallevel(...),thisworksbetterthanthetraining
sessionsalone,whicharemoregeneral”(B1).
However,thisprogrammemanagerwouldhavelikedmoredirectivesonevaluationfromthe
JOGG-office. In contrast, interviewees in Case A said no-one triggered evaluation. To stimulate
evaluation,thepolicyadvisorandhealthpromotionprofessionalfromCaseAthereforesuggestedthat
anepidemiologistfromtheRPHS,theJOGG-office,aprogrammemanagerorthealdermanshouldask
forevaluation. Theevaluationtrainingsessionswerenotperceivedasstimulatingtheprogramme
managerofCaseAtocommenceevaluationplanning:
“(... )thetrainingsessionsweretooformal,extensive,detailorientedandexaggerated”(A1).
The given information was too overwhelming for her and did not increase her motivation to
studytheprocessofevaluationletalonestimulatehertodesignandperformtheevaluationofthe
JOGG-approach.
3.2.2. PerceivedFeasibilityofEvaluation
Interviewees in both cases perceived evaluation of the JOGG-approach as unfeasible.
Theyassumedthatasoundandproperevaluationcouldnotbeperformedwiththeirlimitedbudget,
time and capabilities. For example, the programme manager of Case A and policy advisor of
CaseBsaid:
“There is so much to it (evaluation) if you want to do it well. A lot of time, money (..), and
expertise”(A1).
“Ifyouwanttodoit(evaluation)properly,youhavetomakeabiginvestment”(B2).
Mostintervieweesaddedthattheevaluationaspresentedintheevaluationtrainingandinthe
evaluationmanualoftheJOGG-officerepresentedan“ideal”evaluationwhichwasnotencounteredin
practiceorwasevenincompatiblewithpractice. Someintervieweesstruggledwiththetranslationof
theevaluationmanualforuseintheirownsituation. BoththeepidemiologistandtheRPHS-employee
of Case B explained that evaluation of the JOGG-approach was not systematic, but “emerged”
duringimplementation:
“Idonoticethatpracticeismorestubborn(thantheidealsituation),sothenyou’llhavetolook
furtherandsearchforyourownpathway”(B3).
“Youwanttowritethingsdownasspecificallyaspossibleinsuchan(evaluation)plan. However,
JOGGisalsocharacterizedby(..) abottom-upapproachandsettinggoalstogether. Andsometimes
Iperceivethatasquiteatension”(B4).
3.2.3. KnowledgeofandAttitudestoEvaluation
Inbothcasestheprogrammemanagersindicatedtheylackedsufficientknowledgeofevaluation.
The epidemiologists were believed to be sufficiently equipped to design and perform evaluation.
However,thepolicyadvisorinCaseAsaidthatsomeepidemiologistswhowerebelievedtobecapable
ofdesigningandconductinganevaluationwereinrealitynotabletodoso. Thismightbeexplained
bythefactthatmostinterviewees,includingtheepidemiologists,didnotunderstandthedifference
betweenevaluationandresearch,theyusedthetermsinterchangeably,hencemakingevaluationbigger
andmoredifficultthannecessary. Forinstance,theprogrammemanagerandpolicyadvisorofCaseA
frequentlyusedthetermsevaluationandresearchinterchangeablyandreferredtotheworkinggroup
Int.J.Environ.Res.PublicHealth2016,13,390 10of19
“Research”whentheintervieweraskedaboutevaluation. Askingabouthercompetencesinevaluation,
theprogrammemanagerofCaseAreplied:
“Ididsomeresearchduringmystudybutnotlikeayouthmonitororsomething. Iwouldnotknow
whatquestionstoaskorwhensucharesearchisrepresentativeorreliable(... )”(A1).
Moreover,intervieweesinCaseAdidnotperceiveaprocessevaluationintheformofanoral
communicationofanactivitytodiscusstheprogressandpossibleareasofimprovement,aspartof
theprogrammeevaluation,becauseitwasnotformalenough. InCaseB,theprogrammemanager,
epidemiologist,andRPHS-employeeknewthatprocessevaluationwasimportanttooptimizethe
approach,buttheirattitudestotheprocessevaluationwerelesspositive:
“ithasnoscientificvalue”(B2)and“itisnotsomethingofficial”(B4).
Theepidemiologistfeltthatsheandtheotherprofessionalswerenotfullyequippedtoconducta
processevaluation. Additionally,theprogrammemanagerandRPHS-employeeofCaseBshowed
theywereawareofdifferencesinevaluationbetweendifferentinitiatives,theysaidevaluationcould
eitherbeextensiveorsuperficialdependingonneedsandfunding.
Incontrasttothedifferentpurposesofevaluationandresearchasdiscussedintheintroduction
tothispapertheprogrammemanager,policyadvisor,andhealthpromotionprofessionalofCaseA
allmentionedthatevaluationfindingscouldbeusedto“prove”something. Theseperceptionson
evaluation can limit the use of evaluation for programme optimization and diminish the value of
evaluationatstrategicandtacticallevels.
Regardingrespondents’attitudestowardsevaluation,somerespondentsquestionedwhetherthe
effectsobservedcouldbeallocatedtotheJOGG-approach. Thepolicyadvisorandepidemiologistof
CaseB,forexample,explaineditwouldbedifficulttoproduce“hard”dataabouttherelationship
between the JOGG-approach and effects. Furthermore, several respondents cited that evaluation
requiredalotofinvestmentofresources. Onerespondentcommented:
“Anditisthequestionwhetherit(theevaluation)isworththeinvestment”(B2).
Evaluationingeneralwasconsideredimportantbythemajorityoftherespondents:
“Whenyouevaluate,youknowwhatworksandwhatdoesnot(..). Andyoucankeeppartnerson
board,showresults,(create)apositiveatmosphere,andattractotherparties”(A1).
“Itisimportanttoseewhetherwhatyoudoachieveseffects”(B2).
3.2.4. EvaluationResources
Datarevealedthatlackoftimeandallocatedfinancialresourcesprohibitstheevaluationofthe
JOGG-approach(CaseA).OnmultipleoccasionstheprogrammemanagerofCaseAstatedthatlimited
financialresourceswereavailabletoadministerquestionnaires,tohirepersonnelfordatacollection
andanalysis,andtosetupaproperevaluation. Inthismunicipalityabudgetforevaluationofthe
JOGG-approachwasnotgenerated. SincetheRPHSmonitorshealthandbehaviourofchildreninthe
municipalityitwasdecidedtousethisdatatoshowresults,despitethelimitationsofthisdata. In
CaseB,ontheotherhand,allocatedtimeandfinancialresourceswerebelievedtohavestimulatedthe
evaluationprocess:
“Within the entire process we have allocated time and money in order to conduct a proper
evaluation”(B2).
Interviewees in Case A reported that the annual fee of the JOGG-approach was paid by the
multinationalheadquarteredinthemunicipality,whileinCaseB,theannualfeewasfundedbythe
subsidyforprogrammeXandreservedresourcesfromlocalhealthpolicy. Concerningtime(personnel
resources), hourswereallocatedforanepidemiologistinCaseB,whereasnoadditionalhoursfor