Data-Driven Policy Impact Evaluation: How Access to Microdata is Transforming Policy Design

344 Pages·2019·6.78 MB·English

by  Nuno Crato

Nuno Crato
Paolo Paruolo
Editors
Data-Driven
Policy Impact
Evaluation
How Access to
Microdata is Transforming
Policy Design
Data-Driven Policy Impact Evaluation
Nuno Crato • Paolo Paruolo
Editors
Data-Driven Policy Impact
Evaluation
How Access to Microdata is Transforming
Policy Design
Editors
NunoCrato PaoloParuolo
UniversityofLisbon JointResearchCentre
Lisbon,Portugal Ispra,Italy
ISBN978-3-319-78460-1 ISBN978-3-319-78461-8 (eBook)
https://doi.org/10.1007/978-3-319-78461-8
LibraryofCongressControlNumber:2018954896
©TheEditor(s)(ifapplicable)andTheAuthor(s)2019.Thisbookisanopenaccesspublication.
Open Access This bookis licensed under the terms of the Creative Commons Attribution 4.0Inter-
nationalLicense(http://creativecommons.org/licenses/by/4.0/), whichpermitsuse,sharing,adaptation,
distribution andreproduction inanymediumorformat,aslong asyougive appropriate credit tothe
originalauthor(s)andthesource,providealinktotheCreativeCommonslicenseandindicateifchanges
weremade.
Theimages or other third party material in this book are included in the book’s Creative Commons
license,unlessindicatedotherwiseinacreditlinetothematerial.Ifmaterialisnotincludedinthebook’s
CreativeCommonslicenseandyourintendeduseisnotpermittedbystatutoryregulationorexceedsthe
permitteduse,youwillneedtoobtainpermissiondirectlyfromthecopyrightholder.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Policymakingisundergoingprofoundtransformations,thankstotheavailabilityof
betterdataandthepossibilitytouncovercausaleffectsbyusingimprovedstatistical
methods(namely,counterfactualimpactevaluationtechniques).Thisbookaimsto
document these changes in a simple language, from a European perspective. The
centralideasofthebookcanbesummarisedinfourparagraphs.
Firstly, statistical methods now exist to rigorously evaluate the impact of
policy measures, even when data do not come from controlled experiments.
Micro-econometriciansandstatisticiansspecialisinginthesecounterfactualimpact
evaluationmethodsarewellawareofboththeirpotentialandlimitations.Thescope
for rigorous causal analysis that they offer is, however, not widely understood.
Owing to the inherentcomplexityof society and the many concurrentfactorsthat
influence an outcome, decision-makers often doubt that it is possible to uncover
clearcausalrelationships.
Secondly,toevaluatepolicyimpacts,itiscrucialtohavedataonthebasicunits
targetedbyapolicy,i.e.theso-calledtargetgroup.Similardataneedtobeavailable
fora comparableset ofunits,called thecontrolgroup.Thisoftentranslatesinto a
needtoaccessmicrodata,i.e.dataatthelevelofindividuals,households,businesses
orcommunities.
Thirdly,microdataareavailablefromseveralsources;onesuchsourcethatoffers
numerousadvantagesisadministrativedata(orregistrydata),i.e.datacollectedby
publicentitiesfortheirserviceactivities.Datafromdifferentregistriescanbelinked
together,andalsolinkedwithexternalinformation,suchasdatafromsurveysordata
collected by private businesses, to obtain comprehensivedatasets that are suitable
forpolicyresearch.Ifthesedatasetsareproperlyorganised,policyevaluationscan
beperformedinrealtime.
Fourthly, the use of microdata faces challenges: some real, others imaginary.
Thereareissueswithdatabasequality,linkageandpreservationofanonymity.Most
obstacles can be overcome with appropriate organisation and modern computer
sciencemethods.Therearealso problemsofapoliticalnature;toovercomethem,
one needs to be aware of all the technical solutions that can be used to keep data
v
vi Preface
safe,aswellastobewillingtosubjectpoliciestorealitychecksandtolearnfrom
thosechecks.
To properly analyse these topics from all necessary angles, this book collects
contributionsfromagroupofresearchers,practitionersandstatisticalofficerswho
workin the field, mostly basedin Europe:11 Europeancountriesare represented.
The pool of contributors is complemented by experts from the Organisation for
Economic Co-operation and Development and from Eurostat, who report cases
drawnfromtheirorganisations’particularexperiences.
The various articles in this book consider different policy areas. They include
employment, health, professional training, social security and educational issues.
Many of the contributions explain and apply various counterfactual econometric
methods,servingasausefulreferencefortheirstudyanduse.
Thebookprovidesapanoramaofmicrodataissuesrelatingtopolicyresearch,of
administrativedata availability, of variousexisting systems that can facilitate safe
datause,ofsuccessfulstudiesonpolicyimpactandevenofpolicychangesmadeon
thebasisofsuchstudies.WehopeitwillbeusefultoalargereadershipinEurope
andintherestoftheworld.
Policymakers and public policy scholars will find here various examples of
successfulpolicyevaluation.Theywillfindargumentsinfavourofdata-basedpolicy
studiesandaclearcaseforimprovingtheeffectivenessofpolicymeasures.
Public administrators and technical staff at public administrations will find
systematicexamplesshowingthatpolicyevaluationisaviabletask,withdedicated
methodsandsophisticatedtechniques.Variouschaptersshowin detailthe reasons
whya causalevaluationispossibleandtheconditionsunderwhichtheseanalyses
canprovideconclusiveevidenceontheeffectsofpolicies.
Statisticiansandeconometricianswillfindvariousdiscussionsontheapplicabil-
ityofcounterfactualimpactevaluationmethodsanddetailedcasestudiesthatshow
howvarioustechniquescanbeappliedwithsuccess.Allreaderswillfindpractical
examplesofthemostcommonlyusedofthesetechniques,alongwithadiscussion
of their applicability. Statistical officers and database experts will find a state-of-
the-artreviewofanonymisationissuesandtechniquesandofdatabaselinkageand
security.
To sumup:Microdataexistandcanbesafelyorganisedfora betterknowledge
of society and for policy research; data-based policy assessment, monitoring,
evaluation and improvementare feasible and desirable, and they can be achieved
inunprecedentedlyfastwaysbyproperlylinkingadministrativedata.
The first chapter introduces these issues. The remaining contributions to this
bookaregroupedintofourparts,whicharedescribedbelow.
Microdata for PolicyResearch
Thefirstpartofthisbookdealswithdataissues.PaulJacksonprovidesanupdated
accountoftherecenthistoryoftheuseofmicrodata.Itisonlyrecently,aroundthe
Preface vii
beginningofthetwenty-firstcentury,thatthevalueofmicrodataandadministrative
datastartedtobecomefullyappreciated.Infact,usingmicrodatafromadministra-
tive sources for policy research is still innovative.Previously,microdata collected
forgeneralstatisticswereonlyusedtocomputesummarystatisticssuchasaverage
income,orthenumberofunemployedpeopleinaparticularregion.Originalrecords
weretreatedascompletelyconfidentialandthendiscarded.
Now, microdata are understood to have enormous potential for enabling us to
improveknowledgeaboutpolicies,andtoalargeextenttheyareinexpensive,asthey
havebeenalreadycollected.Officialstatisticalauthorities,bothatnationalleveland
in Eurostat,makemicrodataavailablefor research.Publicadministrationssuchas
unemploymentagenciesandeducationalregistersare increasinglyurgedbysocial
researchers, political parties and independent organisations to make available the
datathattheycollectregularlyand,nowadays,keepindigitalformat.
Ugo Trivellato explains in what sense microdata should be seen as a public
goodandmakesthecasefordataavailability.Hehighlightstheimportanceofdata
release, bothforresearch andfor democraticparticipation.It is a matter of public
interest that data should be made available for a better knowledge of society and
fortheevaluationofpublicpolicies.Inaddition,hereviewssomerecentadvances
in regulations and practices on microdata access—at the level of EU and several
memberstates,andattransnationallevel—andconcludesthatremotedataaccessis
themosteffectivemodeforsafelysharinghighlyinformativeconfidentialdata.
The full use of administrative data requires the linkage of various databases.
Natalie Shlomo explains how modern statistical techniques can be applied to
data linkage and how probabilistic linkage can bypass some data faults and still
provideperfectlyusabledatasets.Shealsoshowshowprobabilisticlinkagerequires
appropriate statistical techniques account for linkage errors when carrying out
statisticalmodellinginlinkeddata.
There are, however, concerns about safeguards around microdata. Some are
imaginary and may be used as pretexts to keep data out of reach and unused for
policy research and public information. Some are real: confidentiality is a major
one.GiovanniLivragaexplainshowmoderncomputersciencetechniquesareable
toanonymisedatasetsandstillprovidetherelevantinformationforsocialresearch.
Microdata Access
Somecountriesandinstitutionsareworldleadersintheuseofadministrativedata.
Theyhavealreadyorganisedplatformsandsystemstomakemicrodataavailablein
asystematicandsafeway.Itisbothreassuringandinspiringtolearnhowthishas
beenachieved.
Eurostatisageneralstatisticalproviderandasupplierofmicrodataforresearch.
Aleksandra Bujnowska describeshow Eurostat serves as an entry pointfor safely
accessingmicrodataprovidedbynationalstatisticalofficesintheEuropeanUnion.
viii Preface
Inprovidingaccesstomicrodata,theInstituteforEmploymentResearch(IAB)
in Germany is one of the most successful institutions in Europe. Dana Müller
andJoachimMöllerdescribeindetailhowtheIABprocessessocialsecuritydata,
linking it to survey data and organising safe access for researchers to this wealth
ofinformation.Thenumbersofusersandofstudiesstemmingfromthisaccessare
growingsteadily:in2016,thereweremorethan1000usersand500projects.
In many European countries, access to administrative data is hindered by
outdated legislation, by security issues and by confidentiality concerns. It is very
interesting to see how Hungary has entered the twenty-first century debating a
changeinlegislationaboutlinkageandaccesstomicrodata,andhowthishasbeen
successfully implemented. Ágota Scharle, who then headed the Finance Ministry
effortsto change the laws and the practices, explains how successful negotiations
ledtoaveryopenandmodernsystemofaccessingadministrativedata.
Other inspiring examples come from the Netherlands. Marcel Das and Marike
Knoefdiscussaninnovativeinfrastructure,theLISSpanel,whichhelpsresearchers
to conductfield surveysand offersthe possibility of integratingthe surveyresults
withexistingadministrativedata.
Counterfactual Studies
Counterfactualimpactevaluationsstemmedoriginallyfromlabourandeducational
economics;severalcontributionstothisbookcomefromthesefields.Inthefirstof
these,PedroMartinsdescribesthe design,implementationandevaluationofa job
searchsupportandmonitoringprogrammethathasbeeninplaceinPortugalsince
2012,usingaregressiondiscontinuitydesignonadministrativedata.
AnotherevaluationofalabourmarketinterventionispresentedbyEnricoRettore
and Ugo Trivellato, who analyse the Italian public programme called ‘Liste di
Mobilità’,whichhandlescollectiveredundanciesintheVenetoregion.Theyusea
crescendoofregressiondiscontinuitydesigntechniques,followinganincreaseinthe
availabilityofadministrativesourcesonthisprogrammeoverthecourseof15years.
Theyemphasisethattheadministrativedatahadbeentherefromthestart,showing
how advancesin policy research are linked to increasedtrust between researchers
andpublicadministrators.
Homeownershipanddebtonhousehaveimplicationsforjobmobility.Andrea
Morescalchi, Sander van Veldhuizen, Bart Voogt and Benedikt Vogt present an
analysisoftheimpactofnegativehomeequityonjobmobilityintheNetherlands,
usingaDutchadministrativepanelfortheperiod2006–2011.Theyusepanelfixed
effects and find that negative home equity has a moderate negative effect on the
probabilityofchangingjobs.
Numerous programmesare active in any given country at any given time, and
their evaluation is not simple. For the Netherlands, Rudy Douven, Laura van
Geest, Sander Gerritsen, Egbert Jongen and Arjan Lejour present some of the
counterfactualworkbeingperformedattheCPBNetherlandsBureauforEconomic
Preface ix
PolicyAnalysis.Theyprovidefourexamples:thefirstconcernsthelabourpartici-
pationofparentswith youngchildren,the seconddealswith tax shiftingby small
corporations, the third evaluates teacher quality and student achievement and the
fourth analyses performance-based pay in mental health care. In these examples,
CPB combines the strengths of structural models and of policy impact evaluation
methods,includingdifferences-in-differences,regressiondiscontinuityandrandom
assignment with large datasets. Furthermore, they emphasize the importance of
goodcommunicationstrategiesoftheresultstopolicy-makers.
For Denmark, Jacob Nielsen Arendt and Mette Verner present a study on the
long-term effects of a social intervention for young people with conduct disorder
problems. Register data enable the authors to construct a comparison group of
youngpeoplewhoarereceivingalternativesocialtreatmentsbutwhohavesimilar
parental characteristics and a similar life-cycle profile in terms of previous social
interventionsandhealthcareuse.Usingpropensityscorematching,theauthorsfind
thatparticipantsare morelikely than similar youngpeopleto take primaryschool
exams,buttheyhavelowerupper-secondaryeducationcompletionratesandlower
employment rates; additionally, they are more dependenton welfare income, and
theyaremoreoftenconvictedofcrimes.
ForItaly,ClaudioDeianaandGianlucaMazzarellainvestigatethecausaleffectof
retirementdecisionsonwell-being.Theyexploittheexogenousvariationprovided
bychangesintheeligibilitycriteriaforpensionsthatwereenactedinItalyin1995
andin1997tocomputeaninstrumentalvariableestimateofthecausaleffect.They
findasizeableandpositiveimpactofretirementdecisiononsatisfactionwithleisure
timeandonfrequencyofmeetingswithfriends.Theirresultsarebasedonamixof
surveyandadministrativedata.
A final study on the economics of education in Chile is provided by Julio
Cáceres-DelpianoandEugenioGiolito,whoinvestigatetheimpactofageofschool
entryonacademicprogressionforchildreninChile,usingaregressiondiscontinuity
design.Thanksto theuse of a verydetailedadministrativedatabase,they areable
tofindthatahigherageatentryhasapositiveeffectongradepointaverageandon
thelikelihoodofpassingagrade,althoughthisimpacttendstowearoffovertime.
Childrenwhoseschoolentryisdelayedarealsomorelikelytofollowanacademic
trackatsecondarylevel.
Useof Results
Policyimpactevaluationcreatesknowledgeofwhatworked,forwhomandwhen.
This information can be organisedby policy areas to summarise the state of play
in the field. Béatrice d’Hombres and Giulia Santangelo present the state of play
on counterfactual evidence for active labour market policies in Europe. Matej
BajgarandChiaraCriscuoloexplainhowtheimpactsofamajorvocationaltraining
programmecanbeevaluatedwiththeuseoflinkedadministrativeandsurveydata,
drawingontheexampleoftheModernApprenticeshipsinScotland.
x Preface
In a different area, Rita Santos, Sara Barsanti and Chiara Seghieri discuss the
useofadministrativedatainthehealthsectortoevaluatetheimpactofprimarycare
pay-for-performanceschemesinEnglandandinItaly.
The final chapter, by Sven Langedijk, Ian Vollbracht and Paolo Paruolo, looks
at the future of microdata access across Europe and lays out the benefits and
issuesinvolvedinincreasesintheuseofadministrativedatabothwithinandacross
Europeancountries.
FinalRemarks
TheideaofthebookwasconceivedattheCompetenceCentreforMicroeconomic
Evaluation (CC-ME) in the European Commission Joint Research Centre (JRC)
in Ispra, Varese, Italy. CC-ME is a centre for research that supportsthe European
CommissionandEuropeanMemberStatesonimpactevaluationandpromotesthe
useofcausalimpactevaluationmethods.
Manychaptersandaspectsofthe bookstem fromtheworkofCC-ME:several
authorsarepartofit,andmostoftheotherauthorshavecooperatedinonewayor
anotherwiththeCentre’sactivities.Theeditorsareindebtedtoallauthorsfortheir
wholeheartedcollaboration.
Special thanks also go to other CC-ME researchers and colleagues at the JRC
whoactedasreviewersandprovidedsuggestionsandtookpartindiscussionsabout
thebook.Inparticular,thanksgotoMassimilianoBratti, MaurizioConti,Claudio
Deiana,LeandroElia,SophieGuthmuller,CorinnaGhirelli,MassimilianoFerraresi,
EnkelejdaHavari,AthanasiosLapatinas,GianlucaMazzarella,AndreaMorescalchi,
GiuliaSantangelo,SylkeSchnepfandStefanoVerzillo.
Finally, the project owes a great deal to the unwavering support of Sven
Langedijk, the head of the Modelling, Indicators and Impact Evaluation Unit,
to which CC-ME belongs. Many thanks go to the European Commission Joint
ResearchCentreforfinanciallysupportingopenaccesstothebook.
Ispra,Italy NunoCrato
August2018 PaoloParuolo