Table Of ContentJ HEALTH POPUL NUTR 2013 Dec;31(4) Suppl 2:S23-S35 ©INTERNATIONAL CENTRE FOR DIARRHOEAL
ISSN 1606-0997 | $ 5.00+0.20 DISEASE RESEARCH, BANGLADESH
Evidence Acquisition and Evaluation for
Evidence Summit on Enhancing Provision
and Use of Maternal Health Services
through Financial Incentives
Elizabeth S. Higgs1,2, Emily Stammer3, Rebecca Roth2, Robert L. Balster2,4
1National Institute of Allergy and Infectious Diseases, Bethesda, MD, USA; 2United States Agency for International
Development, Washington, DC, USA; 3Knowledge Management Services, Washington, DC, USA;
4Virginia Commonwealth University, Richmond, VA, USA
ABSTRACT
Recognizing the need for evidence to inform US Government and governments of the low- and middle-
income countries on efficient, effective maternal health policies, strategies, and programmes, the US Gov-
ernment convened the Evidence Summit on Enhancing Provision and Use of Maternal Health Services
through Financial Incentives in April 2012 in Washington, DC, USA. This paper summarizes the back-
ground and methods for the acquisition and evaluation of the evidence used for achieving the goals of the
Summit. The goal of the Summit was to obtain multidisciplinary expert review of literature to inform both
US Government and governments of the low- and middle-income countries on evidence-informed prac-
tice, policies, and strategies for financial incentives. Several steps were undertaken to define the tasks for the
Summit and identify the appropriate evidence for review. The process began by identifying focal questions
intended to inform governments of the low-and middle-income countries and the US Government about
the efficacy of supply- and demand-side financial incentives for enhanced provision and use of quality
maternal health services. Experts were selected representing the research and programme communities,
academia, relevant non-governmental organizations, and government agencies and were assembled into
Evidence Review Teams. This was followed by a systematic process to gather relevant peer-reviewed litera-
ture that would inform the focal questions. Members of the Evidence Review Teams were invited to add
relevant papers not identified in the initial literature review to complete the bibliography. The Evidence
Review Teams were asked to comply with a specific evaluation framework for recommendations on prac-
tice and policy based on both expert opinion and the quality of the data. Details of the search processes and
methods used for screening and quality reviews are described.
Key words: Incentive; Maternal health services; Maternal mortality; Motivation; Perinatal mortality; Pre-
natal care; Reimbursement
INTRODUCTION vast inequity as well as the possibility of lowering
preventable maternal mortality to a great extent.
Over their lifetime, women are about 100 times
Yet, women in low- and middle-income coun-
more likely to die as a result of pregnancy in sub-
tries (LMICs) with high burden of maternal death
Saharan Africa than in developed regions of the
often underutilize the maternal health services
world (1). These enormous discrepancies highlight
intended to facilitate healthy births and protect
maternal lives. Creating demand for lifesaving
Correspondence and reprint requests:
Dr. Elizabeth S. Higgs services is a challenge shared across many areas
Division of Clinical Research of global health but it is particularly compelling
National Institute of Allergy and Infectious Diseases in maternal health, given the high mortality rates
National Institutes of Health, DHHS
among mothers and the children they leave be-
Bethesda, MD
hind. Financial barriers contribute to both unde-
USA
Email: [email protected] rutilization and lack of availability of potentially
Fax: +301-435-6739 lifesaving services.
Evidence summit on maternal health and financial incentives Higgs ES et al.
A renewed emphasis on the application of research tise and contributors across the US Government,
and evaluation to inform strategic thinking about academia, and policy- and practice-related leader-
development for LMICs is integral to the US Gov- ships from developing countries. The Eunice Ken-
ernment’s efforts to improve health by promoting nedy Shriver National Institute of Child Health and
country-owned, effective and sustainable interven- Human Development at the National Institutes of
tions. To that end the US Agency for International Health joined USAID in the organization and sup-
Development (USAID) is leading a series of evi- port of this Evidence Summit. This paper describes
dence summits focused on important development the Evidence Summit process from its inception
challenges. The aim of these summits is to provide to the post-summit activities. Other papers in this
evidence-based expert recommendations on how Supplement of the Journal present the findings and
to achieve some of the world’s most difficult devel- recommendations.
opment goals, for example, caring for children liv-
Overview of process of the Evidence Summit
ing outside families (2) and supporting community
health workers (3).
The initial planning for the Summit originated in
To address the problem of financial barriers to ma- the Bureau for Global Health at USAID and was
ternal healthcare utilization, the US Government tied to President Obama’s Global Health Initiative
convened the Evidence Summit on Enhanc- (http://www.ghi.gov/), which calls for interagen-
ing Provision and Use of Maternal Health Services cy collaboration, innovation, and doing more of
through Financial Incentives on 24-25 April 2012 what works as guiding principles for US Govern-
in Washington, DC. The Summit brought together ment’s assistance in global health. Like other evi-
leading researchers, development experts, and those dence summits of the US Government, this Evi-
involved in implementing programmes in the field dence Summit was not a single event, but rather
to assess the evidence that will ultimately inform a year-long process that led up to the April 2012
policies, strategies, and programmes relevant to Summit and continues with the implementation
financial incentives and maternal health services of recommendations developed during the Sum-
in LMICs and, in so doing, simultaneously iden- mit process (Figure 1). Following the identifica-
tify gaps in evidence that could shape the future tion of the general topic of creating demand for
research agenda. This Evidence Summit reflects maternal health services as a potential topic for
USAID’s commitment to evidence-based, innova- the Evidence Summit, a Core Group of persons
tive, efficacious, effective and sustainable develop- responsible for the overall organization and direc-
ment efforts of the US Government in partnership tion of the Summit was assembled with 19 experts
with other governments. The rapid application of in maternal health, economics, global health,
knowledge and scale-up of novel discoveries and research, and implementation drawn from three
innovations to populations needing them most re- US Government agencies with ongoing activities
quires a continuum of learning from basic to op- in support of maternal health (http://www.usaid.
erational research and a broad coalition of exper- gov/sites/default/files/documents/1864/mh_sum-
Figure 1. Diagram of the Evidence Summit process from initial organization to activities, still
ongoing, to implement the recommendations
External Peer Review Pre-summit: Call for ERTs: Evidence ERTs Revise Disseminate
Consultation Evidence Charge to Evidence Evidence Summit: and Finalize Findings and
Develops Acquisition Evidence from Synthesis & Feedback on Papers Implement
Focal Review Experts: Recommen- Evidence & Evidence to
Questions Teams Grey & Peer- dations Recommen- Action
30 August reviewed dations Strategy
2011
24 JHPN
Evidence summit on maternal health and financial incentives Higgs ES et al.
mit_interagency_planning_group.pdf). The for- 6. The evidence on the topic can be pooled, syn-
mation of the Core Group was followed by a scop- thesized, shared, and discussed at reasonable
ing exercise consisting of an external consultation, cost; and
which included experts in maternal health, eco- 7. Evidence-based guidance is not available on the
nomics, and implementation to review the topic topic or needs to be updated.
and identify key questions. Leveraging this exter-
The underutilization of maternal health services
nal advice, the Core Group created focal questions
was identified as a general topic for an evidence
to guide the evidence review process, the selection
summit. Additional information on the scoping
and appointment of external and internal experts
exercise and rationale for the topic can be found
to Evidence Review Teams (ERTs), a systematic lit-
in the introductory paper to this Supplement (4).
erature search for documents relevant to the fo-
Recognizing the need to refine and narrow this
cal questions, and the screening and evaluation of
topic further, the Core Group organized a scop-
these documents and the identification of addi-
ing exercise titled “Barriers to the Use of Maternal
tional relevant material by members of the ERTs.
Care: Antenatal Care, Skilled Birth Attendance,
A pre-summit meeting was held to organize and
Facility Delivery, and Emergency Obstetric and
motivate the ERTs, which subsequently prepared
Newborn Care.” Targeted questions on barri-
Evidence Synthesis Papers and drafted recommen-
ers to the utilization of maternal health services
dations for presentation at the Summit. The final
were posed to key informants from the Maternal
step of the Evidence Summit process—the revision
Health Task Force, World Health Organization,
of materials based on feedback at the Summit and
Save the Children, UK Department for Interna-
the implementation of an Evidence to Action Plan
tional Development, University of Aberdeen,
to act upon recommendations—is still underway.
World Bank, Family Care International, and the
The following sections provide more details on
Bill & Melinda Gates Foundation. The scoping
these steps in the Evidence Summit process, fo-
exercise identified financial barriers as key ob-
cusing primarily on the gathering of evidence and
stacles to maternal healthcare utilization. Based
review phases.
on these results, the Core Group refined the topic
GETTING STARTED to “Utilizing Financial Incentives to Create De-
mand for Maternal Health Services.”
Selecting the topic and developing focal
questions The Core Group met over a period of several
months to draft a concept paper describing the
The USAID’s Bureau for Global Health identified
background for the Summit, the goals and antici-
topics for Global Health Evidence Summits, using a
pated outcomes, and a process for accomplishing
set of criteria with seven elements which reflected
the summit objectives. On 30 August 2011, the
their commitment to evidence-informed global
Core Group also convened an external consul-
health decision-making and to engaging external
tation with experts from the fields of maternal
expertise from multidisciplinary experts for com-
health and performance-based financing to re-
plex development challenges:
view the concept paper and identify key questions
1. Enough evidence on the topic is available to for the Evidence Summit. Participants in the con-
permit policy and/or programmatic decision- sultation included experts from the Population
making; Council, Agency for Health Research and Quality,
Broad Branch Associates, Center for Global De-
2. Rigorous studies or systematic analyses are ad-
velopment, Abt Associates/Health Systems 20/20,
equately represented in the body of available
and the Bill & Melinda Gates Institute for Popu-
evidence on the topic;
lation and Reproductive Health/Johns Hopkins
3. The application of evidence will likely result
Bloomberg School of Public Health. During the
in high impact (health outcomes) and/or im-
external consultation, the experts advised that
proved implementation of interventions;
supply- and demand-side interventions and out-
4. There is a strong demand from the field for ev- comes were interdependent and should not be
idence-based solutions to pressing policy/pro- separated in the Evidence Summit. This advice
gramme challenges or questions represented by resulted in additional refinements of the concept
the topic; paper, including a decision to focus on financial
5. The topic informs an important global health incentives that would increase both demand for
issue; and supply of maternal healthcare services.
Volume 31 | Number 4 (Suppl 2) | December 2013 25
Evidence summit on maternal health and financial incentives Higgs ES et al.
Following extended internal discussions and con- B. Focal Question 2: What are the contextual fac-
sideration of recommendations provided during tors that impact the effectiveness of the finan-
the external consultation, the Core Group selected cial incentives?
two key focal questions to be addressed during the
summit process. These focal questions emerged i. Belief: Numerous contextual factors, includ-
from a set of beliefs about the roles of financial in- ing household income and wealth, pro-
centives, the contexts that led to those beliefs, and viders’ compensation, geography, health
the formation of explicit development hypotheses workers’ quality and access, availability of
about what the evidence review might find. These transport, capacity of services to accommo-
are as follows: date more clients effectively, management
of the financial incentive programme, qual-
A. Focal Question 1: What financial incentives, if ity of the health management information
any, are linked positively or negatively to ma- system, the political situation, and so forth,
ternal and neonatal health outcomes, the pro-
are critical to implementation of the incen-
vision or utilization of maternal health services,
tive programmes for maternal health and
or to care-seeking behaviour by women?
for their results.
i. Belief: Financial incentives can influence
ii. Context: While there is potential for signifi-
users’ and providers’ behaviours, including
cant positive changes for health behaviours
the utilization and provision of services and
and health outcomes, those with experi-
can potentially alter maternal and neonatal
ence in implementing and evaluating finan-
health outcomes positively and, in some
cial incentive programmes to date advise
cases, negatively. Some incentives will be
that, without understanding the context in
more influential than others and interac-
which the financial incentives are applied, it
tion of incentives in various combinations
is difficult to generalize from results in any
will produce different results.
one setting. For example, a programme that
ii. Context: In recent years, financial incentives quickly increases service utilization but that
in the form of vouchers, waivers, condi- cannot provide quality services could result
tional cash transfers, variations of pay-for- in fewer clients accessing services over time
performance, and so forth, have galvanized and/or yield negative health outcomes. Fur-
tremendous interest in the public health thermore, supervision and support, social
community. Considerable documentation norms, household support (from husband/
of financial incentives for health, in general, partner/mother/mother-in-law), commu-
has been compiled. To date, there is less in- nity wealth (or wealth inequality), and in-
formation relating to the effect of financial frastructure are some of the other contextual
incentives on maternal health behaviours, issues that can influence the results of finan-
including the use of services, providers’ be- cial incentive programmes.
haviours, and maternal and neonatal health
outcomes. Many governments and donors iii. Development hypothesis: A review of various
are supporting, with substantial invest- levels of evidence about the wide range of
ments, implementation of financial incen- contextual variants in financial incentive
tives for maternal and newborn health but programmes will aid in understanding the
this is based on limited evidence. Because of nuances of designing and implementing
the significant potential to affect the use and policies and programmes for effective results
provision of services, there is a need to iden- in different settings.
tify, synthesize, and analyze the available
To facilitate the organization of the Evidence
evidence to determine positive and negative
Summit and the literature search process, the Core
effects for maternal and newborn health.
Group subdivided the evidence by the type of fi-
iii. Development hypothesis: A review of evidence nancial incentive and development measure as-
of financial incentives and their effects on sessed in the study. As shown in Figure 2, the three
maternal and neonatal health behaviours, groups of financial mechanisms were supply-side
service delivery, and outcomes will increase mechanisms, conditional cash transfers, and oth-
understanding of available interventions er demand-side mechanisms; the outcomes were
and lead to more effective and efficient poli- maternal healthcare utilization behaviour, chang-
cies, programmes, and strategies. es in the frequency, nature or quality of services
26 JHPN
Evidence summit on maternal health and financial incentives Higgs ES et al.
Figure 2. Diagram showing how the evidence was categorized for assignment to Evidence Review Teams
OUTCOMES
Financial mechanism categories
Behaviours Services Health outcomes
(Evidence Review Teams)
Supply-side
Examples include:
Performance-based financing
Contracting (in and out)
Insurance
T
X
E Demand-side: Conditional
T
N cash transfers
O
C Demand-side: Other
examples include:
Vouchers
Subsidies
Waivers
Exemptions
Coupons
provided, or the more distal outcomes of maternal factor influencing the outcomes of different finan-
and infant mortality and morbidity. The supply- cial incentives. It may be that certain characteris-
side mechanisms, offered by governments, health tics of the patient, such as socioeconomic status or
systems, facilities, and NGOs include various forms membership in a stigmatized group, could modify
of performance-based or outcome-based incen- the effectiveness of incentives. The ERTs were asked
tives to providers, direct fiscal transfers or finan- to identify the contextual elements that should be
cial supplements for service provision, contract- examined to answer Focal Question 2.
ing for services within or outside provider groups.
Formation of Evidence Review Teams (ERTs)
Cash transfers to mothers or families conditional
on increased utilization of health services are part
Central to the Evidence Summit process was the se-
of a global interest in conditional cash transfers
lection and organization of experts into three ERTs
in the area of social protection. The application
to assess the evidence on financial incentives and
to health services utilization has not been well-
maternal health services and make recommenda-
studied but represents a unique literature to war-
tions on policies, strategies, and programmes. Ex-
rant consideration in its own right. Many of the
perts were nominated by Core Group members or,
other demand-side incentives derive, in part, from
in some cases, by outside experts who were initially
the same notion that underlies conditional cash
contacted about participating but who nominated
transfers. Those providing subsidies or vouchers
someone else in their place. Many members of the
exchangeable for goods and services or offsetting
Core Group also served on ERTs. The challenge
transportation or childcare costs if patients attend
in selecting ERT members was to achieve balance
clinics form another type of financial incentive.
along several dimensions, such as area and level of
Other examples are exemptions from payment or
expertise, programmatic and research experience,
coupons to defray costs.
and affiliation. ERT members had expertise in ma-
Finally, the careful examination of the context in ternal health, health economics, health systems,
which the research was conducted was critical to development, and other related topics, and many
answering Focal Question 2. There are several di- had experience in implementing programmes or
mensions along which context might be consid- conducting research involving the use of financial
ered. Geographical region, nation, urban versus incentives. They represented a mix of senior and
rural, or culture may be an important contextual mid-level managers, practitioners, and researchers.
Volume 31 | Number 4 (Suppl 2) | December 2013 27
Evidence summit on maternal health and financial incentives Higgs ES et al.
ERT members were affiliated with the following in- Evidence acquisition
stitutions:
Two strategies were employed to acquire evidence
• Non-governmental organizations: Abt Associates; for the Evidence Summit process: (i) a formal lit-
Broad Branch Associates; Center for Global De- erature search conducted by public health profes-
velopment; Futures Institute; JHPIEGO; John sionals, and (ii) a call for evidence issued to mem-
Snow, Inc.; Population Council; Results for bers of the ERTs. Although it was recognized that
Development; RTI International; Save the Chil- high-quality evidence existed in both published
dren; and University Research Co. and unpublished literature, the Core Group opt-
ed to limit the formal literature search to articles
• Academic institutions: Johns Hopkins Bloomberg
published in peer-reviewed scholarly journals, an-
School of Public Health, Harvard School of Pub-
ticipating that relevant documents from the grey
lic Health, London School of Hygiene & Tropi-
literature would be submitted by experts in the
cal Medicine, University of North Carolina,
field in the subsequent call for evidence. The Core
Gillings School of Global Public Health, and
Group also decided to exclude documents based
Virginia Commonwealth University
on research carried out in high-income countries.
This decision reflected the thinking that research
• US Government agencies: Centers for Disease
in high-income countries may have limited rel-
Control and Prevention, National Institutes of
evance to LMICs; if relevant research from high-
Health, and USAID
income countries was identified by experts in the
• Private foundation: Bill & Melinda Gates Founda- ERTs, it could be submitted during the call for evi-
tion dence process.
• Bilateral and multilateral institutions: Depart- Figure 3 shows the results of the initial literature
ment for International Development (United search. Knowledge Management Services (KMS)
Kingdom), World Bank, and World Health Or- conducted the literature search, compiled the da-
ganization. tabase, and conducted the screening and initial
review of literature under contract arrangements
Although the number of ERT members based in
with USAID in collaboration with the Core Group.
LMICs was relatively small, many experts had ex-
The Core Group worked with KMS to select the
tensive experience working and residing in these
search terms from key words identified in relevant
settings. It was decided to bring a greater LMIC per-
articles and consultation with experts in the fields
spective into the process during the Summit itself
of maternal health and performance-based financ-
by inviting participants from some of these coun-
ing. The search strategy for the peer-reviewed lit-
tries. Finally, the Core Group selected ERT members
erature combined terms for financial incentives
based on their knowledge and expertise in mater-
(e.g. pay for performance, results based financing,
nal health services, financial incentives, health sys-
performance based financing, performance based
tems, health economics, development, and other
scheme*, results based incentive*, performance
related topics. A large majority of persons invited
based contracting, results based contracting, pay-
to become ERT members agreed to serve in this ca-
ing for results, contracting in, contracting out,
pacity, which was a significant effort commitment,
performance based aid, performance based dis-
given the long-term nature of the Evidence Sum-
bursement, output based aid, output based financ-
mit process and the work required for these teams.
ing, fee for service, cash transfer*, cash incentive*,
The three ERTs were organized around three cat- financial incentive*, incentive*, incentive scheme*,
egories of financial incentives: supply-side finan- token economy, reinforcement, voucher*, money
cial incentives, conditional cash transfers, and to transport, transport fee*, subsidy, subsidies, sub-
other demand-side financial incentives (excluding sidized care (subsidized-care), exemption*, waiver*,
cash transfers) (Figure 2). The ERT members with user fee*, user charge*, out-of-pocket payment*,
specific expertise in a category of financial incen- coupon*, free care) with terms for maternal and
tives were assigned to that ERT. Other ERT mem- neonatal health (e.g. matern*, antenatal, prenatal,
bers were divided among the three ERTs to achieve preconception, intrapartum, perinatal, postpartum,
a balance of representation from the research and postnatal, pregnan*, childbirth, child birth, birth,
programme communities and government agen- neonate, newborn, neonatal), and the names of
cies as well as of expertise in financial incentives countries and regions categorized as low or middle
and maternal health. income by the World Bank.
28 JHPN
Evidence summit on maternal health and financial incentives Higgs ES et al.
Figure 3. Initial search and screening process to obtain a core bibliography for the Evidence Summit
Search terms: Chosen by Core Group members, based on key words used in relevant articles
and consultation with experts; list of terms tested and consolidated, where possible, into
summary terms by KMS
Terms relating to financial incentives + terms indicating maternal/neonatal as population +
names of all lower/middle-income countries and regions
MeSH terms used in PubMed
Databases searched: Chosen based on review of overlap between databases, indentification
of relevant, content specific databases, and Core Group input
Note: ABI Inform and the Cochrane database did not allow for comprehensive searches
using the three-part search term; therefore, searches were not limited by country or region
ABI Inform and the Cochrane Collaboration PubMed, SCOPUS, EconLit, Embase,
SocioAbstracts, and CABI
Documents pre-screened by title,
removing 116 documents
Duplicates removed
N=470
Screen: Included studies relevant to Focal Question # 1 and had outcome data
Inclusion criteria:
- English language
- Published on/after 1 January 1990
- From low-, middle-, or upper middle-income countries
- Published in a peer-reviewed/scholary journal
- Intervention using financial scheme(s) not including insurance and measured outcome
related to demand for health services, staff providing maternal/neonatal care, or
maternal/neonatal health outcomes
N=65
The databases searched included ABI Inform, Co- • Letters-to-the-Editor
chrane Library, PubMed, SCOPUS, EconLit, Em-
• Obituaries
base, SocioAbstracts, and CABI (Figure 3). Where
• Commentaries/recommendations not based on
possible, the following exclusion criteria were ap-
thorough literature reviews
plied to the database searches:
• Book reviews
• Documents not published in English
• Job postings
• Documents published before 1 January 1990 • Historical accounts
• Research carried out in high-income countries
Following the pre-screen and removal of duplicates,
• Magazine or newspaper articles the initial search of the peer-reviewed literature
Volume 31 | Number 4 (Suppl 2) | December 2013 29
Evidence summit on maternal health and financial incentives Higgs ES et al.
yielded a total of 470 papers. KMS then undertook ary population-level data that collected pre- and
a manual screening of the papers with two goals: post-insurance policy implementation were also
(i) to exclude documents according to the basic included.
exclusion criteria shown before, in case any such
To supplement the papers identified through the
document had not been previously eliminated via
literature search, members of the ERTs were invited
the search limits and pre-screen, and (ii) to include
to submit documents they felt would help address
only those papers with interventions directly re-
the focal questions. This took place through a for-
lated to the Focal Questions and that contained
mal call for evidence prior to the Evidence Summit.
outcome data (Figure 3). The screening algorithm
It utilized an online document submission process
is provided in the USAID Evidence Summit web site
in which ERT members were asked a series of ques-
http://www.usaid.gov/node/7186. The latter goal
tions about the documents they were submitting.
was achieved by including papers that described
research on an intervention involving the use of The Call for Evidence submission protocol can be
one or more financial incentive(s) and included found on the Evidence Summit website: http://
measurement of at least one outcome related to www.usaid.gov/node/7186. In selecting papers for
the demand for or utilization of maternal/neona- submission, the ERT members were encouraged
tal health services by women, the performance of to review the exclusion criteria that had been ap-
health professionals or organizations providing ma- plied to the initial literature search but they were
ternal/neonatal health services, or health-related not required to adhere to those criteria to enable
maternal or neonatal outcomes that resulted from the experts to include papers they deemed highly
changes in the behaviour of patients or providers. relevant to the focal questions. They were also ad-
Maternal/neonatal health services were defined as vised that purely descriptive papers had not proven
routine antenatal visits, special programmes for useful in previous USAID-supported Evidence Sum-
pregnant women (e.g. nutritional support, bednet mits and that priority should be given to the fol-
provision, etc.), care for pregnant women suffering lowing types of papers:
from an illness, intrapartum care, and other neona-
• Papers with primary data of high scientific qual-
tal services through the first 28 days of life. Abor-
ity with maternal or neonatal health outcomes,
tion and family planning services were excluded.
maternal healthcare-seeking behavioural out-
The screening process was completed through a
comes, or provider behavioural outcomes that
review of abstracts and resulted in the retention of
result from the application of a financial incen-
65 papers. In addition to excluding documents that
tive.
did not meet the screening criteria, this screening
step was also used in sorting documents into the
• Papers summarizing interventions or evalua-
type(s) of financial incentive(s) that were studied
tions
(supply-side, conditional cash transfers, or other
demand-side mechanisms) and into the type(s) of • Documents of relevance to low- and middle-
outcome(s) measures that were studied (patients’ income countries, even though these may de-
behaviour, service provision, or health outcome), scribe work done in high-income countries
effectively sorting documents into the matrix
• Documents that had undergone peer review
shown in Figure 2. This was done to assist the ERTs
to subdivide their literature review tasks among
• Systematic reviews
their members.
• Grey literature in the form of studies, reviews,
During the course of the evidence reviews, the ERT
or evaluations
that was asked to examine the literature on supply-
side incentives decided to include interventions A total of 25 documents were formally submitted
that utilized insurance, necessitating a renewed through the call for evidence. Eight duplicates were
literature search for documents on this topic that removed, resulting in 17 additional documents
otherwise met the above criteria for inclusion (Fig- (Figure 4). During the preparation of the evidence
ure 4). The initial search using insurance-related synthesis papers, ERT members were encouraged to
terms as shown in the figure yielded 606 papers cite other documents that they felt were relevant
but, after screening, the number of insurance docu- without submitting these through the form Call for
ments was reduced to 21. Due to the difficulties in Evidence Process. This resulted in an additional 36
undertaking an insurance intervention study that documents. The final bibliography contained 139
includes a control group, articles reporting second- documents. The geographic location where the
30 JHPN
Evidence summit on maternal health and financial incentives Higgs ES et al.
Figure 4. Diagram showing the steps for adding documents to the initial bibliography that include
material on the role of insurance and the documents that were added through the Call For
Evidence process
Insurance literature search for ERT 1-
supply-side financial incentives
Searched Pubmed, ABI Inform, Cochrane,
SCOPUS, EconLit, Embase, SocioAbstracts N=65
and CABI, using "insurance" +
maternal/neonatal term + lower/middle
Articles submitted via
income countries and regions + limits applied
the Call for Evidence
in original search
N=606 N=17
Additional articles
Screen
identified by ERTs
Applied same inclusion/exclusion
N=36
criteria (but included insurance-related
interventions only)
N=21 N=139
studies were conducted for the final bibliography concerning the types of equivalence of the com-
was as follows: Latin American Countries (n=28), parator group, if any; the adequacy of the study
Africa (n=48), Middle East and Southeast Asia design; the fidelity of the intervention; the validity
(n=35), reviews or studies involving many LMICs and relevance of the outcome measures; the data
(n=24), general papers about financial incentive- analysis; the generalizability of the results; and
use in LMICs (n=4) evidence for the sustainability of the intervention.
Each document was assigned to two ERT members
Quality review process
for review. Of the 82 papers, 70 underwent a qual-
ity review with 28 papers reviewed by one reviewer,
After the screening and sorting processes were com-
33 papers reviewed by two reviewers, and 9 pa-
pleted, ERT members were asked to assess the qual-
pers reviewed by three reviewers. Summary qual-
ity of the 65 papers derived from literature search
ity scores between 0 and 1 (0 for lower quality, 1
and screening process and the 17 papers obtained
for higher quality) were derived for use by the ERT
through the Call for Evidence. A complex quality
members in the later evaluation of evidence. The
review framework for both empirical studies and
summary score did not penalize documents if re-
programme evaluation documents had been de-
viewers skipped a question.
veloped previously for the Evidence Summit on
Protecting Children Outside of Family Care (5). Ex-
Pre-summit meeting
perience from that summit suggested that a simpler
quality review tool would be more practical for the The pre-summit was a technical working meeting
purposes of this process, and so the question set was held on 14 March 2012 in Bethesda, Maryland,
revised to a set of eight questions (visit http://www. hosted by the Eunice Kennedy Shriver National In-
usaid.gov/node/7186 for the text of the quality re- stitute of Child Health and Human Development.
view questionnaire). ERT members were assigned Members of the three ERTs came together to learn
sets of papers to review and asked eight questions about the purpose of the Evidence Summit and its
Volume 31 | Number 4 (Suppl 2) | December 2013 31
Evidence summit on maternal health and financial incentives Higgs ES et al.
anticipated outcomes, discuss the initial review of requirements for global health decisions and the
the literature, develop work plans for producing evidence summits are more complex than those
an evidence synthesis and recommendations, and to support best practices guidelines for physician’s/
discuss strategies for increasing the impact of the patient’s decision-making. For global program-
Evidence Summit. The meeting involved the pre- ming, the evidence must not only address ‘efficacy’
sentation of background material and a series of at the individual level or within a specific context
interactive roundtable discussions, and breakout but also ‘effectiveness’ at the community and pop-
sessions to facilitate the attainment of the pre- ulation levels in differing locations and contextu-
summit objectives. ally-varied environments. Further, ‘sustainability’
at the country level is critical for country owner-
Evidence packets summarizing the results of the
ship and long-term feasibility. For governments of
quality review process were provided to ERT mem- LMICs and donors, evidence on the feasibility of
bers at the pre-summit meeting. The packets were an intervention’s implementation on a population
divided according to the ERT categories such that basis and its cost-effectiveness are critical to invest-
there was one packet for those papers relevant to ment and resource allocation decisions leading to
supply-side incentives, one for those relevant to the sustainability of the intervention. These three
conditional cash transfers and one for those rel- streams of evidence typically result from different
evant to other demand-side incentives. Included in research approaches; thus, varying methodologies
the evidence packets were the citations, abstracts, are needed to evaluate the evidence generated by
and individual responses and summary quality each stream. Therefore, the evidence evaluation ap-
scores for each paper. During the pre-summit meet- proach for this Summit allowed for mixed research
ing, the ERTs used these reports to begin consider- methodologies to incorporate relevant evidence
ing how well the identified literature addressed the targeted to the three crucial data streams of efficacy,
two focal questions. effectiveness, and sustainability.
The Core Group outlined the expectations for the Given the complexity of global health and develop-
ERTs and for the products to be completed for the ment questions, using both evidence-base as well as
Evidence Summit in April, which included an evi- expert opinion is important in developing recom-
dence synthesis paper and a presentation summa- mendations for policies and programmes that can
rizing the evidence synthesis, along with recom- maximize their impact. For some practices which
mendations for policy, practice, and research. ERTs have been widely and successfully implemented,
were encouraged to continue adding relevant docu- there may not be rigorous controlled trials dem-
ments to the bibliography throughout the develop- onstrating their efficacy, much less their effective-
ment of the evidence synthesis papers. They were ness and sustainability. It was also acknowledged
also provided with a framework for evaluating the that the quality of the research support for some
evidence and making recommendations based on interventions may not be very high, which is why
both evidence and expert opinion. This framework ERT members were asked to rate the quality of the
is described below. During the pre-summit period, studies they were reviewing. In making recommen-
the participants decided that insurance must be in- dations for interventions to enhance the use and
cluded as an FI mechanism, and an additional liter- availability of maternal health services, ERT mem-
ature search followed to identify relevant studies. bers were advised to use both evidence and expert
opinion. They were also requested to consider the
Evaluating evidence and making
quality of the evidence and to make clear which
recommendations
recommendations relied more on expert opinion
informed by field experience.
ERTs were provided an evidence framework to as-
sist them in drawing conclusions and making rec-
EVIDENCE SUMMIT
ommendations. The framework was developed
for the Evidence Summit on Protecting Children ERTs were asked to prepare two presentations for
Outside of Family Care and is described in more the Evidence Summit: (i) a narrative evidence
detail in a paper describing the methodology for synthesis which contained a review of the evi-
that Evidence Summit (5). Evidence standards ini- dence on the focal questions that pertained to
tially evolved from the medical field where physi- the financial mechanism they were assigned to
cian’s decision-making is determined primarily by consider and (ii) a set of recommendations. The
data derived from randomized clinical trials which Evidence Summit was held in Washington, DC
prove ‘efficacy’ for the individual patient. Evidence on 24-25 April 2012, and the agenda and list of
32 JHPN
