Table Of ContentEvaluating Improvement and
D
Evaluating o
w
n
Implementation for Health lo
a
d
e
d
b
E y
v [ F
a
“This book is to be welcomed for its wide ranging introduction to the many al Improvement culty
approaches to evaluation.” Carolyn M Clancy, Former Director, Agency for u o
a f N
Healthcare Research and Quality (AHRQ) u
ti rsin
n g
“For anyone looking for a readable and complete introduction to evaluation, g , Ch
the search ends here. This book gives an overview of evaluation in action for I and iang
m m
making better decisions about how to improve health outcomes for individuals, a
communities, and nations. The emphasis on including assessments of p i Un
implementation is refreshing and the examples throughout the book illuminate rov iversity
the concepts and pique the reader’s curiosity right to the end.” Dean L. Fixsen, e 5.6
University of North Carolina at Chapel Hill, Senior Scientist, & Co-Director, National m Implementation 2.1
5
Implementation Research Network, USA e 8.1
1
nt 7] at [0
Evaluating Improvement and Implementation for Health describes modern a 7/1
ethvea lausastiuomn pmtieotnhso dosf tihne h eevaildthecnacree- baansde dp ohleicayl tmhcaakrien gm, oavnedm cehnatl:lenges some of nd fffooorrr HHeeaalltthh 8/16]. C
Im opy
• Are innovations always an improvement? rig
p ht ©
• Are they always worth it? le M
c
m G
• Can they be implemented? raw
e -H
• More importantly, should they be implemented? nt ill G
a lo
b
These are questions with practical consequences and questions which ti al E
evaluation can answer – if we choose the right methods. This book will help you o duc
do just that – match the right evaluation method to the questions being asked. n ation
f H
o o
Pragmatic, even-handed and accessible, Evaluating Improvement and r lding
Implementation for Health provides an overview of the many different evaluation H s, L
perspectives and methods used in the health sector. Suitable for health ea LC. N
practitioners, managers, policy advisers, and researchers, its practical and lt ot to
multidisciplinary approach shows how to ensure that evaluation results in action. h b
e
red
Jo istrib
JOHN ØVRETVEIT is an award-winning author and Professor of Health Improvement, h ute
Implementation and Evaluation at the Karolinska Institute Academic Medical Center in n d o
r m
Stockholm where he is Director of Research at the Medical Management Center of the Ø o
d
Learning Informatics Management and Ethics Department. vre John ified in
a
t ny
v w
e Øvretveit ay
it with
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
y
[ F
a
c
u
lty
o
Evaluating Improvement and f N
u
rsin
g
Implementation for Health , C
h
ia
n
g
m
a
i U
n
iv
e
rsity
5
.6
2
.1
5
8
.1
1
7
] a
t [0
7
/1
8
/1
6
]. C
o
p
y
rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
y
[ F
a
c
u
lty
o
f N
u
rsin
g
, C
h
ia
n
g
m
a
i U
n
iv
e
rsity
5
.6
2
.1
5
8
.1
1
7
] a
t [0
7
/1
8
/1
6
]. C
o
p
y
rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
y
[ F
a
c
u
lty
o
Evaluating Improvement f N
u
rsin
g
and Implementation , C
h
ia
n
g
m
a
for Health i U
n
iv
e
rsity
5
.6
2
.1
5
8
.1
1
7
] a
John Øvretveit t [07
/1
8
/1
6
]. C
o
p
y
rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
Open University Press y [ F
McGraw- Hill Education ac
u
McGraw- Hill House lty
o
Shoppenhangers Road f N
u
Maidenhead rsin
g
Berkshire , C
h
England ia
n
g
SL6 2QL m
a
i U
email: [email protected] n
iv
e
world wide web: www.openup.co.uk rsity
5
and Two Penn Plaza, New York, NY 10121- 2289, USA .6
2
.1
5
8
.1
1
7
First published 2014 ] a
t [0
Copyright © John Øvretveit, 2014 7
/1
8
All rights reserved. Except for the quotation of short passages for the /16
purpose of criticism and review, no part of this publication may be ]. Co
p
reproduced, stored in a retrieval system, or transmitted, in any form or yrig
by any means, electronic, mechanical, photocopying, recording or ht ©
otherwise, without the prior written permission of the publisher or a M
c
G
licence from the Copyright Licensing Agency Limited. Details of such ra
w
licences (for reprographic reproduction) may be obtained from the -H
Copyright Licensing Agency Ltd of Saffron House, 6–10 Kirby Street, ill G
London EC1N 8TS. lob
a
l E
A catalogue record of this book is available from the British Library du
c
a
ISBN- 13: 978- 0- 33- 524277- 1 (pb) tion
H
ISBN- 10: 0- 33- 524277- 4 (pb) o
ld
eISBN: 978- 0- 33- 524278- 8 in
g
s, L
Library of Congress Cataloging- in- Publication Data L
C
CIP data applied for . N
o
t to
b
Typesetting and e- book compilations by e
re
RefineCatch Limited, Bungay, Suffolk distrib
u
te
d
o
Fictitious names of companies, products, people, characters and/or data r m
o
that may be used herein (in case studies or in examples) are not intended d
to represent any real individual, company, product or event. ified
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
Praise for this book y [ F
a
c
u
lty
o
f N
u
rsin
g
, C
h
ia
n
g
m
a
“For anyone looking for a readable and complete introduction to eval- i U
n
uation, the search ends at Evaluating Improvement and Implementation iv
e
for Health by John Øvretveit. It provides an overview of evaluation in rsity
action for making better decisions about how to improve health outcomes 5.6
2
for individuals, communities, and nations. The emphasis on including .1
5
8
assessments of implementation as a necessary part of any evaluation is .1
1
7
refreshing and the examples throughout the book illuminate the concepts ] a
and pique the reader’s curiosity right to the end. Reading this book is time t [0
7
well spent.” /18
/1
Dean L. Fixsen, Ph.D., University of North Carolina at Chapel Hill, 6]. C
Senior Scientist, FPG Child Development Institute, Co-Director, o
p
y
National Implementation Research Network, USA rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
b
y
[ F
a
c
u
lty
o
f N
u
rsin
g
, C
h
ia
n
g
m
a
i U
n
iv
e
rsity
5
.6
2
.1
5
8
.1
1
7
] a
t [0
7
/1
8
/1
6
]. C
o
p
y
rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
Contents b
y
[ F
a
c
u
lty
o
f N
u
rsin
g
, C
h
ia
n
g
Foreword ix ma
i U
n
Acknowledgements xi ive
rsity
5
.6
2
.1
Part I Introduction: Concepts 1 5
8
.1
1
7
1 Evaluating health interventions, improvements and their ] at [0
implementation 3 7
/1
8
/1
6
2 Evaluation tools and concepts 16 ]. C
o
p
y
3 Quick start: How do I plan and carry out an evaluation? 30 rig
h
t ©
4 Designs 41 M
c
G
ra
w
-H
Part II Introduction: Example evaluations – do they answer ill G
users’ questions? 59 lob
a
l E
d
u
5 Evaluating interventions to patients and patient performance 63 ca
tio
n
H
6 Evaluating interventions to professionals and performance o
ld
evaluations 78 ing
s, L
L
7 Evaluating interventions to health services and performance 91 C
. N
o
8 Evaluating interventions to health systems and performance 107 t to
b
e
re
d
9 Ehevaalltuha steinrgv ipcoe ppuelraftoiormn ahnecaelt h interventions and public 120 istribu
te
d
o
r m
o
d
Part III Introduction: Evaluation subjects 131 ifie
d
in
10 Evaluating implementation 133 an
y
w
a
y
11 Evaluating digital health technologies 145 w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
viii Contents nlo
a
d
e
d
b
12 Evaluating complex social interventions 158 y [ F
a
c
u
13 Evaluating return on investment and value improvements 176 lty
o
f N
14 Economic evaluation 184 ursin
g
, C
h
ia
Glossary 195 ng
m
a
References 202 i Un
iv
e
Index 215 rsity
5
.6
2
.1
5
8
.1
1
7
] a
t [0
7
/1
8
/1
6
]. C
o
p
y
rig
h
t ©
M
c
G
ra
w
-H
ill G
lo
b
a
l E
d
u
c
a
tio
n
H
o
ld
in
g
s, L
L
C
. N
o
t to
b
e
re
d
istrib
u
te
d
o
r m
o
d
ifie
d
in
a
n
y
w
a
y
w
ith
o
u
t p
e
rm
issio
n
.
D
o
w
n
lo
a
d
e
d
Foreword b
y
[ F
a
c
u
lty
o
f N
u
rsin
g
, C
h
ia
n
g
m
As the author notes by quoting a famous evaluator, one important issue is not whether a
we evaluate, but how well we do it. As health care systems around the world struggle i Un
iv
wthieth g trhoew cinogm pmroenv aclhenacllee nogfe sc horfo nagici nigll npeospsuesla, ttihoenrse, rias peidnloyr mriosiunsg eenxtpheunsidaistumr easn adn da ersity 5
growing marketplace for innovations that promise to address our current dilemmas .62
.1
– preferably painlessly. However, too many interventions are taken up without proper 58
.1
testing, and only later do leaders, clinicians and the public learn that the gap between 17
ideas and effective implementation can be a chasm, e.g., the ‘new new thing’ is less ] at [0
effective than the enthusiasts suggested, there are safety or cost issues or we haven’t 7/1
8
thought through a number of issues such as capacity to customize for a particular set /1
6
of circumstances or the time frame for achieving success. Rodgers famously observed ]. C
o
in his classic book, Diffusion of Innovation, that the loss in effectiveness that occurs py
rig
when interventions are applied to new settings has received far too little scrutiny. h
t ©
With the rapid growth in innovations and market opportunities, we desperately M
c
need independent evaluation to guide our decisions about which to adopt, how G
ra
quickly and when. w
-H
This is important for those allocating financial resources through insurance to ill G
pay for some interventions and services which may be taking resources from lo
b
a
other more effective ones, as well as those leading large health care systems l E
d
struggling with ‘make or buy’ issues. It is also important, as this book observes, to uc
a
evaluate how we implement already-evaluated improvements in different settings tio
n
and contexts. When evaluation shows what may be effective, we may lose lives Ho
ld
or large amounts of resources because take-up is slow and variable, even when in
g
people agree about what is a better way. ‘Implementation’ suggests a straightforward s, L
L
translation from evaluation research to practice, but it is rarely as simple as that. C
. N
We need to use evaluation also to learn which of the different implementation o
methods and infrastructures are effective for enabling take-up of proven improve- t to b
e
ments, and how to customize the applications without losing fidelity to the goals of re
d
intervention. istrib
One reason many interventions and changes are not evaluated is because they may ute
d
not be amenable to evaluation through experimental methods. This does not mean o
r m
that they cannot be evaluated. There are a range of designs, which, if properly chosen o
d
and applied, can provide actionable knowledge. The perfect can be the enemy of ifie
d
the good, so long as the good comes with a clear statement of the limitations of the in
a
n
findings. The ethics of evaluation include evaluators spelling out these limitations, as y
w
they often know best the strengths and weaknesses of their findings, as well as taking ay
w
a properly skeptical and neutral attitude to the intervention. Moreover, far too many ith
o
examples of dramatic improvements in care delivery led by charismatic leaders in ut p
e
rm
issio
n
.
