Bad Data: Why We Measure the Wrong Things and Often Miss the Metrics That Matter

353 Pages·2020·2.868 MB·English

by  Peter Schryvers

BAD DATA

BAD
DATA
Why We Measure the Wrong Things
and Often Miss the Metrics That Matter
PETER SCHRYVERS
Prometheus Books
An imprint of The Rowman & Littlefield Publishing Group, Inc.
4501 Forbes Blvd., Ste. 200
Lanham, MD 20706
www.rowman.com
Distributed by NATIONAL BOOK NETWORK
Copyright © 2020 Peter Schryvers
Cover design by Liz Mills | Cover design Copyright © 2020 Prometheus Books
All rights reserved. No part of this book may be reproduced in any form or by any electronic
or mechanical means, including information storage and retrieval systems, without written
permission from the publisher, except by a reviewer who may quote passages in a review.
British Library Cataloguing in Publication Information available
Library of Congress Cataloging-in-Publication Data
Names: Schryvers, Peter, 1983– author.
Title: Bad data : why we measure the wrong things and often miss the metrics that matter / by
Peter Schryvers.
Description: Lanham : Rowman & Littlefield International, 2019. | Includes index.
Identifiers: LCCN 2019012234 (print) | LCCN 2019980461 (ebook) | ISBN
9781633885905 (cloth) | ISBN 9781633885912 (epub)
Subjects: Quantitative research--Evaluation. | Performance—Measurement.
Classification: LCC QA76.9.Q36 S37 2019 (print) | LCC QA76.9.Q36 (ebook)
LC record available at https://lccn.loc.gov/2019012234
LC ebook record available at https://lccn.loc.gov/2019980461
The paper used in this publication meets the minimum requirements of American National
Standard for Information Sciences—Permanence of Paper for Printed Library Materials, ANSI/
NISO Z39.48-1992
CONTENTS
Introduction ix
Chapter 1: T eaching to the Test: Goodhart’s Law and the
Paradox of Metrics 1
Chapter 2: Th e Ins and Outs: The Logic Model and
Program Evaluation 39
Chapter 3: Th e Long and Short of It: Intertemporal Problems
and Undervaluing Time 65
Chapter 4: Th e Problem of Per: Denominator Errors 95
Chapter 5: Th e Forest and the Trees:
Simplifying Complex Systems 117
Chapter 6: A pples and Oranges: Ignoring Differing Qualities 141
Chapter 7: N ot Everything That Can Be Counted Counts:
The Lamppost Problem 163
Chapter 8: N ot Everything That Counts Can Be Counted:
Measuring What Matters 221
Chapter 9: Th e Measure of Metrics 261
Chapter 10: Gateways Not Yardsticks 283
v
CONTENTS
Acknowledgments 295
Notes 297
Index 315
vi
Not everything that counts can be counted, and not every-
thing that can be counted counts.
—William Bruce Cameron

INTRODUCTION
Man is a measurer of all things.
—Herbert Arthur Klein
T
here is a species on this planet that has dominated nearly every
ecosystem it has come into contact with. From arid deserts to
lush rain forests, from high mountains to low valleys, and everything in
between, it has spread and conquered. There is almost nowhere you can-
not find it. It has learned to grow and harvest plants, to domesticate other
animals to serve its needs, to create intricate and purposeful structures
and habitats, to build complex societies with a division of labor and sep-
arate classes, and even has gone to war and enslaved members of its own
species.
I am, of course, talking about ants.
To say that ants are merely a successful species would be an under-
statement. Not only have they colonized nearly every ecosystem and cli-
mate on our planet, they have done so in large numbers. In almost any
ecosystem, if you were to count up all the animals present, ants would
most likely be the most numerous. In some habitats ants make up not
only the largest number of animals, but outweigh the others in total body
mass.
The success of ants can be attributed to something that sets them
apart from most other insects: They are social. They work together.
Through elaborate divisions of labor and complex methods of communi-
cation and adaptations, ants have found a way to cooperate like few other
species, humans being one of a few exceptions.
Ants work together so well because they communicate. Ants use an
intricate web of signals to identify food sources and potential enemies,
ix