Table Of Contenti
PRAISE FOR LEARNING ANALYTICS
“An outstanding contribution to the field. A very well researched and engaging
book filled with compelling research, sound theory and excellent examples to
help practitioners move beyond traditional activity measures to far more
impactful measures of effectiveness and impact. I strongly recommend this
work to anyone new to the field as well as those who are seasoned practitioners.“
David Vance, Executive Director, Center for Talent Reporting
“The second edition of Learning Analytics reaffirms the fundamentals of the
first edition—that evaluation principles when standardized and enabled by
technology—can provide insights about which programs are helping achieve
business goals and which need improvement. This edition also brings a new
methodology to the table, the Portfolio Evaluation Approach, that helps
L&D organizations align and deeply connect with the business strategy. This
edition shows how to measure informal learning, and finally, the authors
address the broader category of HR analytics and how it can be tackled using
many of the same principles that drive learning analytics. Read it. Use it.
Improve your measurement practices.”
Kimo Kippen, Founder of Aloha Learning Advisors and former Chief Learning
Officer at Hilton Worldwide
“Learning Analytics is the most thoroughly researched book on analytics that
has ever come to market. It covers all the meaningful analytics models,
explaining how they’ve been applied along with the pros and cons of each. The
purpose of analytics is to link the effects of learning programs to organizational
operations. The authors show only a small percentage of organizations venture
beyond Level 2Learning. In my 30 years’ experience, I found training managers
don’t attempt to show value added for two reasons. One, they don’t know
how to do it. Two, they fear their programmes don’t add measurable value. If
they reach the point where they want to analyse and report the effects of
development programs, this book will guide them smoothly along that path.”
Jac Fitz-enz, Founder, Saratoga Institute and Human Capital Source
ii
“If you need to evaluate your investments in talent, or use HR analytics to
inform your business decisions, then this book is a godsend.”
Doug Gray, CEO and Founder of Action Learning Associates and former
President of ATD Nashville
“Learning Analytics not only builds a strong science-based business case for
‘why learning analytics’ is needed, but goes beyond to identify the gap
between what L&D is measuring in most organizations vs what CEOs want
to be measured and achieved. Using a framework of efficiency, effectiveness
and business outcome metrics and methods this book resolves the business
to L&D gap, in a well-crafted step by step process. My favourite phrase in
the book is ‘The future is for those who can predict it’. This book is a fantastic
roadmap, guidebook and treasure trove of research, enabling L&D leaders
and practitioners alike to solve that most obvious yet difficult of challenges,
clearly linking L&D programmes to all areas of talent management in a
cohesive, strategic way to positively impact performance and productivity.”
Jeff Higgins, Founder and CEO, HCMI
iii
Learning Analytics
Using talent data to improve
business outcomes
SECOND EDITION
John R Mattox II
Peggy Parskey
Cristina Hall
iv
Publisher’s note
Every possible effort has been made to ensure that the information contained in this book is
accurate at the time of going to press, and the publishers and authors cannot accept
responsibility for any errors or omissions, however caused. No responsibility for loss or
damage occasioned to any person acting, or refraining from action, as a result of the material
in this publication can be accepted by the publisher or the authors.
First published in Great Britain and the United States in 2020 by Kogan Page Limited
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be
reproduced, stored or transmitted, in any form or by any means, with the prior permission in
writing of the publishers, or in the case of reprographic reproduction in accordance with the terms
and licences issued by the CLA. Enquiries concerning reproduction outside these terms should be
sent to the publishers at the undermentioned addresses:
2nd Floor, 45 Gee Street 122 W 27th St, 10th Floor 4737/23 Ansari Road
London New York, NY 10001 Daryaganj
EC1V 3RS USA New Delhi 110002
United Kingdom India
www.koganpage.com
Kogan Page books are printed on paper from sustainable forests.
© John R Mattox II, Peggy Parskey and Cristina Hall 2020
The right of John R Mattox II, Peggy Parskey and Cristina Hall to be identified as the authors of this
work has been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
ISBNs
Hardback 9781789662986
Paperback 9781789663006
Ebook 9781789663013
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library.
Library of Congress Control Number
2020932529
Typeset by Integra
Print production managed by Jellyfish
Printed and bound by CPI Group (UK) Ltd, Croydon CR0 4YY
v
Dedication
To my family and colleagues, thank you for
your support in this endeavour.
John R Mattox, II
My sincere thanks to colleagues and friends in the learning and
measurement professions who have helped me expand my thinking and
insights about this ever-changing field. My deepest thanks to my husband,
Kevin, whose support on this effort and throughout the years has made it
possible for me work on projects that give me joy.
Peggy Parskey
For Chris and Talon, and all who are passionate about maximizing
human potential through lifelong learning.
Cristina Hall
vi
THIS PAGE IS INTENTIONALLY LEFT BLANK
vii
CONTENTS
List of figures and tables xii
About the authors xviii
Foreword xix
How to read this book xxv
Acknowledgements xxvii
PART ONE
The L&D value gap and how to close it
01 The rise of learning analytics 3
Why is all of this important? 6
Standards are coming 7
Data availability 8
Changing the way talent analytics work gets done 9
Providing unique insight into employee behaviour 12
Learning analytics rises 13
Notes 19
02 What is learning analytics? 21
Introduction 21
Learning analytics today: measure for measure, what should be
measured? 23
Why measure learning? 25
Most organizations start with the simple: measure training activity
and satisfaction 26
Efficiency, effectiveness and business outcomes: closing the learning
measurement gap 28
The journey to learning analytics 29
The Four Levels of Evaluation 30
The Return on Investment Methodology 31
Impact Measurement Framework 33
Success Case Method 34
viii CONTENTS
Performance-Based Evaluation 36
Conclusion 44
Notes 46
03 The value-centred learning organization: A new evaluation
paradigm 49
Volume is not value 49
We’re already delivering value, though… right? 50
Delivering and demonstrating value: the Talent Development Value
Framework 52
The Talent Development Value Framework in action 61
Advancing measurement maturity 63
Conclusion 65
Notes 66
PART TWO
Establishing sound measurement practices
04 Aligning L&D’s value with the C-suite: The Four Value Drivers
and Portfolio Evaluation 71
What the C-suite wants from L&D 71
Connecting L&D with the business strategy 73
The Four Value Drivers 74
Building business alignment 76
Translating value drivers to action: Portfolio Evaluation for L&D 78
Immediate benefits of portfolio alignment 83
Additional benefits: portfolio management 85
Change the conversation 90
Conclusion 91
Notes 91
05 Linking learning to business impact 93
What works? 93
Why does it work? 95
Experimental designs 99
Alternatives to experimental designs 103
Alternative designs: practical ways forward 103
CONTENTS ix
Conclusion 114
Notes 116
06 The new leading indicators of success and how to manage
them 118
Your training programmes may not be as good as you think they
are 118
Scrap learning and how to reduce it 121
Performance improvement 126
Net promoter score 129
Manager support and how to improve it 131
Predictive Learning Impact Model 2.0: Causal Modelling 140
Conclusion 154
Notes 155
07 Developing a sustainable reporting strategy 156
The role of reporting in learning analytics 156
Getting started: design principles 159
Components of an effective reporting strategy 161
Reporting strategy development 170
Critical success factors 172
Perform gap assessment 175
Implementing the strategy 181
Special cases: dashboards and scorecards 183
Monitor the strategy: success indicators 188
Conclusion 189
Notes 190
08 Technology’s role in learning measurement 192
What should technology do? 193
Benefits and costs of learning technologies 194
The requirements for a new technology system in the BI space 200
The challenge of self-reported data 206
What is the ROI of technology systems? 209
Applying principles of business intelligence systems to L&D 210
Additional technologies 215
Conclusion 218
Notes 219
