Table Of ContentRegression Analysis
With the rise of “big data,” there is an increasing demand to learn the skills needed to undertake sound
quantitative analysis without requiring students to spend too much time on high-level math and
proofs. This book provides an efficient alternative approach, with more time devoted to the practical
aspects of regression analysis and how to recognize the most common pitfalls.
By doing so, the book will better prepare readers for conducting, interpreting, and assessing
regression analyses, while simultaneously making the material simpler and more enjoyable to learn.
Logical and practical in approach, Regression Analysis teaches: (1) the tools for conducting regressions;
(2) the concepts needed to design optimal regression models (based on avoiding the pitfalls); and
(3) the proper interpretations of regressions. Furthermore, this book emphasizes honesty in research,
with a prevalent lesson being that statistical significance is not the goal of research.
This book is an ideal introduction to regression analysis for anyone learning quantitative methods
in the social sciences, business, medicine, and data analytics. It will also appeal to researchers and
academics looking to better understand what regressions do, what their limitations are, and what they
can tell us. This will be the most engaging book on regression analysis (or Econometrics) you will
ever read!
Jeremy Arkes is Associate Professor at the Graduate School of Business and Public Policy,
Naval Postgraduate School, U.S.A. He conducts research in a variety of fields, with a focus on
military-manpower policy, substance-use policy, determinants of youth outcomes, sports economics,
and using sports outcomes to make inferences on human behavior.
Regression Analysis
A Practical Introduction
Jeremy Arkes
First published 2019
by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
and by Routledge
52 Vanderbilt Avenue, New York, NY 10017
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2019 Jeremy Arkes
The right of Jeremy Arkes to be identified as author of this work has been asserted
by him in accordance with sections 77 and 78 of the Copyright, Designs and
Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or
utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or in any
information storage or retrieval system, without permission in writing from the
publishers.
Trademark notice: Product or corporate names may be trademarks or registered
trademarks, and are used only for identification and explanation without intent
to infringe.
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
A catalog record for this book has been requested
ISBN: 978-1-138-54140-5 (hbk)
ISBN: 978-1-138-54143-6 (pbk)
ISBN: 978-1-351-01109-9 (ebk)
Typeset in Bembo
by Apex CoVantage, LLC
Visit the eResources: www.routledge.com/9781138541405
To my mother, Judy Arkes (1941–2014). She was always a voice of love,
support, and reason.
Contents
List of figures xi
List of tables xiii
About the author xv
Preface xvi
Acknowledgments xviii
List of abbreviations xix
1 Introduction 1
1.1 The problem 2
1.2 The purpose of research 3
1.3 What causes problems in the research process? 4
1.4 About this book 7
1.5 The most important sections in this book 9
1.6 Quantitative vs. qualitative research 10
1.7 Stata and R code 11
1.8 Chapter summary 11
2 Regression analysis basics 13
2.1 What is a regression? 14
2.2 The four main objectives for regression analysis 15
2.3 The Simple Regression Model 17
2.4 How are regression lines determined? 21
2.5 The explanatory power of the regression 26
2.6 What contributes to slopes of regression lines? 29
2.7 Using residuals to gauge relative performance 30
2.8 Correlation vs. causation 32
2.9 The Multiple Regression Model 33
2.10 Assumptions of regression models 35
2.11 Calculating standardized effects to compare estimates 38
2.12 Causal effects are “average effects” 39
viii | Contents
2.13 Causal effects can change over time 40
2.14 A quick word on terminology for regression equations 41
2.15 Definitions and key concepts 42
2.16 Chapter summary 44
3 Essential tools for regression analysis 47
3.1 Using binary variables (how to make use of dummies) 47
3.2 Non-linear functional forms using OLS 50
3.3 Weighted regression models 57
3.4 Chapter summary 59
4 What does “holding other factors constant” mean? 61
4.1 Case studies to understand “holding other factors constant” 62
4.2 Using behind-the-curtains scenes to understand “holding other
factors constant” 68
4.3 Using dummy variables to understand “holding other factors constant” 72
4.4 Using Venn diagrams to understand “holding other factors constant” 76
4.5 Could controlling for other factors take you further from the true causal effect? 76
4.6 Application of “holding other factors constant” to the story of oat bran
and cholesterol 78
4.7 Chapter summary 80
5 Standard errors, hypothesis tests, p-values, and aliens 82
5.1 Setting up the problem for hypothesis tests 83
5.2 Hypothesis testing in regression analysis 86
5.3 The drawbacks of p-values and statistical significance 101
5.4 What the research on the hot hand in basketball tells us about the existence
of other life in the universe 104
5.5 What does an insignificant estimate tell you? 105
5.6 Statistical significance is not the goal 106
5.7 Chapter summary 107
6 What could go wrong when estimating causal effects? 111
6.1 How to judge a research study 113
6.2 Exogenous (good) variation vs. endogenous (bad) variation 114
6.3 Setting up the problem for estimating a causal effect 115
6.4 The big questions for what could bias the coefficient estimate 117
6.5 How to choose the best set of control variables (model selection) 146
6.6 What could bias the standard errors and how do you fix it? 149
6.7 What could affect the validity of the sample? 156
6.8 What model diagnostics should you do? 158
6.9 Make sure your regression analyses/interpretations do no harm 159
6.10 Applying the big questions to studies on estimating divorce effects on children 160
6.11 Applying the big questions to nutritional studies 161
6.12 Chapter summary: a review of the big questions 163
Contents | ix
7 Strategies for other regression objectives 169
7.1 Strategies for forecasting/predicting an outcome 169
7.2 Strategies for determining predictors of an outcome 172
7.3 Strategies for adjusting outcomes for various factors 176
7.4 Summary of the strategies for each regression objective 180
8 Methods to address biases 183
8.1 Fixed-effects 184
8.2 A thorough example of fixed effects 196
8.3 An alternative to the fixed-effects estimator 200
8.4 Random effects 201
8.5 First-differences 203
8.6 Difference-in-differences 207
8.7 Two-stage Least Squares (instrumental-variables) 211
8.8 Regression discontinuities 217
8.9 Case study: research on how divorce affects children 220
8.10 Knowing when to punt 225
8.11 Chapter summary 225
9 Other methods besides Ordinary Least Squares 231
9.1 Types of outcome variables 232
9.2 Dichotomous outcomes 233
9.3 Ordinal outcomes – ordered models 239
9.4 Categorical outcomes – Multinomial Logit Model 241
9.5 Censored outcomes – Tobit models 243
9.6 Count variables – Negative Binomial and Poisson models 245
9.7 Duration models 247
9.8 Chapter summary 250
10 Time-series models 252
10.1 The components of a time-series variable 253
10.2 Autocorrelation 253
10.3 Autoregressive models 256
10.4 Distributed-lag models 262
10.5 Consequences of and tests for autocorrelation 266
10.6 Stationarity 269
10.7 Vector Autoregression 274
10.8 Forecasting with time series 275
10.9 Chapter summary 279
11 Some really interesting research 281
11.1 Can discrimination be a self-fulfilling prophecy? 281
11.2 Does Medicaid participation improve health outcomes? 286
11.3 Estimating peer effects for academic outcomes 287
11.4 How much does a GED improve labor-market outcomes? 290
