Table Of ContentSMALL SAMPLE SIZE SOLUTIONS
Researchers often have difficulties collecting enough data to test their hypotheses,
either because target groups are small or hard to access, or because data collection
entails prohibitive costs. Such obstacles may result in data sets that are too small for
the complexity of the statistical model needed to answer the research question. This
unique book provides guidelines and tools for implementing solutions to issues
that arise in small sample research. Each chapter illustrates statistical methods that
allow researchers to apply the optimal statistical model for their research question
when the sample is too small.
This essential book will enable social and behavioral science researchers to test
their hypotheses even when the statistical model required for answering their
research question is too complex for the sample sizes they can collect. The statistical
models in the book range from the estimation of a population mean to models with
latent variables and nested observations, and solutions include both classical and
Bayesian methods. All proposed solutions are described in steps researchers can
implement with their own data and are accompanied with annotated syntax in R.
The methods described in this book will be useful for researchers across the social
andbehavioralsciences,rangingfrommedicalsciencesandepidemiologytopsychology,
marketing,andeconomics.
Prof. Dr. Rens van de Schoot works as a Full Professor teaching Statistics for
Small Data Sets at Utrecht University in the Netherlands and as an Extra-ordinary
Professor at North-West University in South Africa. He obtained his PhD cum laude
onthetopicofapplyingBayesianstatisticstoempiricaldata.
Dr. Milica Miočević is an Assistant Professor in the Department of Psychology at
McGill University. She received her PhD in Quantitative Psychology from Arizona
StateUniversityin2017.Dr.Miočević’sresearchevaluatesoptimalwaystouseBayes-
ianmethodsinthesocialsciences,particularlyformediationanalysis.
EUROPEAN ASSOCIATION OF METHODOLOGY
TheEuropeanAssociationofMethodology
(EAM)servestopromoteresearchanddevelop-
mentofempiricalresearchmethodsinthe
fieldsoftheBehavioural,Social,Educational,
HealthandEconomicSciencesaswellasinthe
fieldofEvaluationResearch.
Homepage:www.eam-online.org.
The purpose of the EAM book series is to advance the development and appli-
cation of methodological and statistical research techniques in social and behav-
ioral research. Each volume in the series presents cutting-edge methodological
developments in a way that is accessible to a broad audience. Such books can be
authored, monographs, or edited volumes.
SponsoredbytheEuropeanAssociationofMethodology,theEAMbookseriesis
opentocontributionsfromtheBehavioral,Social,Educational,HealthandEconomic
Sciences. Proposals for volumes in the EAM series should include the following: (1)
title; (2) authors/editors; (3) a brief description of the volume’s focus and intended
audience; (4) a table of contents; (5) a timeline including planned completion date.
Proposalsareinvitedfromallinterestedauthors.Feelfreetosubmitaproposaltoone
ofthemembersoftheEAMbookserieseditorialboardbyvisitingtheEAMwebsite
http://eam-online.org. Members of the EAM editorial board are Manuel Ato (Uni-
versity of Murcia), Pamela Campanelli (Survey Consultant, UK), Edith de Leeuw
(UtrechtUniversity)andVasjaVehovar(UniversityofLjubljana).
Volumes in the series include
Van de Schoot/Miočević: Small Sample Size Solutions: A Guide for Applied
Researchers and Practitioners, 2020
Davidov/Schmidt/Billiet/Meuleman: Cross-Cultural Analysis: Methods and
Applications, 2nd edition, 2018
Engel/Jann/Lynn/Scherpenzeel/Sturgis: Improving Survey Methods: Lessons
from Recent Research, 2015
Das/Ester/Kaczmirek: Social and Behavioral Research and the Internet: Advances
in Applied Methods and Research Strategies, 2011
Hox/Roberts: Handbook of Advanced Multilevel Analysis, 2011
De Leeuw/Hox/Dillman: International Handbook of Survey Methodology, 2008
Van Montfort/Oud/Satorra: Longitudinal Models in the Behavioral and Related
Sciences, 2007
SMALL SAMPLE SIZE
SOLUTIONS
A Guide for Applied Researchers
and Practitioners
Edited by Rens van de Schoot and
̌
Milica Miocevic´
Firstpublished2020
byRoutledge
2ParkSquare,MiltonPark,Abingdon,Oxon,OX144RN
52VanderbiltAvenue,NewYork,NY10017
andbyRoutledge
RoutledgeisanimprintoftheTaylor&FrancisGroup,aninformabusiness
©2020selectionandeditorialmatter,RensvandeSchootandMilicaMiočević;
individualchapters,thecontributors
TherightofRensvandeSchootandMilicaMiočevićtobeidentified
astheauthorsoftheeditorialmaterial,andoftheauthorsfortheir
individualchapters,hasbeenassertedinaccordancewithsections77and
78oftheCopyright,DesignsandPatentsAct1988.
TheOpenAccessversionofthisbook,availableatwww.taylorfrancis.com,
hasbeenmadeavailableunderaCreativeCommonsAttribution-Non
Commercial-NoDerivatives4.0license.
Trademarknotice:Productorcorporatenamesmaybetrademarksor
registeredtrademarks,andareusedonlyforidentificationand
explanationwithoutintenttoinfringe.
LibraryofCongressCataloging-in-PublicationData
Acatalogrecordforthistitlehasbeenrequested
ISBN:978-0-367-22189-8(hbk)
ISBN:978-0-367-22222-2(pbk)
ISBN:978-0-429-27387-2(ebk)
TypesetinBembo
bySwales&Willis,Exeter,Devon,UK
CONTENTS
Introduction viii
RensvandeSchootandMilicaMiočević
Listofsymbols xi
PARTI
Bayesiansolutions 1
1 IntroductiontoBayesianstatistics 3
MilicaMiočević,RoyLevy,andRensvandeSchoot
2 Theroleofexchangeabilityinsequentialupdatingoffindings
fromsmallstudiesandthechallengesofidentifyingexchangeable
datasets 13
MilicaMiočević,RoyLevy,andAndreaSavord
3 AtutorialonusingtheWAMBSchecklisttoavoidthe
misuseofBayesianstatistics 30
RensvandeSchoot,DucoVeen,LaurentSmeets,SonjaD.Winter,
andSarahDepaoli
4 TheimportanceofcollaborationinBayesiananalyses
withsmallsamples 50
DucoVeenandMartheEgberts
vi Contents
5 AtutorialonBayesianpenalizedregressionwithshrinkagepriors
forsmallsamplesizes 71
SaravanErp
PARTII
n=1 85
6 Onebyone:thedesignandanalysisofreplicatedrandomized
single-caseexperiments 87
PatrickOnghena
7 Single-caseexperimentaldesignsinclinicalintervention
research 102
MarijaMaricandVeravanderWerff
8 Howtoimprovetheestimationofaspecificexaminee’s
(n ¼ 1)mathabilitywhentestdataarelimited 112
KimberleyLekandIngridArts
9 Combiningevidenceovermultipleindividualanalyses 126
FayetteKlaassen
10 Goingmultivariateinclinicaltrialstudies:aBayesian
frameworkformultiplebinaryoutcomes 139
XynthiaKavelaars
PARTIII
Complexhypothesesandmodels 155
11 Anintroductiontorestriktor:evaluatinginformativehypotheses
forlinearmodels 157
LeonardVanbrabantandYvesRosseel
12 Testingreplicationwithsmallsamples:applicationstoANOVA 173
MariëlleZondervan-ZwijnenburgandDominiqueRijshouwer
13 Smallsamplemeta-analyses:exploringheterogeneityusing
MetaForest 186
CasparJ.vanLissa
Contents vii
14 Itemparcelsasindicators:why,when,andhowtousethemin
smallsampleresearch 203
CharlieRioux,ZacharyL.Stickley,OmololaA.Odejimi,
andToddD.Little
15 Smallsamplesinmultilevelmodeling 215
JoopHoxandDanielMcNeish
16 Smallsamplesolutionsforstructuralequationmodeling 226
YvesRosseel
17 SEMwithsmallsamples:two-stepmodelingandfactorscore
regressionversusBayesianestimationwithinformativepriors 239
SanneC.SmidandYvesRosseel
18 Importantyetunheeded:somesmallsampleissuesthatare
oftenoverlooked 255
JoopHox
Index 266
INTRODUCTION
̌
Rens van de Schoot and Milica Miocevic´
Researchers often have difficulties collecting enough data to test their hypotheses,
either because target groups are small (e.g., patients with severe burninjuries); data
are sparse (e.g., rare diseases), hard to access (e.g., infants of drug-dependent
mothers),ordatacollectionentailsprohibitivecosts(e.g.,fMRI,measuringphono-
logical difficulties of babies); or the study participants come from a population that
ispronetodrop-out(e.g.,becausetheyarehomelessorinstitutionalized).Suchobs-
tacles may result in data sets that are too small for the complexity of the statistical
model needed to answer the research question. Researchers could reduce the
requiredsamplesizefortheanalysisbysimplifyingtheirstatisticalmodels.However,
thismayleavethe“true”researchquestionsunanswered.Assuch,limitationsassoci-
ated with small data sets can restrict the usefulness of the scientific conclusions and
mightevenhamperscientificbreakthroughs.
Thefieldofmethodologicalsolutionsforissuesduetosmallsamplesizesisdevelop-
ingrapidly,andfastsoftwareimplementationsofthesemethodsarebecomingincreas-
inglyavailable.However,theselectionoftextsonstatisticalmethodsforsmallsample
researchwithcomplexmodelsissparse.InMarch2018,weorganizedthefirstedition
of the Small Sample Size Solutions conference (S4; www.uu.nl/s4) with the goal of
bringingtogetherappliedresearcherswhoencounterissuesduetosmallsamples,and
statisticiansworkingonsolutionstosuchissues.TheaimoftheS4Conferencewasto
share information, learn about new developments, and discuss solutions for typical
smallsamplesizeproblems.Thechaptersinthecurrentvolumedescribesomeofthe
solutions to small sample size issues presented at the first S4 Conference. The list of
contributors includes both established authors who provide an overview of available
methods in a particular field, and early-career researchers working on promising
innovative solutions. The authors ofthe chapters reviewed at least one other chapter
in this volume, and each chapter was written with the goal of being accessible for
applied researchers and students with a basic knowledge of statistics. Note that
Introduction ix
collecting more data, if at all possible, is always preferred, and that the methods
describedinthecurrentbookarealastresort.
The current book provides guidelines and tools for implementing a variety of
solutions to issues that arise in small sample research, along with references for fur-
ther (technical) details. The book includes solutions for estimation of population
means, regression analyses, meta-analyses, factor analyses, advanced structural equa-
tion models with latent variables, and models for nested observations. The types of
solutionsconsistofBayesianestimationwithinformativepriors,variousclassicaland
Bayesian methods for synthesizing data with small samples, constrained statistical
inference, two-step modeling, and data analysis methods for one participant at
a time. All methods require a strong justification of the choice of analytic strategy
andcomplete transparencyaboutallstepsinthe analysis. The bookisaccompanied
by state-of-the-art software solutions, some of which will only be released
next year. All proposed solutions are described in steps researchers can implement
with their own data and are accompanied with annotated syntax in R available on
the Open Science Framework (osf.io/am7pr/). The content of the substantive
applications spans a variety of disciplines, and we expect the book to be of interest
to researchers within and outside academia who are working with small samples
sizes.
The book is split into three parts:
Part I contains several chapters that describe and make use of Bayesian statis-
tics. Chapter 1 offers a gentle introduction to the main ingredients in Bayesian
analyses and provides necessary information for understanding Bayesian param-
eter estimation and Bayes Factors. Chapter 2 offers a discussion of exchangeabil-
ity and its role in the choice of sources of prior information in Bayesian
analyses, which is relevant when combining datasets. Chapter 3 provides an
extension of the When-to-Worry-and-How-to-Avoid-the-Misuse-of-Bayesian-
Statistics (WAMBS) checklist, which is a 10-point checklist used to ensure opti-
mal practices when applying Bayesian methods, extended to include prior and
posterior predictive checking. Chapter 4 illustrates difficulties that can arise
when implementing Bayesian solutions to a complex model and offers sugges-
tions for avoiding these issues by making use of the effective sample size and
divergent transitions. Chapter 5 provides a tutorial on Bayesian penalized regres-
sion for scenarios with a small sample size relative to the complexity of the stat-
istical model by applying so-called “shrinkage priors” that shrink small effects
towards zero while leaving substantial effects large.
Part II is composed of chapters on methods for analyzing data from a single
participant. Chapter 6 introduces single-case experimental designs (n¼ 1) and
provides background information for analyzing a single-case experimental design
(SCED) using unilevel design-based analysis. Chapter 7 discusses SCEDs in
detail and provides an example of tests of effectiveness and change processes.
Chapter 8 introduced a shiny app that allows researchers to supplement test
scores of a single participant with teacher input or scores from other students in
order to obtain a more accurate estimate of a given student’s ability. Chapter 9
