Alison Ledgerwood UC Davis

13 Pages·2015·5.74 MB·English

D G M :
ISTINGUISHING OALS AND EANS
T C P -R
HE ASE OF RE EGISTRATION
Alison
 Ledgerwood

UC
 Davis
Goals to
Means: Specific
improve our
research
science
practices
Distinguish high vs.
low confidence
findings
WHAT
 MAKES
 US
 CONFIDENT
 IN

OUR
 OWN
 RESULTS?

•  Given
 what
 we
 now
 know
 about
 power,

p-­‐hacking,
 sequenKal
 analyses,
 etc.

•  When
 do
 we
 trust
 our
 own
 findings
 a
 lot
 vs.

more
 tentaKvely?

•  High
 confidence
 vs.
 low
 confidence
 findings

•  High
 confidence:
 I’d
 bet
 money
 I
 could
 replicate
 it

•  Lower
 confidence:
 Let’s
 replicate
 before
 beSng
DEVELOPING
 LAB
 BEST
 PRACTICES:

THE
 PROCESS

1.  Learn
 about
 and
 criKcally
 evaluate
 the

suggesKons
 (lab
 meeKngs/journal
 clubs)

2.  Develop
 a
 set
 of
 shared
 (and
 sKll
 evolving)

best
 pracKces
 for
 our
 lab

•  What
 maximizes
 the
 info
 we
 get
 from
 our

work?

•  What
 are
 our
 resource
 limitaKons?

•  What’s
 comfortable
 and
 feasible
 right
 now?
LEDGERWOOD
 LAB
 BEST
 PRACTICES:

SAMPLE
 SIZES

•  Set
 target
 n’s
 ahead
 of
 Kme

•  Power
 analysis
 when
 possible

•  Based
 on
 previous
 findings,
 meta-­‐analysis,
 etc.

•  Consider
 power
 analyses
 that
 take
 into

account
 effect
 size
 heterogeneity
 and

publicaKon
 bias

(see
 McShane
 &
 Bockenholt,
 2014
 PPS;
 Perugini
 et
 al.,
 2014
 PPS)

•  Otherwise,
 minimum
 target
 cell
 size
 n
 =
 50
WHY
 N
 =
 50?

•  Completely
 arbitrary

•  But,
 beher
 than
 n
 =
 20

•  Power
 to
 detect
 a
 small
 effect:
 Less
 than
 10%

•  When
 power
 dips
 below
 10%,
 you’re
 not
 only
 likely

to
 overesKmate
 the
 effect
 size…you
 may
 get
 the

sign
 wrong
 (Gelman
 &
 Carlin,
 2014)

•  Power
 to
 detect
 a
 medium
 effect:
 34%

•  When
 power
 gets
 below
 50%,
 start
 having
 effect

size
 exaggeraKon
 problems
 (i.e.,
 results
 are
 less

informaKve).

See Gelman & Carlin, 2014 PPS for more on Type S and Type M errors
WHY
 N
 =
 50?

•  n
 =
 50

•  Power
 to
 detect
 a
 small
 effect:
 17%

(But
 at
 least
 avoid
 sign
 problems)

•  Power
 to
 detect
 a
 medium
 effect:
 70%

(Avoid
 exaggeraKon
 problems)

See Gelman & Carlin, 2014 PPS for more on Type S and Type M errors
DEVELOPING
 LAB
 BEST
 PRACTICES:

THE
 PROCESS

1.  Lab
 meeKngs/discussions
 about
 arKcles
 and

conference
 presentaKons

2.  Develop
 a
 set
 of
 shared
 (and
 sKll
 evolving)

best
 pracKces
 for
 our
 lab

•  What
 maximizes
 the
 info
 we
 get
 from
 our

work?

•  What
 are
 our
 resource
 limitaKons?

•  What’s
 comfortable
 and
 feasible
 right
 now?

3.  What
 infrastructure
 will
 make
 this
 easy?
EXPERIMENT
 ARCHIVE
 FORM

Available at http://ledgerwood.faculty.ucdavis.edu/resources