Null hypothesis p-value Chi-square test G-test t-test ANOVA Non-parametric tests Statistical power

58 Pages·2017·14.14 MB·English

t-test
Null hypothesis
p-value
Chi-square test
G-test
Non-parametric tests ANOVA
Statistical power
Multiple test corrections
WRONG
t-test
Null hypothesis
p-value
Chi-square test
G-test
Non-parametric tests ANOVA
WRONG
Statistical power
Multiple test corrections
power = 80% 80% detected False positive rate
80 true positives
false positives
real effect in 10% 𝐹𝑃𝑅 =
no effect
100 tests
20% not detected 45
𝐹𝑃𝑅 = = 0.05
Pr(real) = 0.1 20 false negatives
900
1,000 tests
95% tested negative False discovery rate
855 true negatives
false positives
no effect in 90% 𝐹𝐷𝑅 =
discoveries
900 tests
5% “detected” 45
𝐹𝐷𝑅 = = 0.36
significance level = 5%
45 false positives
45 + 80
Colquhoun D., 2014, “An investigation of the false discovery rate
and the misinterpretation of p-values”, R. Soc. open sci. 1: 140216.
power = 80% 80% detected
80 true positives
real effect in 10%
100 tests
20% not detected
Pr(real) = 0.1 20 false negatives If you publish a 𝑝 < 0.05
result, you have a 36%
1,000 tests
chance of making a fool
95% tested negative
of yourself
855 true negatives
no effect in 90%
900 tests
5% “detected”
significance level = 5%
45 false positives
Colquhoun D., 2014, “An investigation of the false discovery rate
and the misinterpretation of p-values”, R. Soc. open sci. 1: 140216.
What’s wrong with p-values?
Marek Gierliński
Division of Computational Biology
Hand-outs available at http://is.gd/statlec
❌
❌
❌
Statistical testing
Statistical model
Data
Statistical test against H
0
Null hypothesis
H : no effect
0
All other assumptions
Significance level
𝛼 = 0.05
p-value: probability that the
observed effect is random
𝑝 < 𝛼 𝑝 ≥ 𝛼
Reject H Insufficient evidence
0
(at your own risk)
Effect is real
p-value:
Given that H is true, the probability of
0
observed, or more extreme, data
It is not the probability that H is true
0
P-value is the degree to which the data are
embarrassed by the null hypothesis
Nicholas Maxwell
“All other assumptions”
All other assumptions
about biology are
Instruments correct NNoo ootthheerr eeffffeeccttss
calibrated
Null hypothesis
Data collected
H : no effect
0
correctly
Experimental
Significance level
protocols followed 𝛼 = 0.05
No silly misakes
𝑝𝑝 << 𝛼𝛼 𝑝 ≥ 𝛼
RReejjeecctt HH Do not reject H
00 0