Philosophy and Psychology L7: Replication crisis, Ioannidis, Discovery & replication paradigms, Prevalence, Power, Improvement

When only reporting significant results instead of reporting non-significant results, this inflates the average power. This is what caused the replication crisis.

Replication crisis:

- significant results (are/are not) reported

- non-significant results (are/are not) reported

- power is (less/more) than 50%

What does a Repicability/R-Index below 50% mean?

R-Index < 50% - suggests that a significant result was obtained only by capitalizing on chance level

Ioannidis: sciencie (is/ is not) necessarily self-correcting

Ioannidis: sciencie is not necessarily self-correcting:

- scientific credibility decreases in some fields, but increases in others (yes/no)

- if fields with (low/high) credibility become dominant, they can hurt overall scientific credibility

- continuous scientific propgress (is/ is not) guaranteed

- stigma of replication (has/ has not) to change

- research is conducted in a few popular scientific fields, but there are no genuine (null/non-null) effects to be discovered

- findings (are/ are not) just the result of random error and bias

"scientific progress is clearly visible in our everyday life"

= (weak/strong) argument

1. practtical progress does not imply efficiency

2. practical progress tells nothing about the proportion of incorrect scientific results

3. linking scientific progress to scientific credibility is tricky: opioid crisis released through opiod research

What are the 4 discovery & replication paradigms?

optimal, self-correcting, false non-replication, perpeptuated fallacy

1) Optimal= to make a correct discovery and correctly replicating it

1) Optimal

2) Self-correcting

3) False-non replication

4) perpeptuated fallacy

3) False-non replication = the original discovery is correct, but the replication is wrong

1) Optimal

2) Self-correcting

3) False-non replication

4) perpeptuated fallacy

2) Self-correcting = the discovery is wrong, but the replication allows to correct the error

1) Optimal

2) Self-correcting

3) False-non replication

4) perpeptuated fallacy

4) perpeptuated fallacy = both replication and discovery are wrong

1) Optimal

2) Self-correcting

3) False-non replication

4) perpeptuated fallacy

4 discovery & replication paradigms

1) Optimal (support/undermine) scientific progress

2) Self-correcting (support/undermine) scientific progress

3) False-non replication (support/undermine) scientific progress

4) perpeptuated fallacy (support/undermine) scientific progress

Prevalence: match 1,2,3,4 with A, B, C, D

1) Optimal A: direct replications are uncommon, only a fraction are successful

2) Self-correcting C: clear refutations of proposed discoveries are even less common than successful replications

3) False-non replication B: are very rare

4) perpeptuated fallacy D: probably most dominant paradigm

A: direct replications are uncommon, only a fraction are successful

B: are very rare

C: clear refutations of proposed discoveries are even less common than successful replications

D: probably most dominant paradigm

power = the ability of a study to detect an effect, given that an effect exists

power = the ability of a study to detect an effect, given that an effect exists

small sample sizes:

- (are/ are not) often statistically significant, and therefore (less/more) likely to be published

- (does/ does not) reflect the true effect

- the true effect is the (null/non-null) effect

- flexible study design, flexible statistcal analyses & small size have (low/high) statistical power

2 problems with low power studies:

1. statistical problems, even when research is conducted without bias, the to create (negative/positive) results

2. problems that reflect bias: (do/ do not) become worse in small, underpowered studies

smaller studies can only detect (large/small) effects and (are/are not) susceptible to inflated effect size estimates, selective data analyses and publication bias

What is true?

1: In research Type I error is considered more often than Type II

2: In research Type II error is considered more often than Type I

How can science improve?

- perform an a priori calculation

- disclose methods and findings transparently

- pre-register your study-protocol and analysis plan

- make study material available

- work collaboratively  to increase power and replicate findings

correlation = the actual cause is probably a third variable

causation = causal relation between correlated variables can be established by random control trial of 2 groups of subjects which receive different treatment

identifiying a causal effect of X on Y requires:

1. (causation/correlation): empirical association between X and Y

2. temporal priority of variable (X/Y)

3. (spuriousness/ non-spuriousness): no confounding 3rd variable that causes X and Y

Feyerabend:

- science (should/ should not) be seperated from the state

- sciencie (is/ is not) rational, and therefore it (is/ is not) privileged

- parents (should/ should not) be able to determine their childrens options, they (should/ should not) be bound to scientific standards

- science (should/should not) be subjected to democtratic control: research topics (should/ should not) be determined by popular election

Dutch National Research Agenda = 1. reveal issues challening dutch society 2. show where research should focus on

What are problems with Null Hypothesis Significance Testing (NHST)

1. no information about effect size

2. no information about the alternative hypothesis

3. no information about significance threshold

4. all of the above