Reproducibility and error
In this lesson
By the end, you’ll be able to
- Say what a replication actually needs and distinguish the senses of reproducibility
- Read a funnel plot and reason about what the published literature is not showing
- Audit a study, including undisclosed AI-assisted steps, for what would let a stranger reproduce it
Sign in to save your progress
You can keep reading without an account, but completed lessons won’t be saved.
A finding that no one else can reproduce is a rumor with a p-value. Over the last two decades, large efforts to re-run published studies have found that a substantial share do not hold up: effects shrink or vanish. This is not a scandal about a few bad actors; it is the predictable result of publication pressure, analytic flexibility, and selective reporting acting on ordinary, honest scientists. Reproducibility is the property that turns a claim into knowledge, and it has to be built in, not hoped for.
‘Reproducible’ is used for several different things, and confusing them causes needless arguments. Methods reproducibility: given the same data and code, can someone re-run the analysis and get the same numbers? Results reproducibility (replication): does an independent new study, with new data, find the same effect? Inferential reproducibility: would others draw the same conclusion from the evidence? The first needs shared materials: the dataset, the analysis code, software versions, a random seed. The second needs enough procedural detail to run the study again. A paper that reports its methods only as ‘standard statistical methods’ fails all three, no matter how good the writing.
Categorize
A stranger wants to reproduce a study's reported numbers from scratch. Sort each item by whether it actually enables reproduction, or cannot do so on its own.
Could you re-run the numbers with this alone? Data, code, and settings, yes; a PDF or image, no.
Signature interactive
Replication Audit
A paper claims an AI tutor “significantly improved essay scores.” You are the replicator. For each artifact, read what the paper actually says and judge whether it is truly provided or effectively missing. Then complete the audit.
The raw dataset
“Data are available from the authors on reasonable request.”
The analysis code
Posted in a public repository, linked in the paper.
Software versions & random seed
Not reported anywhere in the paper or repository.
The AI model, version, and prompts
The pipeline used ‘an LLM’ to clean data; no model, version, or prompts are named.
A pre-specified analysis plan
No preregistration; the paper reports one significant subgroup.
The outcome measure & scoring rubric
Described in full in an appendix, with inter-rater agreement.
Constructed example. The point is the reasoning, not this specific paper.
Even a shelf of well-run studies can mislead you if you are only seeing some of them. Publication bias is the systematic tendency for positive, significant results to get published while null results stay in the file drawer. The consequence is that the visible literature overstates effects, sometimes badly, and a naive average of published studies inherits the inflation. A funnel plot is one diagnostic: plot each study's effect against its precision, and if small studies cluster only on the positive side, leaving an empty corner where small null studies should be, you are likely looking at a filtered record. The fix upstream is the same discipline as before: preregistration and a culture that publishes null results.
Exercise
A meta-analysis plots each study's effect against its precision and finds the small studies are almost all on the 'positive' side, leaving a gap where small null studies should be. What does this asymmetry most suggest?
If null small studies go unpublished, which corner of the funnel goes empty?
Written response
Lab (constructed study). A paper reports that an AI writing tutor 'significantly improved essay scores.' It gives no link to the data, describes the analysis only as 'standard statistical methods,' does not say which model or prompts the tutor used, and notes it is the team's fourth study on the tool (the first three are unpublished). Write a short reproducibility audit: what would a stranger need to reproduce this, what are the most serious threats, and what does the mention of three unpublished studies imply?
Missing materials (data, code, model/prompts, rubric) → threats (vague methods, undisclosed AI) → what the unpublished studies imply (file drawer).
0/70 words
Checkpoint · item 1 of 5
A colleague says 'our study is reproducible, it's published in a good journal.' What is the problem with this claim?
Does peer review actually re-run the analysis on the data?
Reflection