Inference and its limits

Statistics, in a paper, is not mathematics for its own sake; it is a discipline of honesty under uncertainty. It gives you tools to say how much the data should move your belief, and, just as importantly, how easily you can fool yourself. Almost every tool in this lesson is more often misread than read. The goal is not to compute; it is to interpret without deceiving yourself or your reader.

A p-value answers one narrow question: if there were truly no effect, how surprising would data at least this extreme be? A small p-value means the data sit awkwardly with the null hypothesis; nothing more. It is not the probability that the null is true, not the probability your result is a fluke, and not one-minus-the-probability the effect is real. Those readings all smuggle in the very thing a p-value cannot give you: the probability of a hypothesis. And 'p < 0.05' is a convention, not a law of nature; it is a threshold someone chose, not a boundary between real and unreal.

Significance and size are different questions. A p-value tells you whether an effect is distinguishable from zero; an effect size tells you how big it is; a confidence interval tells you the range of effects reasonably compatible with your data. Sample size drives significance, so in a big enough study a trivial difference can reach p < 0.001 and still not matter to anyone. Always ask for the magnitude and its interval, and read the interval as a compatibility range, not as a 95% probability that the truth sits inside this particular one.

Now the trap that undoes more findings than any other. Every analysis involves choices: which subgroup, which outcome, which cutoff, which covariates. Each defensible choice is a fork, and a researcher who decides after seeing the data can wander down whichever path reaches significance, often without any sense of wrongdoing. This is the garden of forking paths, and its arithmetic is brutal: run twenty independent tests on data where nothing is real and you have about a 64% chance of at least one 'significant' result. A finding cherry-picked from many comparisons is not strong evidence; it is the expected noise. The discipline that defeats it is to fix your one primary analysis before the data arrive, and to label everything else as exploration.