What does a p-value actually mean?

TL;DR

A p-value is the probability of seeing data as extreme as yours (or more) if the null hypothesis were true. It is not the probability that the null is true, not the probability your result is a fluke, and not a measure of effect size. A small p tells you the data are surprising under "no effect"; it says nothing about how big or important the effect is.

What a p-value is

Imagine there's truly no difference between your treated and control groups (the null hypothesis). You'd still see some difference in any given experiment, just from sampling noise. The p-value answers: if there were really no effect, how often would random sampling alone produce a difference at least as large as the one I observed?

p = 0.03 → a difference this large (or larger) would happen about 3% of the time by chance alone if nothing were going on.
That's suggestive that something real is happening, but 0.05 is a convention, not a law of nature.

Slide the observed test statistic below and watch the shaded tail: the p-value is that area under the null curve:

Test statistic z2.00

The shaded area under the null curve beyond your test statistic is the p-value: the chance of a result this extreme if there were no real effect. Slide the statistic right and watch the tail and p both shrink. p is a tail area, not the probability the null is true.

What a p-value is NOT

This is where most papers and reviewers slip. A p-value does not tell you:

The probability the null hypothesis is true. p = 0.03 is not "97% chance the effect is real." That's a different quantity that depends on how plausible the effect was to begin with.
The probability your finding is a false positive. The real false-positive rate depends on power and on how many hypotheses you tested.
How large the effect is. A tiny, biologically meaningless difference can give a tiny p-value if your sample is huge. A large, important effect can give p > 0.05 if your sample is small. The p-value confounds effect size and sample size, which is exactly why you also report effect size and confidence intervals.
Whether the result matters. Statistical significance ≠ biological significance.

p > 0.05 does not mean "no effect." Absence of evidence isn't evidence of absence; it often just means your study was underpowered.

"Significant" vs. "not significant"

A non-significant result is not proof of "no effect"; it often just means the study was underpowered → power and sample size.
A hard 0.05 cutoff is arbitrary. p = 0.049 and p = 0.051 are essentially the same evidence. Report the actual p-value rather than just "p < 0.05," and interpret it alongside the effect size.

A biology example

You compare colony counts in treated vs. control plates and get p = 0.04. Correct reading: "If the treatment had no effect, we'd see a difference this large about 4% of the time by chance." Then you look at the effect size: the means differ by 8%, with a 95% CI of 1%–15%. Now the reader knows both that the result is unlikely under no-effect and roughly how big the effect is and how precisely you've pinned it down. The p-value alone would have hidden that the effect could be as small as 1%.

Common mistakes

"The p-value is the chance the result is wrong." No. See above.
P-hacking: trying many tests/subgroups/exclusions and reporting the one that crossed 0.05. Decide your analysis in advance.
Optional stopping: peeking at the data and adding samples until p dips below 0.05. This guarantees false positives because p bounces around as n grows.
HARKing (Hypothesizing After Results are Known): finding a pattern, then presenting it as if it were the original hypothesis. This is a hidden form of multiple comparisons → multiple comparisons.
Reporting only "p < 0.05" with no effect size or CI. Reviewers increasingly reject this.

Effect size and confidence intervals · Power and sample size · Multiple comparisons

This topic is widely debated among statisticians; the guidance here reflects mainstream practice (e.g., the American Statistical Association's statement on p-values).