Do I need to correct for multiple comparisons?

TL;DR

Every test you run at α = 0.05 has a 5% chance of a false positive. Run many tests, and false positives pile up fast. Correct for it. Use Bonferroni / family-wise methods when any single false positive is costly (a few key confirmatory comparisons); use FDR (Benjamini–Hochberg) for large-scale screens (omics) where you want to control the proportion of false discoveries.

The problem

Test 20 independent hypotheses at α = 0.05 with nothing real going on, and you expect about one "significant" result purely by chance. Test 1,000 genes and you expect ~50 false hits. Without correction, the more you look, the more spurious "findings" you generate. This is the engine behind a lot of irreproducible biology.

It hides in many forms: pairwise comparisons after ANOVA, testing many genes/proteins/metabolites, multiple time points, multiple subgroups, and HARKing (searching for any significant pattern, then presenting it as the original hypothesis) → p-values.

Two philosophies of correction

Use family-wise methods (Bonferroni, Holm) for a few confirmatory comparisons where any false positive is costly. Use FDR (Benjamini–Hochberg) for large screens where you want a reliable shortlist, not zero false positives.

1. Control the family-wise error rate (FWER): the probability of even one false positive across the whole family of tests.

Bonferroni: compare each p-value to α / m (m = number of tests), or equivalently multiply p-values by m. Simple and very conservative: it controls false positives well but costs power, especially with many tests.
Holm / Šidák: stepwise refinements that are uniformly more powerful than plain Bonferroni while still controlling FWER. Prefer Holm over Bonferroni in most cases.
Tukey / Dunnett: purpose-built FWER control for ANOVA post-hoc comparisons → ANOVA.

Use FWER methods when the number of comparisons is small and pre-specified, or when a single false claim would be serious (confirmatory studies).

2. Control the false discovery rate (FDR): the expected proportion of false positives among the results you call significant.

Benjamini–Hochberg (BH): the standard. Much more powerful than Bonferroni when you have many tests, and well suited to discovery/screening.
Reported as q-values or "adjusted p-values." A BH q = 0.05 means you accept that ~5% of your called-significant hits are expected to be false.

Use FDR for large-scale exploratory work, such as differential gene expression, proteomics, and GWAS-scale screens, where the goal is a reliable shortlist, not zero false positives.

How to run it

R: p.adjust(pvals, method = "holm") or method = "BH". In genomics, limma, DESeq2, and edgeR apply BH automatically and report adjusted p-values.
GraphPad Prism: post-hoc dialogs offer Tukey, Dunnett, Holm–Šidák, and FDR (Benjamini–Hochberg) options.
Python: statsmodels.stats.multitest.multipletests(pvals, method="fdr_bh").

A biology example

RNA-seq comparing two conditions tests ~20,000 genes. At raw p < 0.05 you'd expect ~1,000 false positives, which is uninterpretable. Apply Benjamini–Hochberg and report genes at q < 0.05: now you control the expected false-discovery proportion among your hits, giving a defensible candidate list. Using Bonferroni here would be so conservative you'd miss most true signals.

Common mistakes

No correction at all when running many comparisons: the default failure mode.
Bonferroni on a genome-wide screen, which is far too conservative; use FDR.
FDR on a single confirmatory comparison. When one false positive really matters, control FWER.
Correcting only the comparisons that "didn't work" while leaving favorable ones uncorrected. Count all the tests you actually ran.
Hidden multiplicity: subgroups, time points, and exploratory re-analyses all count. Pre-register or at least pre-specify the primary comparisons.

P-values · ANOVA · Power and sample size