TL;DR
When your outcome is a category (survived/died, infected/not) and you're comparing groups, use a chi-square test of independence. Switch to Fisher's exact test when sample sizes are small (any expected count < 5). For paired categorical data use McNemar's test. Report an odds ratio or risk ratio with its CI as the effect size, not just the p-value.
The setup
Categorical comparisons are organized as a contingency table: counts cross-classified by group and outcome:
| Outcome + | Outcome − | |
|---|---|---|
| Treatment | a | b |
| Control | c | d |
The question is whether the outcome distribution differs between groups (equivalently, whether group and outcome are associated).
Which test
- Chi-square test of independence: the default for comparing proportions across 2+ groups, when sample sizes are reasonable. Rule of thumb: it's reliable when all expected cell counts are ≥ 5.
- Fisher's exact test: use when expected counts are small (any < 5), common in small biology experiments. It computes an exact p-value rather than a large-sample approximation. Fine to use it whenever counts are modest.
- McNemar's test: for paired categorical data. The same subjects measured twice (e.g., a diagnostic result before and after, or two assays on the same samples). Using a plain chi-square here is wrong because the observations aren't independent.
- Chi-square goodness-of-fit: compares observed counts to expected ratios (e.g., testing a 3:1 Mendelian segregation).
Effect size for categorical data
A p-value alone doesn't convey magnitude. Report one of:
- Risk ratio (relative risk): the event probability in one group divided by the event probability in the other, intuitive as "how many times more likely the event is in the treated group than in control."
- Odds ratio: ratio of odds. What logistic regression returns, and standard in case-control designs.
- Risk difference: absolute difference in proportions.
Always with a 95% CI.
Drag the four cell counts and watch the odds ratio, risk ratio, and χ² p-value respond:
Drag the four cell counts and watch the effect sizes move. The odds ratio and risk ratio describe how much exposure shifts the outcome; the p-value alone wouldn't, so report the effect size with it. (χ² is illustrative here, not a substitute for an exact test at small counts.)
Beyond two-by-two: regression
For multiple predictors or to adjust for confounders with a binary outcome, use logistic regression, which yields adjusted odds ratios. For count outcomes (number of events per sample), use Poisson or negative-binomial regression rather than forcing counts into a t-test.
A biology example
In a survival assay, 18/20 treated vs. 10/20 control organisms survive. Cells are small enough that Fisher's exact test is the safe choice. Report the p-value and the effect: e.g., survival risk ratio 1.8 (95% CI 1.2–2.7), so treated organisms were about 1.8× as likely to survive. The ratio plus CI tells the reader the size and precision; the p-value alone wouldn't.
Common mistakes
Running a t-test on proportions or percentages is a common error. A percent-survival value summarizes categorical/count data, so analyze the underlying counts, not the percentages.
- Running a t-test on proportions or percentages. A percent-survival value is summarizing categorical/count data: analyze the underlying counts.
- Chi-square with tiny expected counts. Switch to Fisher's exact.
- Ignoring pairing. Paired categorical data need McNemar's, not chi-square.
- Reporting percentages without the denominators. "80% survived" from n = 5 is very different from n = 500, so always give the counts.
- No effect size. Add an odds/risk ratio with CI.
Related
Effect size and confidence intervals · Correlation and regression · Survival analysis