Which statistical test should I use? Complete decision tree

If you arrived here with your spreadsheet ready and the feeling that any choice you make might be wrong, here is the good news: choosing a statistical test is not a matter of opinion or of your supervisor's preference. It is a decision determined by four objective features of your study. Once you identify them, the test almost picks itself.

What usually causes confusion is the order in which people attack the problem. The typical mistake is to start by asking "which test is best?" — as if there were a quality ranking. There isn't. The t-test is not superior to Mann-Whitney; they answer the same question under different assumptions. Running a t-test on data that violate normality is not using a better tool: it is using the wrong tool.

The four decision criteria

Criterion 1 — The type of your outcome variable

The outcome (or dependent) variable is what you measured as a result. If you want to know whether training improves performance, the outcome is performance, not training.

• Numeric, continuous or discrete: weight in kilograms, time in seconds, a score from 0 to 10, number of correct answers, serum concentration. Means and standard deviations are meaningfully interpretable.
• Categorical nominal: pass or fail, symptom present or absent, type of treatment chosen. Only counts and proportions apply — computing a mean here is meaningless.
• Ordinal: Likert scales from 1 to 5, disease stages, socioeconomic classification. Order exists, but the intervals between categories are not necessarily equal — the distance between "agree" and "strongly agree" is not the same as between "disagree" and "neutral".

The ordinal case generates the most frequent debate. The prevailing practice in health and social sciences is to treat single Likert items as ordinal (using non-parametric tests) and summed scores from multiple items as numeric — for example, the total of the 21 items of the Beck Depression Inventory can legitimately be treated as a continuous variable, even though each item is ordinal. If your committee is conservative on this point, the safe route is the non-parametric test.

Criterion 2 — The goal of the analysis

Three goals cover almost every undergraduate and master's project:

• Compare. "Does the group that trained perform differently from the control group?" You have one variable that splits the sample and one that measures the result.
• Associate. "The more hours studied, the higher the grade?" Two variables measured in the same individuals, with no group split and no causal claim.
• Predict. "How much does the grade increase per additional hour of study?" You want an equation, not merely evidence that a relationship exists.

Note that "associate" and "predict" use the same data but answer different questions and produce different results. We will return to this.

Criterion 3 — Number of groups and pairing

Comparing two groups and comparing three groups are not the same operation with a larger number. Running three t-tests across three groups is one of the most serious and most common errors in student research. The reason is the accumulation of Type I error: with α of 5% per comparison, the chance of at least one false positive across three tests rises to roughly 14%. With six comparisons it reaches 26%. That is why ANOVA exists — it tests the global hypothesis with a single α, and only then, if significant, do you apply a post hoc test corrected for multiple comparisons.

Pairing is the second axis. Groups are paired when observations correspond one to one: the same participants measured before and after an intervention, twins, the right and left eye of the same patient, cases and controls matched on age and sex. They are independent when different people make up each group. Paired tests are substantially more powerful because they remove between-subject variability — using an independent test on paired data wastes information and reduces your chance of detecting a real effect.

Criterion 4 — Normality

Parametric tests assume the data follow an approximately normal distribution. This is formally checked with the Shapiro-Wilk test, complemented by visual inspection of the Q-Q plot. The logic of the result is counterintuitive and worth memorising:

Here the "good" p-value is the high one — the opposite of what you want in your main test. The null hypothesis of Shapiro-Wilk is "the data come from a normal distribution"; rejecting it means the assumption is violated.

Two important caveats. First: test normality within each group, not on the pooled sample — two groups with different means produce a bimodal distribution that fails the test even when each group is perfectly normal. Second: with very large samples (n above 300) Shapiro-Wilk becomes hypersensitive and rejects normality for trivial deviations with no practical consequence. In those cases the Q-Q plot, skewness and kurtosis are more reliable guides than the p-value. Check this on the normality test page.

The decision table

The errors reviewers flag most often

Multiple t-tests instead of ANOVA

Detailed above. If you compared three or more groups with pairwise t-tests and no correction, the work has a real statistical problem — not a stylistic one. The fix is to rerun with ANOVA (or Kruskal-Wallis) and apply the post hoc test only if the global result is significant.

Confusing paired with independent

A pre/post study with the same twenty participants has twenty participants, not forty. Treating the forty measurements as independent artificially inflates n, violates the independence assumption and produces optimistic, invalid p-values.

Testing normality on the pooled sample

If you have two groups with different means, the pooled distribution will be bimodal and fail Shapiro-Wilk even when each group is normal. Always test separately.

Reporting the p-value alone

This is the most universal error and the easiest to fix. The p-value tells you whether an effect is detectable, not whether it is meaningful. With a large enough sample, clinically irrelevant differences reach p < .001. Always report the effect size (Cohen's d, eta squared, r) and the confidence interval. APA 7 requires effect sizes, and reviewers increasingly ask for them.

Concluding "there is no difference" from p > .05

Absence of evidence is not evidence of absence. A non-significant result may reflect a genuine absence of effect or a sample too small to detect it. The correct wording is "no statistically significant difference was observed", accompanied by a discussion of statistical power. Check this on the power calculator.

Claiming causality from correlation

An r of .80 between hours studied and grades does not prove that studying causes high grades — a confounding variable (motivation, socioeconomic status, family support) may influence both, or the relationship may run the other way. Causality requires an experimental design with randomisation, not merely a high coefficient. Linear regression does not establish causality either: the term "predictor variable" describes the mathematical structure, not a demonstrated causal relationship.

Correlation or regression?

The same data, two distinct questions:

• Correlation answers "is there a relationship, how strong, in which direction?" It is symmetric: correlating X with Y gives exactly the same result as Y with X. It produces a coefficient between −1 and +1.
• Regression answers "what is the expected value of Y for a given X?" It is asymmetric: swapping predictor and outcome changes the equation. It produces coefficients in interpretable units — "each additional hour of study is associated with 0.42 points on the grade".

If your research question is "is there a relationship between A and B", use correlation. If it is "A explains B" or "how much does B change when A changes", use regression. Reporting both is redundant in almost every case — choose according to your stated objective.

Cases that require more advanced analysis

Recognising the limits of basic tools is a sign of methodological maturity, not weakness. Seek specialist guidance if your study involves:

• Two or more simultaneous independent variables → factorial ANOVA, multiple regression
• The need to control for confounders → ANCOVA, multiple regression
• A binary outcome with multiple predictors → logistic regression
• Time to event, with loss to follow-up → Kaplan-Meier, Cox regression
• Nested data (students within classes within schools) → mixed-effects models
• Instrument validation → factor analysis, Cronbach's alpha

Practical checklist

1. Identify and write down your outcome variable and its type explicitly.
2. State the question in one sentence: compare, associate or predict.
3. Count the groups and check whether they are paired or independent.
4. Test normality within each group using Shapiro-Wilk.
5. Consult the table above and run the test.
6. Report the test statistic, degrees of freedom, p-value, effect size and confidence interval.
7. Interpret practical magnitude, not only statistical significance.

References

• American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.).
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
• Field, A. (2018). Discovering statistics using IBM SPSS Statistics (5th ed.). Sage.
• Royston, P. (1995). Remark AS R94: A remark on Algorithm AS 181. Applied Statistics, 44(4), 547–551.
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values. The American Statistician, 70(2), 129–133.