SC QUALITY GUILDHypothesis test selector
SCQUALITYGUILD.COM/CALCULATORS/HYPOTHESIS-TEST-SELECTOR.HTML
Four questions and you have the right test, with the reasoning and the conditions it depends on. Choosing wrong is more common than computing wrong.
ANSWER FOUR QUESTIONS, GET THE RIGHT TEST · FREE · NOTHING YOU TYPE LEAVES YOUR BROWSER
Q-BotClick for an explanation of this calculator.Hypothesis test selector
ANSWER FOUR QUESTIONS, GET THE RIGHT TESTChoosing the wrong test is more common than computing the right one incorrectly, and it is examined directly on the Green Belt, Black Belt and Quality Engineer papers. Work down the tree; the reasoning is printed with the answer.
For educational use. This page is a study aid. The method behind it has far more depth and detail than one page and one tool can cover, and what you see here is not everything the method offers or everything this site offers. For full understanding and application, use the governing standards and texts, and seek further education, training or qualified advice. Results depend on the data you enter, so check them before relying on them.
The four questions
Almost every routine comparison in quality work is settled by four answers. The software will compute whatever you ask it to; the skill is asking for the right thing.
- What kind of data? Continuous measurements, or counts and proportions. This alone eliminates most of the list.
- What are you comparing? One group against a target, two groups against each other, or more than two.
- Are the observations paired? The same parts measured before and after, or the same parts by two gauges, are paired — and pairing is information you throw away if you ignore it.
- What can you assume? Normality, equal variances, independence. Every test rests on something, and the assumption is usually what fails rather than the arithmetic.
Comparing more than two groups is where it goes wrong
The most common serious error is not picking a t-test over a z-test. It is comparing several groups by running t-tests on every pair.
Each test carries its own five percent chance of a false alarm. Run several and those chances accumulate:
| Groups compared | Pairwise tests | Chance of at least one false positive |
|---|---|---|
| 2 | 1 | 5.0% |
| 3 | 3 | 14.3% |
| 4 | 6 | 26.5% |
| 5 | 10 | 40.1% |
| 6 | 15 | 53.7% |
Comparing six suppliers pairwise, you are more likely than not to find a difference that is not there. You will then spend real money chasing it.
The fix is to ask the question once. ANOVA tests whether any of the groups differ while holding the overall error rate at five percent, and only if it fires do you go looking at pairs — with a method that accounts for how many comparisons you are making.
A p-value above 0.05 does not mean the groups are the same. It means this data was not sufficient to show a difference. With a small enough sample you will fail to detect almost anything, and "no significant difference" quietly becomes "we did not look hard enough". If the result matters, the question to ask is what difference the study was actually powered to find.
What the common tests assume
- One-sample t. A mean against a target. Assumes roughly normal data, or enough of it that the mean is normal anyway.
- Two-sample t. Two independent groups. Classically assumes equal variances; Welch's version does not and is the safer default when you are unsure.
- Paired t. Before and after on the same items. Uses the differences, and is far more powerful than the two-sample test when the pairing is real. Using the wrong one here throws away the whole benefit of the design.
- ANOVA. Three or more groups. Normality and equal variances across groups.
- Chi-square. Counts in categories. Needs adequate expected counts per cell, not just adequate totals.
- Proportions test. Pass and fail counts. Needs enough of both; it behaves badly when one is very rare.
- Non-parametric equivalents — Mann-Whitney, Wilcoxon, Kruskal-Wallis — when normality genuinely fails and you cannot get more data. They ask a slightly different question, which is worth knowing before you quote the result.
Where these go wrong in practice
- The test was chosen after seeing the data. Trying tests until one returns a p-value you like is the statistical equivalent of re-sampling a rejected lot.
- Statistical significance read as practical significance. With 5,000 parts you can prove a difference of a micron. It does not follow that anyone cares.
- Independence assumed and absent. Consecutive parts off one machine are not independent observations, and no test on this list survives that.