P-values and statistical significance: comparing two results with a t-test
What a p-value is and isn't, what statistical significance means, and how a t-test compares two sets of lab results, with a worked HPLC example.
By Zachery Casey · Published · Reviewed · Editorial policy
A p-value is the probability of seeing a difference at least this large if there were no real difference. Two sets of measurements almost never give the same average. Two lots tested by HPLC might average 99.0% and 98.7%. Is that a real difference, or just the normal scatter of the method? A t-test is the standard way to ask that question, and the p-value is the number it produces. This guide explains what a p-value means, what it doesn't mean, and how to run a simple t-test on two sets of lab results.
It follows on from confidence intervals, standard error and error bars and accuracy, precision and significant figures. Terms are defined in the glossary.
What is a p-value?
Every statistical test starts from a null hypothesis: a plain assumption that there is no real difference. For two sets of purity results, the null hypothesis is that both come from the same underlying value, and any gap is just measurement scatter.
The p-value answers one question. If the null hypothesis were true, how likely would it be to see a gap at least as big as the one measured? A small p-value means the data would be surprising if there were no real difference. A large one means the gap is the kind of thing normal scatter produces all the time.
The American Statistical Association put it this way in 2016: a p-value shows how incompatible the data are with a stated statistical model (ASA, 2016).
What a p-value doesn't tell you
The same statement lists the common misreadings (ASA, 2016):
- It isn't the chance the hypothesis is true. A p-value of 0.03 doesn't mean there is a 3% chance the two results are the same.
- It isn't the chance the result is a fluke. It is worked out on the assumption that the null hypothesis is true, so it can't also measure that.
- It doesn't measure how big a difference is. A tiny, unimportant gap can give a very small p-value if there are enough measurements.
- It isn't a pass mark. A result at p = 0.049 is not meaningfully different from one at p = 0.051.
What is statistical significance?
A result is called statistically significant when its p-value falls below a threshold chosen in advance. The most common threshold is 0.05. Below it, the gap is judged unlikely to be scatter alone.
The 0.05 line is a convention, not a law of nature. The ASA statement warns against treating it as a bright line that decides a question on its own (ASA, 2016). Report the actual p-value, the size of the difference and the number of measurements, rather than just "significant" or "not significant".
The t-test
A t-test compares two means and asks whether the gap between them is large compared with the scatter in the data. The two-sample version, for two independent groups of measurements, works like this (NIST/SEMATECH e-Handbook):
- Work out the mean and standard deviation of each group.
- Combine the two standard deviations into one pooled standard deviation.
- Divide the difference between the means by the pooled standard deviation, scaled for the number of measurements. The result is the t statistic.
- Look up how likely a t value that large would be if there were no real difference. That gives the p-value.
Written as a formula, with means x̄₁ and x̄₂, group sizes n₁ and n₂ and pooled standard deviation sp:
t = (x̄₁ − x̄₂) ÷ (sp × √(1/n₁ + 1/n₂))
where sp² = [(n₁ − 1)s₁² + (n₂ − 1)s₂²] ÷ (n₁ + n₂ − 2) (NIST/SEMATECH e-Handbook).
The degrees of freedom for this test are the total number of measurements minus two. They decide which t-distribution to use.
A worked example
Two lots are each tested five times by HPLC. These figures are an illustration, not results for any real lot.
| Lot A | Lot B | |
|---|---|---|
| Results (%) | 98.9, 99.1, 99.0, 99.2, 98.8 | 98.6, 98.8, 98.5, 98.9, 98.7 |
| Mean | 99.0% | 98.7% |
| Standard deviation | 0.158 | 0.158 |
| Number of results | 5 | 5 |
- Difference between the means: 99.0 − 98.7 = 0.30 percentage points.
- Pooled standard deviation: both groups have the same spread here, so it is 0.158.
- Standard error of the difference: 0.158 × √(1/5 + 1/5) = 0.100.
- t statistic: 0.30 ÷ 0.100 = 3.0.
- Degrees of freedom: 5 + 5 − 2 = 8.
- p-value: for t = 3.0 with 8 degrees of freedom, the two-tailed p-value is about 0.017.
At the usual 0.05 threshold, this gap would be called statistically significant. With five results each, a 0.3-point gap is larger than the scatter would usually produce.
Note what the test can't settle. It assumes both lots were tested by the same method, under the same conditions, by the same lab. If they came from different labs, the gap could reflect a difference between the labs rather than the lots. See accuracy, precision and significant figures on reproducibility.
Which t-test? One-sample, paired, two-sample and Welch's
The worked example uses one kind of t-test. The right one depends on how the data were collected (NIST/SEMATECH e-Handbook, two-sample test; one-sample test):
| Test | Use it when | Example |
|---|---|---|
| One-sample | Comparing one set of results with a fixed value | Do five injections differ from a reference value of 99.0%? |
| Paired | Each result in one group matches one result in the other | The same five samples tested by two methods; the test works on the differences |
| Two-sample, equal variances | Two independent groups with similar spread | The two-lot example above |
| Welch's (unequal variances) | Two independent groups whose spread differs | Two labs, one with much more scatter than the other |
Welch's test doesn't pool the standard deviations, and it uses an adjusted number of degrees of freedom (the Welch–Satterthwaite formula), which holds up when the groups differ in size or spread.
One-tailed and two-tailed tests
A two-tailed test asks whether the means differ in either direction. A one-tailed test asks only whether one is higher than the other. A one-tailed p-value is half the two-tailed one, about 0.009 in the example. Choose the type before looking at the data. Switching to a one-tailed test afterwards to get under 0.05 is a known way of misleading yourself.
Statistical vs practical significance
A difference can be statistically significant and still not matter. If both lots meet a specification of at least 98%, a 0.3-point gap may make no difference to how either is used. The ASA statement makes the same point: statistical significance is not the same as scientific importance (ASA, 2016).
The reverse is also true. With only two or three measurements, a real difference can fail to reach significance because the scatter is too large to see past. "Not significant" means "not shown", not "shown to be the same".
A confidence interval for the difference is often more useful than a p-value alone. It shows both whether the gap could be zero and how large it plausibly is.
A short checklist
When a result is called significant, check:
- what the two groups were, and whether they were measured the same way,
- how many measurements each group had,
- the actual p-value, not just "significant",
- how big the difference is, and whether that size matters for the purpose, and
- whether the test was chosen before the data were seen.
Related guides
HPLC purity tests are in vitro measurements, and these statistics apply to any repeated lab measurement of that kind. For how a single purity figure is produced, see HPLC purity explained. For how labs check a method's spread before running samples, see reading a chromatogram. Every lot has its own COA, published on the product page and on that lot's own page. See Quality.
More on research basics
- Looking up a compound: CAS numbers, PubChem and the literatureWhat a CAS number is and how to check one, plus how to identify a research compound with PubChem, UniProt and PubMed, and why salts get their own entries.
- In vitro, in vivo, in silico: what the research terms meanWhat in vitro, in vivo and in silico mean in research, how in vitro and in vivo work differ, and how the terms relate to research-use-only reagents.
- Confidence intervals, standard error and error bars: reading the spread in lab resultsWhat a confidence interval is, how standard error differs from standard deviation, how to work out a 95% interval, and how to read error bars on a graph.