Peer reviewers are rarely asked to verify arithmetic, and almost never given the time to. Yet a meaningful fraction of published statistics are internally inconsistent — the numbers in a paper contradict each other, independent of whether the science is any good.
Here is a practical order of operations. It is arranged by return on effort: the first checks take seconds and catch the most.
1. Recompute the p-values
What it catches: a reported p that does not follow from its own test statistic and degrees of freedom.
Every inferential result of the form t(28) = 2.10, p = .04 contains a testable
claim. The p is a deterministic function of the statistic and its degrees of
freedom. You can recompute it exactly.
The important variant is the decision error: cases where the reported and recomputed p fall on opposite sides of the significance threshold. A result reported as significant that is not — or vice versa — changes the paper’s conclusions, not just its typography.
One caution: reason over the rounding interval. A statistic printed as 2.10
was somewhere in [2.095, 2.105). Recompute across that range, not from the
printed value alone, or you will flag rounding as error.
2. Test the means for granularity (GRIM)
What it catches: means that no dataset of the stated size could produce.
For integer data — counts, Likert items, test scores — the mean must be a
multiple of 1/N. With 20 participants the achievable two-decimal means are
…3.40, 3.45, 3.50…. A reported 3.43 is impossible.
Requires: integer measurements, and an N small enough that the grid is coarser
than the reported precision (N < 100 for two decimals). See
the GRIM test explained for the details and the
traps.
3. Test the standard deviations (GRIMMER and range bounds)
What it catches: SDs that cannot coexist with the reported mean and N.
Two independent constraints:
Integer structure (GRIMMER). For integer data the sum of squares must be an
integer, and it must have the same parity as the sum — because x² ≡ x (mod 2)
for every integer. Many (mean, SD, N) triples fail this outright.
Scale bounds (SPRITE). For any data confined to [min, max] with mean μ,
the variance cannot exceed (μ − min) × (max − μ). On a 1–7 scale with a mean of
4.0, the SD can never exceed about 3.08. A reported 3.50 is impossible regardless
of what the data looked like.
These two catch different things, which is why running both is worth it.
4. Recompute the test from the group descriptives
What it catches: a reported t or F that the group means, SDs and sample sizes do not imply.
If a paper prints both the descriptives and the test statistic, they must agree. Recomputing an independent-samples t from two groups’ summary statistics takes one line of code.
Interpret this one carefully: a mismatch is not proof of error. The authors may legitimately have run an adjusted model — ANCOVA, a paired test, a covariate-adjusted comparison — which would differ from the naive recomputation for entirely good reasons. Treat it as a prompt to read the methods section, not a finding.
5. Look at the distribution of the digits
What it catches: almost nothing on its own. Use it last, and lightly.
Benford’s law and terminal-digit uniformity are genuinely useful in forensic accounting, and genuinely oversold in research integrity. Bounded scales, rounded reporting, and any variable that does not span orders of magnitude all violate Benford innocently.
A digit anomaly is a reason to look more closely. It is never, by itself, evidence of anything.
The discipline that matters most
Sort your findings into two piles and never let them merge:
- Proven impossible. GRIM, GRIMMER, range bounds, and p-value recomputation that survives the rounding interval. These are arithmetic. The numbers cannot all be right.
- Worth a look. Digit distributions, baseline similarity, descriptive mismatches. These are heuristics. They prove nothing.
A tool — or a reviewer — that presents the second category with the confidence of the first will eventually make a false accusation, and will deserve to lose its credibility when it does.
What to do with a finding
Ask the authors. Neutrally, specifically, and with the arithmetic attached.
The overwhelming majority of internal inconsistencies are typos, transcription slips, misreported sample sizes, or unreported missing data. Framing a first contact as a question rather than a charge gets you a faster answer, a more accurate literature, and no defamation risk.
Doing all five at once
Running these by hand takes a trained reader the better part of an hour per paper, and requires stitching together several single-purpose R packages.
Statlint runs all five — plus the Carlisle baseline test for randomised trials — in one pass over a pasted results section or an uploaded PDF, and returns them separated into proven and heuristic, each with the working shown. It is free to use.