A tool that finds impossible numbers in research papers carries an obvious risk: that its findings get read as accusations of fraud. They usually should not be.
This matters enough to state plainly, because getting it wrong damages real careers and discredits the method along with them.
The base rates
When a mean fails the GRIM test, the realistic explanations, roughly in order of frequency:
- A typo. A digit transposed between the analysis output and the manuscript.
- A misreported sample size. The paper states the study N while the measure had its own — a different subset, a different questionnaire.
- Unreported missing data. Three people skipped the item; the mean is over 17 values, not 20, and is perfectly achievable.
- A different analysis than described. Values were trimmed, winsorised, or transformed in a way the methods section did not mention.
- Copy-paste error between tables. Endemic, and usually harmless.
- Fabrication. Real, and rare.
Fabrication is last on that list for a reason. An arithmetic check proves that the reported numbers cannot all be correct. It has nothing to say about why.
Proof and suspicion are different things
There are two fundamentally different kinds of finding, and conflating them is the central failure mode of this whole field.
A proof is arithmetic. A mean of 3.43 from 20 integers is impossible in the way that 2 + 2 = 5 is impossible. There is no dataset, no analysis choice, no reasonable rounding convention that produces it. If a tool says “impossible” it should mean this and only this.
A suspicion is a heuristic. Baseline arms that match unusually well, a leading-digit distribution that departs from Benford, a test statistic that does not match the group descriptives. Each has innocent explanations — stratified randomisation, bounded scales, a covariate-adjusted model. These are reasons to look closer. They are not evidence.
A tool that renders both in the same red box, with the same confidence, will eventually get someone accused over a Benford deviation. That is why Statlint encodes the distinction in the type system itself: proofs and heuristics are different values, rendered differently, scored differently, and labelled explicitly in every report. It is not a presentational choice.
The false-positive problem is the whole problem
An integrity tool’s credibility is destroyed by its first wrong accusation, not earned by its hundredth correct one. So the engineering priority is inverted relative to most software: missing a real error is far less costly than inventing a fake one.
Concretely, that means:
- Handle every reported number as an exact decimal, never a float.
2.50and2.5carry different information about precision, andfloat()destroys it. - Reason over the rounding interval, not the printed value. A statistic shown
as
2.10was somewhere in[2.095, 2.105). - Accept any defensible rounding convention.
2.345is2.35under round-half-up and2.34under banker’s rounding. Declare a value impossible only when no convention rescues it. - Validate against real data. Statlint’s test suite sweeps thousands of randomly generated genuine integer datasets through every proof-style check and requires that not one is flagged. A single false positive fails the build.
That last one is the only real guarantee. Everything else is intention.
How to raise a finding with authors
Assume error. Ask a question. Attach the arithmetic.
In Table 2 the anxiety measure is reported as M = 3.43 with N = 20. For 20 integer responses the achievable means are multiples of 0.05, so the nearest possible values are 3.40 and 3.45. Could there be missing data on this item, or a different N than the study total?
That framing costs nothing and gets you a real answer. It also survives the very common outcome in which you are the one who was wrong — because the item had 17 respondents, or the scale averaged four sub-items, or the data were not integers at all.
What not to do: post it publicly first, describe it as evidence of misconduct, or aggregate several heuristic flags into a claim you cannot defend individually.
Why this restraint makes the tools more useful, not less
There is a temptation to make forensic tools maximally aggressive — flag everything, let humans sort it out. It fails for a structural reason: the humans who would have to sort it out are exactly the overloaded editors and reviewers who have no spare capacity. A tool with a 10% false-positive rate does not save them time; it creates a new triage burden and gets switched off.
A tool that only speaks when it can prove something is one an editor can act on immediately. That is a much smaller set of findings, and a much more valuable one.
The goal was never to catch fraudsters. It is to make the published record more accurate — and most of that work is fixing typos.