Retrieve a published dataset, rerun the analysis it claims to produce, compare the outputs, and document the discrepancies — each one run as a Facetwork FFL workflow on a distributed fleet.
The first three studies below reach the same uncomfortable place by different routes: a comparison that is arithmetically correct and answers the wrong question. Neither failure announces itself. Both runs are green.
Each page opens by describing what the original work does, with links to its primary sources, before describing ours — because "we got a different number" means nothing until you can see which step differs. In one case we compare two published outputs without re-deriving either; in the other we deliberately omit corrections the publisher applies, and the gap we measure is those corrections. The fourth asks a different question entirely: not whether a published result is right, but whether the effort that produced it was needed.
NIST Genome in a Bottle HG001, releases v3.3.2 vs v4.2.1 — 3.6 M and 3.9 M variants, compared in under a minute.
The trap is scope. The comparison reported 242,958 variants
"retired, prior validation void". 41% of them were chromosome X,
which the newer release does not cover at all — the files are named
CHROM1-X and 1_22. Nothing was retired. Corrected,
site-level carryover is 95.9%, not 93.2%.
Lesson: every rung of a comparison ladder assumes both artifacts describe the same key space. Check that first, or measure content differences between things that were never comparable.
NOAA 1991–2020 U.S. Climate Normals recomputed from GHCN-Daily observations, across 12 stations — 65 tasks.
The trap is direction. At New York every residual sits inside a generous tolerance and every one falls on the same side per column — TMAX high 12/12, TMIN low 12/12. A tolerance is a per-value predicate and cannot see that. Across 432 values the median error is 0.23 °F, but reproduction quality spans a twentyfold range: Los Angeles to 0.04 °F, Chicago off by 0.88.
Lesson: "36 small random errors" and "36 errors in the same direction" are different claims about a reproduction, and only the second says the method differs. Also: n = 1 produced a compelling and wrong explanation, which twelve stations refuted — and a relative tolerance on a temperature produced a second wrong answer, retracted on the page.
Galaxy Zoo 1 morphology flags recomputed from the rule its own data release states — 667,944 galaxies, compared exactly.
The trap is provenance within a single artifact. The catalogue publishes bias-corrected likelihoods and classification flags side by side, and the flags use a different bias correction from the likelihoods — documented, but easy to miss. Thresholding the published column at the documented 0.8 disagrees for 89,332 galaxies (13.4%), dropping 22.6% of every spiral in the catalogue.
Lesson: adjacent columns are not necessarily commensurable, and nothing in a file format can say so. This one needed no tolerance at all — flags are categorical, so the verdict cannot be an artifact of a threshold.
Hubble Asteroid Hunter — what a matched filter recovers from the same archive images the volunteers looked at. 40 trails, 168 tasks, ~7 GB of HST exposures fanned out across a four-host fleet.
The trap here is different: nothing is wrong. The catalogue is right, the method is sound. The question is whether the human effort was necessary — usually asserted, rarely measured. Signal recoverability collapses between magnitude 21 and 23 and does not return, and the catalogue's median trail is 22.6: more than half of it lies where the signal cannot be recovered even when handed the exact position.
A second workflow then searches the same frames blind, with no knowledge of where any trail is. Told where to look, 11 of 39 trails are recoverable; searching blind, 3 land in the top 10 of ~900 candidates per frame. The brightest asteroid in the whole catalogue ranks 65th among the line-like features in its own image.
Lesson: finding lines is trivial and fast; deciding which lines are asteroids is the entire problem, and that is what the CNN and the volunteers supplied. Running both also disproved a claim this page had already published — that the targeted measurement upper-bounds blind detection — because being told where to look also locks you to the catalogue's angle and length. Neither method dominates. Also the only study here that reproduces itself: an accidental duplicate agreed on 40 of 40 verdicts, and the one value it disagreed on exposed a defect in our own code.
Each study records what went wrong in building it, not only what it found. The GIAB page was published twice with figures that were wrong in the same way — arithmetically sound, meaning something other than their label. The normals study survived a unit error, a hand-rolled CSV parser that silently shifted every field, and a residual test that pooled columns until opposite biases cancelled.
All three were caught by checking a result against an independent path rather than by anything failing. That is the actual method these studies are arguing for, so hiding the misses would misrepresent it.