← Reproduction studies

When one file contains two derivations of the same quantity

Reproducing the SPIRAL / ELLIPTICAL / UNCERTAIN flags of Galaxy Zoo 1 from the rule its own data release states, across 667,944 galaxies, as a Facetwork FFL workflow. September 2026.

1. Purpose of the analysis

Galaxy Zoo 1 is the morphology catalogue a great deal of extragalactic work is built on — if you want a clean sample of spirals or ellipticals from SDSS, this is where you get it. Its data release publishes the entire chain in a single table: what volunteers voted, the bias-corrected likelihoods derived from those votes, and the final classification flags.

That makes it unusually checkable. The question here is narrow: does applying the documented rule to the published columns produce the published flags? If it does not, anyone who builds a sample the obvious way gets a different sample from the one the catalogue's authors intended.

2. What the original work does

Galaxy Zoo asked volunteers to classify SDSS galaxy images; each galaxy received a median of around 40 independent classifications. Raw vote fractions are not directly usable as morphology, because the vote is biased by distance: a spiral far enough away looks smooth, so it collects elliptical votes. The published catalogue therefore carries bias-corrected ("debiased") likelihoods alongside the raw fractions.

The data release's one-line summary of the flags is that they "are based upon a vote fraction threshold of 0.8". Read further and the same note says something much more specific:

There are two different debiasings of the same quantity in adjacent columns. Both are correct. They answer different questions: the bias in a likelihood and the bias in a thresholded classification are not the same quantity, and Bamford et al. found the latter is worse. Nothing in the file marks the two as incommensurable.

Method: Lintott et al., Galaxy Zoo: morphologies from visual inspection of SDSS galaxies (2008); Galaxy Zoo 1 data release (2011); debiasing in Bamford et al., morphology and colour vs environment (2009).

What we do differently

Deliberately naive, and the naivety is the experiment: this is precisely what a reader who stops at "a vote fraction threshold of 0.8" would compute.

3. What our FFL workflow found

Both sides are read from the same file — one applies the documented rule, one reads the published flags — so any difference is provably the rule, not the data. No tolerance and no normalisation: these are flags, so the comparison is exact.

verdict   : DIFFER at level `keys`
expected  : flags-as-published.csv         (667,944 records)
actual    : flags-from-documented-rule.csv (667,944 records)
differing : 89,332
normalised by : (nothing — compared as-is)

89,332 galaxies — 13.4% — are classified differently. The verdict sits at keys: every galaxy is present on both sides, so nothing was lost or filtered. The two artifacts describe exactly the same population and disagree about it.

4. How our workflow compares to the published work

This study does not evaluate Galaxy Zoo's classifications and could not: the volunteer votes, the redshift-dependent bias model and the validation are the scientific work, and none of it is reproduced here. The flags are the more careful artifact and the catalogue is right to publish them.

What the workflow measures is the distance between a published result and its own one-line summary — how far a documented subtlety moves a sample when a reader does not follow it through. The note names its own victim: "users who do not want to get into the details too much".

Why this needed no tolerance

The companion NOAA study spent considerable effort on choosing a tolerance, and got it wrong the first time. Here there is nothing to choose: flags are categorical, so the comparison is exact and the verdict cannot be an artifact of a threshold. That is worth noticing when designing a reproduction — where you can arrange to compare categories rather than magnitudes, do.

5. Reproducing this

fw ffl run --primary examples/repro-galaxyzoo/gz1_morphology_repro.ffl \
           --library facetwork/ffl/file.ffl \
           --workflow repro.galaxyzoo.ReproduceMorphologyFlags

The catalogue is 20 MB gzipped, 667,944 rows. Both derivations are FFL script blocks bound to fw.file.FileOps — no handler and no domain package. The comparison indexes 667,944 keys.

6. Limitations

• This is not a defect report. The behaviour is documented in the data release, the reasoning behind it is sound, and the published flags are the artifact to use. The finding is about how easily a documented subtlety is missed, not about whether Galaxy Zoo is correct.
• We do not reproduce the debiasing itself, which depends on redshift and apparent magnitude. Only the final thresholding step is recomputed, which is why the study can say that the two disagree but not re-derive the correct flags from raw votes.
• Table 2 covers galaxies with SDSS DR7 spectra. Table 3 — those without — publishes no debiased columns at all, because the bias cannot be estimated without redshifts, so this comparison does not apply there.
• The 0.8 threshold is applied as >=. The data release does not state the boundary convention; at three-decimal precision a handful of galaxies sit exactly on it.