← Reproduction studies

Is the signal even there, when you are told exactly where to look?

Measuring whether the signal behind each published Hubble Asteroid Hunter trail is recoverable from the archive images at its known position, as a function of trail brightness. A Facetwork FFL fan-out, 168 tasks. September 2026.

1. Purpose of the analysis

Citizen science projects are often justified on the grounds that the task is too hard to automate. That is usually asserted rather than measured. This study measures a necessary condition for automation: at what brightness does the signal stop being recoverable at all, even when its exact position is known?

Below that point automation is not merely difficult, it is impossible by this route — there is nothing for a detector to find. Above it, automation might work or might not; this study does not say.

The answer is not a verdict on the project. It is the quantity that decides whether the human effort was necessary, and it can be computed because both the catalogue and the underlying images are public.

2. What the original work does

Hubble Asteroid Hunter showed archival HST images to volunteers, who marked asteroid trails; a convolutional network assisted, and the identifications were checked against known orbits. The published catalogue (Kruk et al. 2022, VizieR J/A+A/661/A85) contains 1,701 trails across 1,318 observations, each with sky endpoints, trail length, exposure timing and an observed magnitude.

It is given the answer — position, orientation and length — and asked only whether the signal is present and strong enough that an automated method could plausibly have found it. Note that this both helps (position is known) and constrains (angle and length are fixed to the catalogue's), so it is not simply an easier version of blind detection. §5b measures both on the same trails and finds neither dominates.

3. What our FFL workflow found

Fraction of published trails whose signal reaches 5 sigma when integrated along the catalogue path:

11 of 39 measurable trails recovered (28%). Faintest recovered: mag 22.5. Brightest missed: mag 19.3.

Brightness alone does not predict recoverability

Recoverability is not monotonic in magnitude. A magnitude 19.3 trail is missed while a 22.5 one is recovered. Detection depends on what the trail crosses: a faint trail on empty sky integrates cleanly, a brighter one over a galaxy does not. Any estimate of "how faint can we go" that quotes a single limiting magnitude is hiding this.

4. Three things that had to be right first

Each of these would have produced a confident, wrong answer, and each was found by measurement rather than by the run failing.

The trails are not in the drizzled products

At the published position of the first trail tested, the standard drizzled image reads −0.09 sigma — nothing. Drizzling rejects cosmic rays, and a moving object sits at different pixels in every exposure, so the rejection deletes it. A study built on the obvious data product would have found no trails anywhere and concluded the catalogue was wrong. The workflow fetches individual FLT exposures instead: four times the data, and the only thing that works.

The astrometry is off by tens of pixels

A magnitude 15 trail sat 59 px from where its published sky coordinates predicted — an 8,000-sigma feature, unmissable by eye, invisible to a detector that trusts the WCS. ACS/WFC exposures carry SIP distortion but need the additional lookup corrections. The detector therefore scans a window of offsets and reports the offset it required: "found at the predicted pixels" and "found 60 px away" are different claims.

Thresholding before integrating destroys the signal

A faint trail sits at roughly 0.4 sigma per pixel. Thresholding at 3 sigma and then looking for lines finds nothing. Integrated along its length the same trail is about 5 sigma. The detector must integrate first; the obvious order is backwards.

5. Reproducing this

fw ffl run --primary examples/repro-asteroids/hah_trail_recovery.ffl \
           --library facetwork/ffl/file.ffl \
           --workflow repro.asteroids.RecoverPublishedTrails

Selection filters to trails with Nex=1 whose MJDsTrail equals MJDsExp, pinning each to the first exposure of its association — the one fetched. Without that the detector searches an exposure the asteroid was never in and reports a false non-detection. The sample is spread evenly across magnitude: taking the brightest 40 would return a flat 100% that answers nothing. foreach … limit 4 bounds how many ~170 MB exposures are in flight.

It reproduces itself

An accidental duplicate launch ran the same 40 trails a second time on the same fleet. The two runs agreed on 40 of 40 recovered/not verdicts. They disagreed on one sigma value, which is how a defect in this study was found: a non-measurement had been encoded as sigma = −99.0, sitting in a measurement column beside real values. measured is now a separate field — a measurement that did not happen is not a number, it is a different fact.

5b. The same 40 trails, searched blind

A second workflow searches each exposure with no knowledge of where any trail is: bin the frame, rotate to each of twelve orientations, integrate along rows, and keep local maxima above 3 sigma. Angles within 15° of the readout axes are skipped, because bleed columns and diffraction spikes are linear features a line filter cannot distinguish from an asteroid. The published position is used only afterwards, to ask where the real trail landed in the ranking.

Both workflows share the same selection facet, so the sample is identical.

The result that matters

The brightest asteroid in the entire catalogue — magnitude 15.0, an unmissable 199.5 sigma when you know where to look — is only the 65th most line-like feature in its own frame. Sixty-four stars, bleed columns and diffraction spikes outrank it.

So the hard part was never finding lines. A frame yields ~900 candidates in about a second. The hard part is deciding which of them is an asteroid rather than an instrumental artifact, and a matched filter has no way to tell: they are all bright lines. That is exactly the judgement a CNN and human review supply, and it is why "we found linear features" and "we found asteroids" are different achievements.

5c. Can the candidate list be thresholded? No.

Section 5b ends on a claim that was qualitative: a matched filter cannot tell an asteroid from an instrumental artifact. The obvious next move is to keep only the strongest candidates. We measured whether that works, on the 20 richest fields in the catalogue — 182 exposures, 364 chips, 327,600 candidates, 81 trail×exposure observations, about 31 GB streamed.

There is no cut that makes the list reviewable. At the threshold where recall reaches zero, 98 candidates per exposure still survive. Real trails have median z = 128; the background has median z = 84 and a 99th percentile of 465. The distributions overlap almost entirely.

The magnitude breakdown carries a second finding, and it is not the expected shape:

The brightest bin does worse than the middle. Section 4 records why capping replaced masking: masking at 50 sigma once deleted a trail that was itself an 8,000 sigma feature. But capping at 30 sigma flattens a magnitude 15 trail to the same level as a bright star, discarding exactly the advantage that should make it trivial to find.

Differencing the frames does not fix it either

Every one of the 1,316 fields with a published trail has at least two exposures — 4,059 in total. The asteroid moves between them; the sky does not. So a median stack of the association is a model of the static sky, and subtracting it should remove the structure that makes z meaningless. We tested this on j96v05020: 12 exposures, 5 published trails, a standard ACS-WFC-DITHER-LINE pattern with all twelve pointings identical.

Resampling each frame through its full WCS — distortion lookup tables included — improved the background by 4×, so misalignment was a real part of it. It did not come close to fixing it. The reason is measurable, so we measured it: bin every pixel by how bright the static sky model is there, and look at the residual left behind.

What this costs to answer at scale is now known rather than guessed: the 1,316 fields hold 4,059 individual exposures, about 684 GB. The subset worth searching first is smaller — 1,107 frames (~187 GB) that sit inside a field already known to contain an asteroid but which no published trail is recorded in. None of it is worth downloading until a discriminant exists that survives the table at the top of this section.

6. Limitations

• These are not recall, and not an upper bound either. The measurement is given the published position, orientation and length. That helps in position and constrains in angle and length, so a blind search is not strictly worse — measured in §5, where a blind search ranked one trail first that this measurement missed entirely.
• No detection was attempted. Building a blind detector — candidate generation across orientations and lengths, then rejection of stars, cosmic rays and diffraction spikes — is the hard part and is not done here. This study establishes only that below the cliff there would be nothing for such a detector to find.
• 40 trails, so individual bins hold as few as one object. The cliff spans several bins and is robust; the exact percentages are not.
• Only the first exposure of each association is searched. Trails appearing in later exposures are excluded by the selection rather than counted as misses.
• A trail whose predicted position falls outside every science extension is reported as not measurable and excluded from the fractions, not counted as a non-detection.
• This measures whether the signal is recoverable, not whether the published identification is correct. Orbit matching, which is how the catalogue attributes trails to known objects, is not reproduced.