Writing/The Numbers Mostly Hold

The Numbers Mostly Hold

An agent re-ran published political science papers from scratch. What broke was rarely the statistics.

August 17, 2026

Reproducibility gets discussed as a crisis. I wanted to know what it looks like as an engineering problem, so I built something that does it: an agent fetches a published paper’s replication package from its archive, installs the exact dependencies, locates the code that produces each reported number, runs it, and records per result whether the number regenerates. Then it writes down what happened, including when it fails and when it could have made a failure disappear.

What follows comes from five reproduction units across three papers. That is a tiny sample and I want to be unambiguous about it: nothing here is a claim about a literature, a field, or a rate. It is a claim about what the failure modes look like up close, which turned out to be different from what I expected.

The headline result

For the first paper, Adams et al., “Can’t We All Just Get Along? How Women MPs Can Ameliorate Affective Polarization in Western Publics” (APSR, 2023), the run reproduced all thirteen primary coefficients in the main table. All four analysis sample sizes matched exactly: 1,842 / 1,842 / 1,836 / 1,833 directed party dyads, to the row. Twelve of thirteen clustered standard errors matched. Every supplementary table in the package, twenty-three model columns plus four multilevel specifications under three clustering variants each, produced cleanly.

The second paper, Arias and Blair’s “Changing Tides: Public Attitudes on Climate Migration” (JOP, 2022), asked for one conjoint effect published at +0.076. The run regenerated +0.075688, a deviation of −0.41%. Respondent count and observation count both matched exactly.

So the numbers mostly hold. There is no scandal here. The useful part is the handful that didn’t hold, because none of them look like what “failed to reproduce” sounds like.

Two published numbers, two different groupings

The APSR paper’s Figure 1 reports two descriptive statistics for the same variable, the proportion of women MPs in a party: mean 0.29, SD 0.16.

The run computed both from the variable the figure actually plots. The mean came back 0.2892, rounding to 0.29, which matches. The SD came back 0.1827, rounding to 0.18, against a published 0.16, which does not.

There is a second defensible way to group that variable, over its distinct values rather than its plotted rows. Under that grouping the SD is 0.1597, rounding to 0.16 and matching the paper exactly. But the mean then becomes 0.2986, rounding to 0.30, and stops matching.

No single grouping reproduces both published numbers. One grouping gets the mean, the other gets the SD, and the paper reports both as though they came from one computation.

Nothing here is wrong, exactly. Both numbers are real, both are computable, and either grouping is arguable. What doesn’t exist is a single procedure that produces the pair, which means the figure caption as written is not a specification. You cannot re-derive it without guessing. That is a different category from a result failing to replicate, and I don’t think it has a good name yet.

The one that had to be left broken

The other miss is smaller, and more instructive about the machinery than about the paper.

In the fourth column of the main table, one standard error regenerated as 0.10458, which rounds to 0.10 against a published 0.11. Off by one unit in the last published digit. The coefficient it belongs to matched exactly, and every other standard error in that column matched exactly. The likely cause is a small finite-sample adjustment difference in the clustered-SE estimator, or the authors rounding a boundary value.

Resolving it would have meant changing the estimator, which the protocol forbids. So the record says could-not-regenerate, on a result that is almost certainly fine.

That should sound unsatisfying, because it is. A verification system that lets you nudge the method until the number lands is not verifying anything. The cost of that rule is failures like this one, real by the standard and trivial in substance, and the record has to carry them without softening. If you only report clean outcomes, nobody can tell whether the standard or luck produced them.

The same discipline caught a near-miss in the other direction. On the climate migration paper, the automated sample check coded a choice task as a respondent×task pair, got 9,483, compared it to the published 18,966, and printed FAIL. But 18,966 is the profile-level observation count, which the data matched exactly. That is a label difference rather than a sample discrepancy, and it was recorded transparently instead of silently overridden. Had the sample genuinely differed, the target would have been abandoned rather than reported.

What actually breaks is the toolchain

The dominant obstacle across every unit was not statistics. It was that five-year-old research code does not compile on a current machine.

One 2020-era package needed three separate diagnoses before a single number came out:

None of these are the authors’ fault, and none would have existed on the machine the paper was written on. This is ordinary software rot, and it means a reproduction attempt is mostly a dependency-archaeology exercise with a regression at the end. Pinning a contemporaneous R and package snapshot is now the leading candidate for the environment spec.

Each patch had to be argued as target-neutral before it was allowed. car was neutralized rather than version-swapped, and the record shows why: all sixteen of its calls act on respondent covariates that appear in no formula and no row filter, confirmed empirically by the sample count landing exactly on 1,086 respondents and 18,966 observations. The loader patch replaced exactly one statement, and the record includes both the original and the replacement so someone else can check the reasoning rather than trust it.

The environment diverged from its own spec, silently

The sandbox spec is versioned. Version 0.3 installs six OS development headers. A run found five of the six missing at runtime.

The cause is a line of defensive shell. Every install in the setup script is armored with || true, so one failure can’t abort the rest of the script. That also means a failed install leaves no error, no exit code, and no log line. The documented spec and the live environment diverge by construction, with nothing reporting it.

I keep running into this: the safety measure is what hides the fault. || true exists so a broken package doesn’t kill the run, and its cost is that you can’t tell a working environment from a broken one.

The resolution was not to remove the armor but to stop treating the spec as evidence. Every unit now verifies its own dependencies at runtime and records what it actually found. The documented environment is a plan. The recorded environment is what actually exists.

The agent scheduled its own wake-ups

One finding has nothing to do with reproduction, and it is the one I would most want another builder to know.

Mid-run, believing it was blocked waiting on a decision from me, the agent scheduled itself an hourly wake-up. Harmless here, since I was watching. But the intended deployment is unattended, on a contributor’s own account, overnight. A self-wake loop in that setting can re-arm indefinitely and quietly consume someone else’s quota while producing nothing.

The rule that came out of it: an agent at a decision point should exit rather than idle. Blocking is a terminal state, not a waiting state. Any agent that can schedule its own future execution can build a loop nobody asked for, and the failure is hard to see because from the outside it looks like the thing is working.

What I’d keep

Reproduction failures were mostly under-specification rather than error. The headline results held everywhere I checked. What didn’t hold was a figure caption that silently combined two groupings, which is a defect in the description of a computation rather than in the computation.

A verification standard is only as good as the worst report you are willing to publish under it. The 0.10 against 0.11 miss is the system working, and publishing it unsoftened is the point of the exercise.

Most of the work is software archaeology, so budget for dependency rot rather than statistics. Never let a spec stand in for a measurement: verify at runtime, record what you found, and treat the documented config as a hypothesis. And don’t let an agent schedule itself.

Five units across three papers will not support a conclusion about the literature, and I’m not going to pretend otherwise. What it does support is that the work is tractable. The hard parts are dependency management and writing down honest failures, and neither of those requires a lab.

← All writing