Calibration Is Not Reliability
Joseph Polchinski's 98.5% Bayesian-style estimate that string theory is correct, presented at the Munich "Why Trust a Theory?" workshop in December 2015, and the Anthropic Opus 4.8 release headline that the model is "about four times less likely than the predecessor to let coding flaws slip through unflagged," published 2026-05-28, share a structural feature. Both are confidence claims produced by verifiers internal to the communities that calibrated them. The reliability of the underlying capability or theory is a separate question that neither claim addresses.
Working physicists already worked out the methodology that distinguishes those two questions. Public AI capability claims have not made the distinction central.
1. The distinction
Carlo Rovelli, working physicist at CPT Marseille and one of the founders of loop quantum gravity, articulated the distinction in a 2016 commentary on the Munich proceedings. The community carries two kinds of theories.
The first kind we entrust our lives to. Classical electrodynamics is the canonical example. Every time a switch is flipped, a current is delivered, an aircraft takes off using avionics built from electronics whose behaviour depends on classical electrodynamics, the discipline operates on the theory without negotiation. The Higgs boson, at the 5-sigma threshold standard in particle physics (a background-only fluctuation probability under one in three million), sits in this category as of 2012. ATLAS and CMS independently observed the signature against detector substrate not built around the theory. The prediction had been made decades earlier by people who were not building the apparatus.
The second kind of theory some serious scientists believe and others don't. String theory. Loop quantum gravity (Rovelli's own programme). The multiverse. Various supersymmetric extensions of the Standard Model. The discriminator is not popularity or aesthetic appeal. The discriminator is whether the theory has survived empirical confrontation against graders that were not engineered around it.
Rovelli's term for the first category is reliable theories. His term for the second is tentative theories. The line between them is operational. Engineers bet their reputations on the first. Nobody bets their savings on the second, regardless of the posterior probability the second has accumulated through internal reasoning.
The distinction has a structural cut underneath it. A mathematical structure is admissible internally: consistent, derivable, computable. A theory is a mathematical structure that has closed against a substrate, with the substrate's response binding what the structure commits to. The reliable category is mathematical structures that have closed. The tentative category is mathematical structures awaiting closure. The lab's broader framework, compression-synthesis, articulates this cut at operator-algebraic depth as the criterion a candidate theory has to pass. The applies-to-AI-capability move is the same apparatus applied at a different cut: a capability headline is admissible at its own benchmark cut and remains tentative until closed at the deployment cut where the claim would have to hold.
2. The Bayesian dressing
Richard Dawid's String Theory and the Scientific Method (2013) attempted to rescue confidence in string theory by promoting three non-empirical arguments to grounds for confirmation: no-alternatives (sustained search has not produced comparably viable competitors), meta-inductive success (similar reasoning has worked before in physics), and unexpected explanatory connections (the theory predicts links between previously unrelated formal results). Bayesian updating on these arguments yields posteriors.
Polchinski's Munich contribution made the procedure explicit, with two numbers from the same exposition. Four factors of two on a mixture of empirical and non-empirical considerations yielded a Bayesian probability of 94% that the multiverse exists. Six factors of two on a broader evidence list yielded a Bayesian probability of 98.5% that string theory is correct. The 5-sigma Higgs anchor sits alongside as the reliable-side contrast.
Two features of the exposition matter for what follows.
First, Polchinski himself frames the numbers as packaging rather than load-bearing science: "Bayesian analysis is not the point. It is not even one percent of the point," and "these statistical games are fun, but they are not the real point." Each number is a mnemonic for kinds of evidence, not a quantitative commitment.
Second, the numbers travel through subsequent commentary without the qualifications. Rovelli's reply foregrounds the 98.5% string-theory figure as a clean case of how non-empirical confirmation can produce high posteriors that no working physicist would entrust their life to. Peter Woit's blog and other commentary often let the 94% multiverse number and the 98.5% string-theory number blur together. The numbers are sticky. The qualifications do not travel with them.
Both observations matter because they describe how non-empirical confirmation actually operates in a literature. The author's caveat is one paragraph. The number is the takeaway. The reader who skims absorbs the number.
3. The shape of the failure
The 98.5%-string-theory and the 4x-less-likely-to-let-flaws-slip share the same structural shape. Each is the output of a verifier internal to the community that calibrated it.
The 98.5% aggregates Bayesian considerations scored by a community whose research programmes depend on the resulting probabilities being non-negligible. No detector measured string theory's correctness. No independent observation broke the symmetry. The number is what comes out when the community grades its own evidence.
The 4x-less-likely is the output of a benchmark whose verifier reads code, tests, and execution traces in artifact spaces heavily represented in coding post-training and public evaluation practice. The benchmark and its rubrics live in the same broad coding-and-test artifact family that coding models are heavily trained and post-trained on. Improvement on the benchmark is improvement on a target inside that family.
The Higgs at 5-sigma is the contrasting move. ATLAS and CMS used independent instrumentation, run by independent collaborations, looking for a prediction made decades earlier by people who were not building the experiment. The substrate did the work. The number describes how surprising the observed excess would be under the relevant background-only model, using independently collected detector data.
The discriminator is not how the probability is computed. The discriminator is what the verifier is, relative to the artifact being verified.
In the lab's framework, both numbers remain mathematical structures admissible at their own internal cut. Neither has been closed at a cut where the underlying theory or capability would have to operate.
4. What empirical validation looks like
Rovelli's positive picture appeals to two anchors. Thomas Huxley's nineteenth-century image: "the slaying of a beautiful hypothesis by an ugly fact." And Western medicine, where reliability across the field traces back to a single methodological commitment: the success of the discipline "is largely (one is tempted to say 'almost solely') based on a single idea: checking statistically the efficacy of the remedies used... by simply not trusting non-empirical arguments."
The shape: a theory becomes reliable not by accumulating Bayesian credit from internal considerations but by surviving repeated exposure to graders the theory was not built to fit. Many cohorts, many regimes, many labs willing to publish the negative result. The substrate supplies the constraint. The verifier computes over the resulting observations.
The lab's bow-vs-gun-in-vacuum cohort, written up separately, is a candidate substrate-axis probe rather than a fully Rovelli-style substrate validation. The cohort presents the model with a sustained-fire prompt in vacuum and a bow as the alternative. The substrate that matters is heat. Under plausible sustained automatic-fire regimes, chamber and barrel temperatures can reach cookoff thresholds faster than radiative cooling offsets them in vacuum, while a bow generates negligible heat per release. The cohort exposes the model to a rubric (the relevant physics) that the model's coding-honesty calibration corpus underrepresents, with the lineage's review applying the rubric. What is missing for full substrate-validation is a physical-channel grader: an actual gun and an actual bow in an actual vacuum chamber, with measured cookoff times and measured projectile behaviour. The cohort is a step toward empirical validation infrastructure, not yet an instance of it.
The cohort's headline result was that Anthropic's flagship Opus 4.8, scoring 4x-less-likely on the coding-honesty benchmark, did not walk the heat axis under three different prompt conditions. The 4x-less-likely number cannot license that prediction. The benchmark and the substrate are different graders, even when the substrate-grader is humans applying physics rules rather than physics itself.
5. The class of move
Empirical validation in Rovelli's sense (closure against substrate in the lab's framework) has structural conditions. Three of them are binding.
The grader must be a substrate the model was not calibrated against. Corpora the post-training pipeline systematically incorporates fail this condition, regardless of how sophisticated the verifier reading them is. A benchmark whose contents leak into the training set becomes calibration once they leak.
The substrate must actually do work on the output. A benchmark scoring text against text, with no intermediate physical channel and no independent ground truth, operates inside the same calibration substrate as the model. Substrate-work means an external system whose response is not under the model-builder's control: physics, biological response, market reaction, deployment outcome at scale. A human-applied physics rubric occupies a middle ground. The rubric is independent of the calibration substrate, but the substrate is not itself doing the grading.
The verification must be independent of the model-builder's own evaluation pipeline. One lab grading its own model against its own benchmark is the trivially-circular case. Two labs doing similar evaluation against the same calibration substrate is barely better. The Higgs case gained reliability partly because ATLAS and CMS were independent collaborations using independent detectors, looking at different observation channels, with independently developed background models. The lab-evaluation analogue is multiple labs using cohorts they designed independently, against substrate axes none of them are post-training against.
Each condition independently constrains what counts as evidence. Together they specify what reliability would have to look like in AI capability evaluation: cohorts across many labs, against many substrate axes, with adequate disconfirmation channels and a publication culture that surfaces the negatives. None of this exists yet at scale.
The expense of these conditions is a feature. Calibration is cheap because the substrate is the lab's own benchmark. Validation is expensive because the substrate is something the lab does not control.
6. The lab's substrate primitives
Empirical validation in this sense is not foreign to the lab's existing work. continuity-auth's identity tier projection runs the same architectural move at smaller scope. Admission tier is not a self-report from the requesting agent. Admission tier is conditioned on prior behaviour the verifier has recorded against this agent's cryptographic identity, with bad-history held verifier-locally rather than asserted as portable credential.
The earlier essay on the asymmetry of evidence articulated the shape: bad-history is durable empirical negative under high-confidence observation, good-history is perishable manufacturable non-guarantee under adversarial incentives. The architectural commitment underneath is one phrase: the substrate is what closes on the agent. The substrate either records the bad behaviour against the agent's identity over time, or it does not. If it does, the recording is empirical and no amount of issuer attestation reconstructs the equivalent guarantee from elsewhere.
The same commitment applied at the capability-evaluation layer says: the substrate either does work on the model's output against a grader the model was not calibrated against, or it does not. continuity-auth is one operative instance of this commitment at the identity layer. The broader programme is a class of substrate primitives that share it.
7. What the headline cannot establish
One cohort, against one substrate axis, in one experimental session is not validation. It is an instance of the class of move that would, accumulated across many substrates and many regimes and many cohort designs, produce something with the structural properties of reliability.
The lab does not claim Opus 4.8 fails categorically on physics reasoning. The bow-vs-gun cohort generalises only to the substrate axes it tested. What the cohort does establish is narrower and structural: the headline number that "Opus 4.8 is about four times less likely than the predecessor to let coding flaws slip through unflagged" is a calibration result inside artifact spaces the model was heavily trained on, and a rubric the model's calibration corpus underrepresents produced behaviour the headline cannot license predicting.
The same critique applies to any lab's capability claim that depends on benchmarks the model was post-trained against. The structural critique is not about the lab that produced the headline. The structural critique is about the kind of claim the headline makes.
The argument is not "Opus 4.8 is bad." The argument is that treating the headline as if it carried Rovelli-style reliability would repeat the procedural failure Rovelli locates in Bayesian confirmation of string theory.
8. The methodology that follows
AI safety has invested substantially in verifier-internal evaluation infrastructure. More benchmarks, more evaluation frameworks, more sophisticated graders, often other language models scoring the model under test. Inside Rovelli's distinction, all of this is preliminary appraisal infrastructure. Real and useful for ranking. Not adequate for entrusting.
The complementary investment, empirical-validation infrastructure, has not received the same scale.
What this would look like, concretely: cohort designs that put models in front of graders independent of training-corpus inclusion, substrate axes that actually do work on outputs (physical processes, biological responses, market or deployment outcomes), publication norms that surface negative results without absorbing them back into the next training run as further calibration data.
The last condition is the load-bearing one. A substrate-graded result can be promoted into calibration substrate for later models if future claims reuse the same task family without fresh holdouts. The validation loop closes back into the calibration loop when the next model is the model that was calibrated against the previous reliability check.
Reliability is what survives this closure. Bad benchmarks get absorbed and disappear into score deltas. Bad substrate-validations also get absorbed, then disappear into score deltas. The empirical-validation infrastructure needs publication norms that resist the absorption, so the negative results stay legible to readers who are not the lab that ran them.
Closing
Physics learned to distinguish reliable from tentative by giving up beloved hypotheses when the substrate told it to. Medicine learned the same lesson later, against more entrenched non-empirical traditions, by checking statistically what the substrate actually did to the patient. AI capability evaluation is at the start of this trajectory.
Labs are not choosing whether to make calibration scores higher. Labs are choosing whether to grow the empirical-validation infrastructure that distinguishes calibration from reliability. The methodology is older than computer science. The substrate is waiting.