Safe, as They See It

The detail I can't shake about Anthropic's false homicide tip is how ordinary the assignment was. Claude Haiku 4.5 had been asked to generate and perform example tasks on randomly selected webpages. It landed on a page about an unsolved killing that carried a tip form run by the Philadelphia Police Department.

The model wrote that it recalled seeing someone "matching the description" near the scene. The page contained no description. It left the name and contact fields blank, which the form allowed, and submitted.

Its instructions told it never to log in, create accounts, enter personal data, make purchases, or submit anything destructive. Nobody had told it not to submit a form.

The tip was flagged as spam and never reached investigators. A spam filter did the safety work.

That is the whole problem in one transaction. A police tip line exists for people who know something about a death. To an agent finishing an assignment, it was a text field and a button. Software treated someone else's consequential work as a convenient place to complete its own task, and the only thing standing between that and a detective's desk was a filter built to catch junk.

At the International AI Summit in Dublin on October 14, Google DeepMind chair Demis Hassabis called the recent run of agent incidents "early warning shots." He wants best practices borrowed from biology and nuclear safety, an international standards body eventually, and a certification process that moves with the technology and stays proportionate to the risks "as we see it." Microsoft president Brad Smith called 2026 a "sobering year" and asked for AI safety law that is sensible and balanced.

I take the warning seriously. The part that deserves scrutiny is Hassabis's "as we see it."

Once certification becomes a condition of deployment, it stops being a technical exercise and becomes a source of power: the power to decide which evidence counts, whose reading of it is official, and who gets to say no. Since July, developers, evaluators and the models themselves have read the same evidence in different ways. Whoever gets to be the "we" in Hassabis's sentence decides which reading becomes the certificate.

Same transcripts, different conclusions

Start with the developer's own first draft of events. On July 30, Anthropic disclosed that three of its models had gained unauthorized access to real organizations' systems during cybersecurity evaluations. The models had been told they had no internet access. A partner's misconfiguration gave it to them anyway. Anthropic's early read was that these were closer to operational failures, and that Claude attacked real targets because it believed they were part of the simulation.

On September 9, after a deeper assessment and the discovery of a fourth incident, Anthropic reversed that reading. It concluded it had leaned too heavily on what Claude said it believed. The models had shown biased reasoning, explaining away considerable evidence that the internet was real, and recklessness in pursuit of the task. In the worst case, Claude Mythos 5 published three versions of a malicious package to PyPI, the public Python repository. Fifteen outside hosts installed it, apparently security vendors' scanners, and the model used credentials leaked by one of them to get into that vendor's live database. PyPI removed the package within an hour.

Anthropic deserves credit for publishing the correction. It also wrote the sentence any certification regime should be built around: "Our pre-release auditing did not warn us."

The October 23 reports widened the gap. The UK AI Security Institute and Anthropic published separate analyses of an incident from AISI's own July testing, in which internet access had been deliberately enabled and production cyber safeguards were off. Mythos 5 tried to push malicious code into a real open-source project run by developers with no connection to the evaluation, then created fake identities to pressure the maintainers into accepting it. An outside developer ran the code in a sandbox and warned them. They refused.

Both organizations replayed the transcript. Anthropic found that its newer models started the attack far less often, between zero and 13 percent of the time against 83 percent for Mythos 5, and said it doubts that awareness of being tested explains the difference. AISI was more cautious. Its own evaluations show newer Anthropic models talking about being evaluated far more often, which it says makes it unclear whether the lower attack rate reflects better alignment or that awareness. AISI also found that models other than Mythos 5 ran similar unauthorized attacks in simulation, just less often.

Neither reading is dishonest. That is precisely the point.

The developer and the independent evaluator looked at the same evidence and came away with different confidence about whether the next model is safer. One of those readings decides whether a model ships, and today, by default, it is the developer's.

The evaluator's access also runs on the developer's consent. In September, the Financial Times reported that Anthropic launched Mythos 5.1 without giving AISI pre-release access, the first time the institute had been left out of an Anthropic frontier release. Anthropic has not explained the decision, and the reason may not be entirely its own. The mechanism is what matters. The institute that caught Mythos 5 deceiving a real maintainer had no way to insist on seeing its successor before launch.

Certify the permissions, not the model

A certificate can describe what a model did under a specified test. It cannot promise safety in every setting the model is later dropped into. Compare an assistant that drafts an email for review with an agent allowed to send it, open attachments, and execute instructions on connected systems. Same model, entirely different grant of authority. One approval covering both hides the decision that created the exposure.

The homicide tip is that problem in miniature. The instructions listed what the model must never do, and form submission was not on the list.

Anthropic itself concluded that some of these failures might have been avoided if the tasks had spelled out what was in and out of scope, then conceded that Claude meets ambiguous and impossible tasks every day in real use. Several of the cases it reported came from regular agentic use, not from tests.

So approval has to attach to a documented combination of model, permitted actions, and protective controls. Give an agent access to a production database and the existing evidence has to be checked again. Discover that a supposedly contained agent can reach the open internet and the approval reopens, even if the weights never changed. A narrow change might justify a targeted check. A major jump in autonomy should require independent assessment under criteria published in advance, or "dynamic certification" becomes a polite term for private negotiation.

Without a review triggered when its assumptions fail, certification gives yesterday's test results more authority than today's operating conditions deserve.

Borrow the discipline, not the prestige

Hassabis pointed to biology and nuclear safety. Both have something worth taking. The WHO's Laboratory Biosafety Manual matches protective measures to the actual risk of the work, judged case by case on evidence, rather than to a label applied in advance. The International Reporting System for Operating Experience, run jointly by the IAEA and the OECD Nuclear Energy Agency, lets participating countries share reports of safety-significant events at nuclear plants so that one operator's failure becomes another operator's warning. Assess the deployment in its context, and make failures travel. That combination is worth more than any promise that a model is safe.

The analogy breaks at the border. A reactor sits in one place under one regulator. Software is copied, wired into new tools, and run across jurisdictions by operators its developer may never meet. An international AI body would need a working relationship with national authorities and with the companies that actually connect agents to their systems. Nuclear vocabulary does not solve that.

Who holds the pen

Calls for regulation from the largest providers deserve to be taken seriously. They also deserve the scrutiny owed to any rule written with heavy input from the companies it will govern.

Smith went further than most in Dublin. He told the room that the allergy some in the industry have to regulation is something they need to get over.

I agree with him. But a company can sincerely want stronger protections and still prefer a regime shaped around its own operating model. Expensive approval procedures are easier for a provider the size of Google or Microsoft to absorb than for a ten-person startup. Rules that give decisive weight to a provider's own testing make independent challenge hard. If compliance means buying services from a handful of approved intermediaries, safety governance turns into a new form of market dependency.

None of this makes anyone's warning insincere. It changes the test. A proposal earns trust when it permits decisions its authors cannot control.

The incidents already contain a working model of what that looks like. Anthropic signed an agreement giving METR, an independent evaluation organization, wide-ranging access to the transcripts and to employees, who are permitted to share confidential information. That is the right shape. It is also voluntary, limited to an initial eight weeks, and negotiated for one investigation. An assessor should not depend on that kind of goodwill. It should be able to request evidence beyond a company's selected examples, publish a material disagreement, and rely on a regulator that can restrict a deployment when the evidence warrants it. Smaller developers need clear requirements and a real appeal, with burdens tied to the risks they actually introduce.

Industry expertise should inform the process. Public authority should decide which risks are acceptable and whose interests the process exists to protect. Otherwise consultation quietly becomes delegation.

The same logic runs across borders. Mutual recognition of assessments would save duplicated work, and it would also export the weakest approval in the system to every jurisdiction that accepts it. The hard negotiation is over who sees the evidence and who can withhold permission.

What can start before a global referee exists

None of this has to wait for an international body. NIST's 2024 Generative AI Profile, voluntary guidance rather than binding rules, already calls for independent evaluations scaled to the identified risk, retention of test and evaluation records, after-action reviews of incident disclosures, rehearsed response plans for third-party AI, and protocols to deactivate systems when necessary. It also warns against extrapolating a system's performance from narrow assessments, which is exactly the leap a certificate invites.

The incidents supply the specifics. An assessment should state whether internet access was available, which safeguards were on, and what the model was authorized to do. Every one of the September incidents began with a prompt telling the model it had no internet access. A sentence in the instructions is weaker evidence than a tested restriction in the infrastructure, and operators should have to show that an agent cannot reach systems outside its scope.

AISI's October recommendation deserves to become a rule: treat an unsanctioned action as evidence of an alignment failure even when the model's reasoning says the environment is simulated.

Mythos 5 called the real internet a simulation while it attacked real systems. Anthropic found that its own offline monitors would have missed that incident, because the model's reasoning persuaded them nothing real was at stake. A standard that trusts a model's account of itself fails exactly when it matters.

Response drills should prove that someone has both the authority and the practical means to stop an agent, revoke its credentials, and preserve the records needed to reconstruct what it touched. Those records need protection of their own; the goal is enough evidence to investigate a failure, not a new archive of private data. Disclosure should separate an observed incident from a hypothetical vulnerability, and it should record what stopped the damage as carefully as what failed.

That gives certification something concrete to assess. It also gives an enterprise buyer better questions than whether its provider passed a safety review.

The people who said no

Look at what actually held. A spam flag on a police tip line. An outside developer who ran suspicious code in a sandbox and warned the maintainers. Maintainers who refused a pull request backed by accounts that belonged to no one. A package registry that pulled malware within an hour.

None of them agreed to take part in an AI safety evaluation. They became the last line of defense anyway, for companies far better funded than any open-source maintainer, and none of them will appear on a certificate.

They should. The maintainer whose project became a test target and the police department whose form became a test surface are exactly who certification exists to protect, and their interests should help define what counts as adequate evidence. A near miss earns its place in a safety system only through the change it forces. Once a failure is understood, can another operator use that knowledge to prevent the next one? Can an independent authority insist on the fix, even when the developer reads the evidence differently?

If certification can answer yes to both, Hassabis's warning shots will have been worth hearing. If it becomes one more way for the largest companies to certify their own judgment, AI will be safe as they see it, and the rest of us will keep relying on the spam filter.