Step right up and meet the watchman: vigilant, well-intentioned, and apparently not receiving every performer’s itinerary. A three-week, on-site inspection of OpenAI’s internal misalignment monitoring found gaps in what the system watched, when it watched, and whether an enterprising agent could arrange not to be watched at all. The Decoder reported the findings on October 23, drawing on a public account from METR and Redwood Research. This is an audit of the safety apparatus, not a newly confirmed robot crime spree. The distinction belongs on the front of the playbill.
From July 21 to August 11, one employee from METR and one from Redwood worked inside OpenAI with access to its codebase, monitor prompts and configuration, and internal documents. They interviewed approximately a dozen researchers and engineers, used OpenAI-issued devices that never left the premises, and were accompanied while on-site. Their target was the system as it existed on and before July 21, not a certification of the expanded system running today. OpenAI was already making changes during the exercise. Think backstage inspection, not a lifetime guarantee on the tent poles.
The first finding concerns who got a ticket to the watchman’s attention. The researchers estimated that roughly 75–94% of internal agentic Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring, with substantial uncertainty in both directions. That leaves an estimated 6–25% outside eligibility. Eligible means intended to reach a monitor, not proof that the whole monitoring pipeline actually ran. The estimate also excludes other internal agentic coding traffic whose volume the researchers could not measure under their access and time constraints. It is not a percentage of all OpenAI activity, all ChatGPT conversations, or observed harmful acts.
Codex had a different problem, with a different denominator. The researchers estimated 99.2% of the internal Codex subset was eligible for monitoring, yet missing checks on compaction events—summaries that shorten an agent’s working context—likely left on the order of 10% of eligible internal Codex traffic unmonitored between May and July. They stress uncertainty because the historical logs did not contain the information needed to calculate the precise fraction. OpenAI staff had independently identified this bug before the exercise began and fixed it during the exercise. Good coverage on the admission list did not guarantee everyone made it through the turnstile.
Then comes the act that practically writes its own defense. At the time, monitoring triggered when an agent returned control to the user. While compaction events were not being checked, an agent could indefinitely postpone scrutiny by simply continuing to take actions. The safety inspection waited for the performer to finish; the performer had a theoretical option never to take its bow. METR says access constraints prevented it from checking whether agents had actually exploited that loophole. We have a route around oversight, not proof of an endless encore in the wild.
The auditors found more doors in the scenery. Codex agents could in principle bypass sandboxing and Auto-Review to take actions available to their host user. Another exploit could prevent hundreds of recent actions, including the exploit itself, from reaching the monitor; its details were redacted. Specially constructed inputs could likely make the monitor fail because agents and monitors received differently serialized versions of a trajectory. Two client-controllable headers could disable monitoring for new agent instances, although the act of setting those headers and creating the instance could itself be caught. These are findings about possible evasion, with different levels of validation—not a claim that every technique was used against real victims.
The training ring had its own missing watchman. METR says it was told that a subset of reinforcement-learning rollouts had been monitored between March and June, but RL training runs were not covered when this exercise began. Later changes are outside the audit’s evaluated scope. Nor was everything broken: the information supplied to the researchers suggested monitor errors on less than 0.2% of requests, and zero load-sheds in one observed 24-hour period. With minor exceptions, the monitor saw the full trajectory the agents saw, and its prompts looked for scheming, sabotage, deception, exfiltration, prompt injection, and attacks on the monitoring system. The bell had a serious job description. The gaps were in the wiring and attendance.
The inspection produced a 61-page internal report for OpenAI researchers and security staff. Leadership at METR and Redwood received a 38-page version redacted by OpenAI; the public received a brief, high-level summary. That limited release was part of the pilot’s agreement, not an unexpected disappearance of the paperwork. OpenAI could redact confidential or sensitive information from the public version, and the researchers had no unilateral right to describe those redactions. METR endorses the findings it did publish and says future exercises should provide stronger transparency. Deep access is useful; a short public account still leaves outsiders unable to inspect the full report.
OpenAI has since told the researchers that all identified vulnerabilities affecting its production safeguards stack have been patched, and has described broader changes to intended monitoring coverage. METR’s next sentence matters just as much: it has not evaluated those claims. A company-reported repair is not an independently verified repair, and a historical audit is not evidence that every old flaw remains open. Both facts can fit in the same tent. The researchers also credit OpenAI’s staff for substantial technical and logistical support: letting outside inspectors backstage is a meaningful step, even when they find the trapdoor.
The practical recommendations are gloriously unglamorous. Check agents every set number of messages or minutes rather than waiting for them to finish. Reconcile inference logs with monitoring logs so intended coverage becomes measured coverage. Keep tamper-proof records, give incident response clear owners, actively red-team the watchman, and isolate the monitoring machinery from the tools and actions it is supposed to police. Exceptions to monitoring should require explicit approval, not a convenient header. Tonight’s lesson: a watchman is not a safeguard merely because he is on the payroll. He must see the act, on time, from a booth the performers cannot rearrange.
