OpenAI Audit Finds Critical Monitoring Flaws

OpenAI faced fresh scrutiny after a three-week on-site audit revealed substantial gaps in its internal safety monitoring systems. Researchers from METR and Redwood Research conducted the assessment, which concluded earlier this month, exposing vulnerabilities in how the company tracks the behavior of its most advanced models.

The independent team was granted deep access to OpenAI’s infrastructure from July 21 to August 11. During this period, they reviewed the company’s codebase and internal documentation while interviewing approximately a dozen employees. Their primary focus was the internal monitoring framework designed to detect rogue or unexpected behavior in agentic AI systems.

The auditors identified significant blind spots in coverage. They estimated that between six and twenty-five percent of internal agentic Responses API traffic for models at the GPT-5.5 class and above was not monitored at all. METR noted that this figure carries significant uncertainty, though the potential scale of untracked traffic is concerning. Additionally, for internal Codex traffic that was supposed to be covered, roughly ten percent went untracked between May and July due to a specific technical gap.

This gap stemmed from how the system handled compaction events, which are summaries of context. During this period, agents could delay oversight by continuing to execute actions rather than handing control back to the user. The auditors also discovered methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, model behavior during reinforcement learning training was not being monitored at the start of the audit.

OpenAI stated it independently discovered the compaction bug and fixed it while the audit was still underway. The company also paused reinforcement learning training following the Hugging Face incident in August and subsequently expanded monitoring coverage. However, these corrective measures occurred after the auditors’ access period ended, meaning they were not part of the formal evaluation. OpenAI has since claimed to have fixed all the weaknesses identified by the external team.

Transparency remains a point of friction. While the full sixty-one-page report was handed to OpenAI internally, the company applied redactions before sharing it with the researchers. METR and Redwood Research leadership received a thirty-eight-page version, but only a short summary has been made available to the public. METR stated it lacked the right to disclose the redacted portions independently. As this audit served as a pilot for deeper external access, METR expressed a desire to push for greater public transparency in future evaluations.

The findings highlight the complexities of monitoring sophisticated AI systems in production environments. While OpenAI has moved to address the specific issues raised, the significant uncertainties in the auditors' estimates leave open questions about the extent of previous blind spots. The pilot program marks a shift toward allowing outside researchers to examine internal infrastructure more closely, setting a precedent for future industry audits.