On November 19, 2026, OpenAI described how it uses Codex agents to investigate and fix performance regressions in ChatGPT. Its internal system, called Performance Factory, opens a task when an alert fires and records the diagnosis, supporting evidence and proposed fix in that task. An engineer reviews the change and decides what goes into production. After deployment, the system assigns an agent to measure whether the service has returned to its previous level of performance.

A regression in this context is a deterioration in a metric for a service that previously performed at a given level. One example OpenAI gives is first-token latency, the time until a response starts arriving. The company explains that its monitoring and profiling tools predate the automation. Agents take over part of the investigation previously conducted with those tools and record their findings for the next person or session assigned to the task.

Hands on a laptop keyboard with code on screen and two monitors nearby
Christina Morillo

An engineer decides whether to ship the proposed fix

Production performance remains the final measure. When the diagnosis is accepted, a Codex session prepares a change to the code and opens a pull request for review. The engineer can adjust the proposal or use the explanation to write a different fix and decides whether to merge the change. The task report brings together the impact, proposed cause and measurements, including charts and execution traces where they support the findings.

A comment on the proposed change can make the task ready to assign to an agent again. Deployment completion starts a different step. An agent checks production metrics to compare performance after the change with the baseline. OpenAI describes three outcomes. If performance returns to baseline, the task closes. Partial recovery opens a new task for the remaining problem. If performance does not improve, the result is recorded and the original investigation resumes.

What happens after a fix is deployed
Measured resultAction described by OpenAI
Back to baselineThe agent marks the task resolved.
Partial recoveryThe original task closes. A new task tracks the remaining slowdown.
No improvementThe agent records the result and sends the task back for investigation.

The diagnosis compares similar requests and the timing of changes

An alert shows when a metric crosses a threshold. OpenAI’s team explains that the deterioration can start earlier. The agent looks back through measurements to find when the behavior changed and separates requests into comparable groups. In one situation, requests of the same kind are taking longer. In another, the service is processing a higher proportion of requests that were already slow. The two explanations point to different lines of investigation.

Once the time window and affected groups are identified, the agent examines code deployments and changes to feature configuration. Logs and request traces help check when a change reached the affected users. OpenAI introduced a review in which a separate session tries to disprove the diagnosis. The reviewer can compare exposed requests with otherwise similar requests and look at what happened to the metric when the suspected change was rolled back.

In one incident reported by OpenAI, the investigating agent linked problems in ChatGPT conversations to image download errors. The reviewing agent observed that the internal file location service had recovered while image requests were still failing. The difference invalidated the explanation for the remaining errors. The agent reopened the investigation and recorded what still needed to be established.

Printed charts, a laptop and a mug on a work desk
Gustavo Fring

The incident state persists between sessions

In the system OpenAI describes, engineers define the workflow states and transitions. A task service tracks the work and stores findings, and a dispatch mechanism starts a Codex turn when the current step is ready. Recording the state outside any one agent’s conversation lets a task move from investigation to review and allows another agent or an engineer to take it up later with the same evidence.

The work runs through an internal API in a cloud environment within OpenAI’s network. A service identity provides access to code repositories, logs, metrics, traces, CPU profiles and memory snapshots. The company uses that identity to limit permissions and identify the agents’ actions. When a task is waiting on a review comment or deployment, the coordination system watches for the event that makes it ready to assign again.

OpenAI also distinguishes a local fix from an architectural decision. A cache, for example, can reduce response time by reusing a result, but it can also delay the display of data that has since changed. The comparison includes both speed and data freshness. In these situations, an engineer who knows the service can decide what trade-off is acceptable before merging the change.

Investigations and diagnoses are measured separately

OpenAI reports that it tracked more than 1,700 investigations in just over three months and identified causes in more than 600 cases. The company describes that pace as about 17 automated investigations a day. In a separate assessment, engineers manually reviewed 100 proposed root causes and judged 70% of them accurate. The other 30% needed further refinement or had yet to establish causality sufficiently, according to OpenAI’s assessment.

The company separately tests the quality of diagnoses and the usefulness of the reports handed to engineers. In retrospective tests, a session starts with the initial signal from a real incident, prepared to avoid revealing the answer. The team separately checks whether the agent found when performance first deteriorated, which requests were affected and how it used evidence to rank possible causes. It evaluates reports on whether they explain the impact, retain useful charts and support the proposed next step.

The volume of investigations created another coordination task. An incident can appear in several alerts, and the system groups linked records and sorts them by estimated impact. OpenAI says an agent also checks recent runs every day for tool or access errors. When the same problem interrupts multiple investigations, the workflow opens a task to fix the tool, with engineers retaining responsibility for reviewing the change.