AI agents can investigate production incidents and suggest fixes. As they gain access to operational systems, engineers need to understand what the agents can execute and how those actions are controlled.
An application starts returning HTTP 500 errors shortly after a production deployment. The on-call engineer receives an alert, but an AI incident-response agent has already started investigating.
It checks application logs, recent deployments and monitoring data. The agent identifies the latest release as a possible cause and recommends rolling it back.
The timing makes the recommendation plausible. However, the logs also show database connection failures. The engineer needs to establish whether the deployment caused those failures before reversing it.
This fictional scenario illustrates a decision that becomes more important as AI agents move beyond answering technical questions and begin interacting with production systems.
AI agents are already part of incident management
Microsoft’s Azure SRE Agent, generally available since March 2026, can investigate incidents, examine operational information and recommend or execute permitted remediation actions. Google’s Gemini Cloud Assist also supports AI-assisted infrastructure investigations, although its investigation capability remains in Preview with restricted access.
These tools can reduce the manual effort involved in gathering information from different sources. Instead of opening several dashboards and searching deployment histories, engineers may receive an initial investigation containing relevant logs, changes and possible explanations.
The usefulness depends partly on the quality of the information available. An agent investigating an application with incomplete traces, inconsistent logging or missing deployment records faces many of the same difficulties as an engineer working with those systems.
There is also a difference between identifying an event associated with a failure and establishing its cause. A deployment appearing immediately before an outage is worth investigating, but its timing alone does not justify a rollback.
What happens when the agent attempts a repair?
Research published in September 2026 offers some insight into the difficulties of autonomous incident response.
The Incident-Arena benchmark evaluated AI agents against 20 fault-injected incidents involving deployed applications. The evaluated frontier models scored below 64.3% on its evaluation, with failures including incomplete repairs and unsafe regressions.
Another benchmark, SRE-Marathon, examined agents working with overlapping failures in continuously running Kubernetes environments. The research found that identifying problems was often easier than completing repairs while the environment continued to change.
Both studies are preprints, and neither establishes the performance of commercial incident-response products. Their results depend on the environments, tasks and evaluation methods used.
They are nevertheless relevant to the rollback scenario.
Suppose the deployment included a database migration that changed the structure expected by the application. Rolling back the application alone might introduce additional compatibility problems.
An engineer investigating the incident would need to understand that dependency before approving the proposed action. An AI agent attempting the same repair needs access to the relevant information and a reliable way to account for the consequences.
Production access changes the responsibility
An agent that reads logs has a different operational role from one that can restart services, roll back deployments or modify infrastructure.
Azure SRE Agent supports configurable execution modes, including review controls for certain Azure infrastructure operations. However, its documentation distinguishes those operations from actions available through other connected tools.
This is an important detail for teams evaluating agent-based automation.
A review setting should not be assumed to cover every integration. The actual permissions depend on the tools connected to the agent and how their access policies are configured.
For example, an agent might have read-only access to monitoring data but also possess credentials for a deployment system capable of modifying production.
Even when approval is required, the engineer needs enough information to understand the proposed action. In our scenario, approving a rollback requires more than agreeing that the latest deployment looks suspicious.
The approval process should expose the intended operation, affected resources and relevant evidence so the engineer can assess its consequences.
What should engineers prepare for?
The immediate change for SRE and platform teams may be in how investigations are conducted.
Engineers may spend less time gathering routine diagnostic information and more time evaluating the explanations and recovery options produced by an agent.
That still requires detailed knowledge of application architecture, dependencies, deployment procedures and observability.
There is also a potential longer-term consequence. If routine incidents are increasingly handled through automation, junior engineers may have fewer opportunities to practise troubleshooting. Organizations will need to consider how that experience is developed, particularly when an unfamiliar failure requires a human response.
This remains a concern worth examining rather than a demonstrated consequence of current incident-response products.
Existing runbooks can help define how agents operate. A useful runbook describes the conditions under which a recovery action is appropriate, the steps required and how successful recovery should be verified.
Teams can begin by allowing agents to investigate and recommend actions while keeping production changes under established approval processes. More automated execution can be evaluated for specific, well-understood operations with appropriate safeguards.
Returning to the opening incident, the agent may have identified the relevant deployment and database errors quickly. Whether rolling back is appropriate still depends on the relationship between those events and the state of the application.
That is where the engineer’s knowledge of the system remains essential.
References and further reading
Each source was opened and checked on 14 October 2026. The two research papers are preprints, and the Gemini Cloud Assist capability is in Preview with restricted access.
- Microsoft: Azure SRE Agent overview. Product capabilities and investigation workflows.
- Microsoft: Azure SRE Agent run modes. Review and autonomous modes, including where approval applies and where other controls are needed.
- Microsoft: Azure SRE Agent general availability announcement. Published March 10, 2026. A vendor announcement, not independent evidence of effectiveness.
- Google Cloud: Gemini Cloud Assist investigations. Preview status and access limitations.
- Fu et al.: Incident-Arena: Getting agents to the last nine of reliability. Preprint submitted September 30, 2026. Benchmark-specific findings.
- Tian et al.: SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents. Preprint submitted September 26, 2026. Continuous-fault benchmark.