AI incidents are not limited to system outages. A workflow can be technically online and still produce a bad customer message, update the wrong field, expose sensitive data, route work incorrectly, or make a recommendation that people trust too quickly.
The response plan should be written before launch. When a bad output appears, the team should not be debating who owns the workflow, whether to stop the agent, or where the audit evidence lives.
Treat bad AI output as an operational incident when it changes work
If the output affects customers, records, money, access, compliance, or downstream decisions, respond with a runbook, not a Slack thread.
Triage the incident by impact
The first move is classification. Did the issue stay inside a draft? Did a human catch it before action? Did it reach a customer? Did it write into a system of record? Did it expose sensitive information? Did it repeat across multiple runs?
IBM’s Cost of a Data Breach Report is a reminder that security and data incidents quickly become business events. The 2025 report puts the average breach cost at $4.4M and notes that 97% of organizations with an AI-related incident lacked proper AI access controls. For AI workflows, the impact review must include data exposure, operational downtime, rework, customer trust, regulatory obligations, and whether permissions or governance failed before the bad output appeared.
The incident runbook
AI workflow incident runbook
Use this runbook to turn a bad output into containment, correction, explanation, and prevention.
Response step: Detect
- Owner question
- Who can declare that an AI workflow produced a harmful or suspicious result?
- Evidence needed
- Run ID, reporter, workflow name, output, affected object, and initial severity
Response step: Contain
- Owner question
- Should the workflow, tool, permission, prompt, retrieval source, or write-back be paused?
- Evidence needed
- Current active runs, tool scopes, recent changes, and blast-radius limits
Response step: Correct
- Owner question
- What records, messages, decisions, or customer interactions need repair?
- Evidence needed
- Before/after state, rollback path, customer list, and reviewer decision
Response step: Explain
- Owner question
- Why did the workflow produce the bad output?
- Evidence needed
- Prompt version, model version, retrieval bundle, source data, policy checks, and human edits
Response step: Prevent
- Owner question
- Which eval, approval gate, permission, or monitoring rule must change before restart?
- Evidence needed
- Root cause, test case, owner, deployment note, and restart approval
Containment is not always a full shutdown
Teams need more than one switch. Sometimes the right response is to disable customer send while keeping draft generation. Sometimes it is to remove write access but keep read-only analysis. Sometimes it is to block a specific retrieval source that carried bad context.
flowchart LR
A["Bad output reported"]
A --> B["Classify impact"]
B --> C["Pause risky capability"]
C --> D["Repair affected work"]
D --> E["Root cause review"]
E --> F["Eval or control update"]
F --> G["Restart approval"] The OWASP LLM Top 10 helps incident teams look beyond “the model was wrong.” Was there prompt injection from an email or document? Sensitive information disclosure from retrieval? Improper output handling before a tool call? Excessive agency in a write-back? Vector or embedding weakness that retrieved the wrong source? A bad output is often a symptom of a missing system control.
Close the loop with evals and ownership
The NIST AI RMF emphasizes ongoing measurement and management across the AI lifecycle. Incident response should create that loop. Every meaningful incident should produce at least one concrete change: a regression eval, a permission limit, a new approval gate, a better retrieval filter, a monitoring rule, or a training update for reviewers.
Metacto Continuous AI Operations is the operating home for that loop: monitoring, evals, tuning, incidents, runbooks, and monthly reviews are how production AI gets better because incidents are captured and converted into system changes.
The incident is not closed when the bad output is fixed. It is closed when the same class of failure is less likely to happen again.