The Free Training Set Most Teams Ignore
Every postmortem your team writes is a labeled example: this change caused this outcome. Most teams treat postmortems as historical artifacts. The teams that wire postmortems into their AI coding pipeline turn each incident into a permanent guardrail that prevents the next instance of the same class of mistake.
This is not theoretical. Across the deployments we operate, the teams that maintain a postmortem-to-prompt pipeline see escape rate drop 40-60% in the second quarter of operation, with most of the improvement attributable to incident-derived rules.
What Counts As A Useful Incident
Not every postmortem is wired worthy. The useful ones share three properties:
- The root cause is a pattern, not a one-off. A typo in a config file is not generalizable. A null-pointer dereference at a layer boundary is.
- The fix is articulable as a rule. "Always check for empty arrays before indexing into them" is a rule. "Be more careful" is not.
- The pattern is repeatable. If the AI can produce another version of this mistake in another file or another repo, the rule will catch it. If the mistake is unique to one specific historical context, the rule will not generalize.
Our team rejects about 35% of postmortems as not pipeline-worthy. The remaining 65% become rules.
The Rule Format
A rule has four parts:
- Trigger: A code pattern detectable by the AI's pre-PR validator. Often expressed as a regex, an AST pattern, or a semantic test.
- Reason: A one-sentence statement of why the pattern is dangerous, with a link to the incident.
- Remediation: The specific safer pattern to use instead.
- Scope: Which repositories and which file globs the rule applies to.
Example:
Trigger: Calls to db.execute() without preceding transaction.begin()
Reason: Incident 2026-Q1-014: missing transaction wrapper caused partial writes during the migration outage
Remediation: Wrap multi-statement writes in transaction.begin()/.commit()
Scope: services/payments/**, services/billing/**The rule is concrete, scoped, and tied back to an incident. When the validator flags it, the reviewer sees the incident reference and understands immediately why.
The Wiring
The flow that works:
- Postmortem closes. The action items include "add validator rule" alongside the engineering fix.
- Rule is drafted by the postmortem author. They know the pattern best. Drafting takes 10 minutes.
- Rule goes through a review. Same review process as a code change. Validate the regex, validate the scope, validate that it does not produce false positives on existing healthy code.
- Rule deploys to the validator. Now every AI-generated PR is checked against it. A hit blocks the PR or routes it to elevated human review.
- Rule is monitored. False positive rate over time. If a rule starts firing on healthy code, it gets tuned or retired.
Most teams stop at step 1. The work to operationalize is real but small relative to the savings.
What To Avoid
Three patterns that look right and fail in practice:
Over-broad rules. "Never use raw SQL" is too broad. It will block legitimate uses and the team will turn it off. Rules should be scoped to the actual conditions of the incident.
One-off rules with no scope discipline. Every incident gets a rule applied repo-wide, and a year in you have 800 rules, half of which fire on every PR. The validator becomes noise. Tag rules with scope and prune them on a schedule.
Rules without remediation. "This pattern is dangerous, do not use it" leaves the AI without guidance on what to do instead. The next attempt produces a variant of the same dangerous pattern. Always include a remediation that the AI can follow.
The Per-Rule Decay
Rules age. A rule written for a service that was deprecated six months ago is dead weight. We review rules quarterly:
- How many times did the rule fire?
- What was the true-positive rate?
- Did the underlying code area still exist?
- Was the incident still considered representative?
Rules that fail any of these get retired. Roughly 15% of rules retire each quarter. The rule set stays sharp because we prune it.
A Concrete Win
One team's first quarter postmortem set produced 23 rules. Across the second quarter, those rules fired 412 times in AI-generated PRs. The validator blocked or escalated those PRs before they reached production. Estimated incidents avoided, using the team's historical incident-per-PR rate for the affected categories: 4-7 for the quarter.
This is the durable kind of progress. Each incident, instead of being a sunk cost, pays a dividend forever.
What To Build First
If you are starting:
- Pick the three most recent postmortems with concrete pattern-shaped root causes.
- Draft a rule for each, in the format above.
- Wire them into the AI's pre-PR validator (or a CI check, if you do not have one yet).
- Watch them fire for a month.
- Tune false positives. Retire any that are not catching real issues.
The first three rules are the hardest. After that, the pattern is yours and incidents stop being just bad news.
For more on the validator architecture, see enterprise safety layers. For how this fits into a broader learning loop, see self-improving AI from code reviews.
Frequently asked questions
How do you turn postmortems into AI coding guardrails?
Make 'add validator rule' a standard action item on every qualifying postmortem. The author drafts a scoped rule, it goes through the same review as a code change, it deploys to the pre-PR validator, and it's then monitored for false positives. Every AI-generated PR is checked against it, and a hit either blocks the PR or routes it to elevated human review. For the validator architecture this plugs into, see enterprise safety for AI-generated code.
What makes an incident useful for training an AI coding agent?
Three properties: the root cause is a repeatable pattern rather than a one-off (a null-pointer dereference at a layer boundary qualifies, a config typo doesn't), the fix can be stated as a concrete rule ('always check for empty arrays before indexing'), and the agent could plausibly reproduce the mistake elsewhere. In practice roughly a third of postmortems are rejected as not generalizable.
How do you write a validator rule from a postmortem?
Give it four parts: a trigger (a code pattern the validator can detect via regex, AST, or a semantic test), a reason (one sentence on why it's dangerous, linked to the incident ID), a remediation (the specific safer pattern to use instead), and a scope (the repositories and file globs it applies to). The postmortem author drafts it in about ten minutes because they know the pattern best.
How do you keep AI safety rules from becoming noise?
Scope every rule to the actual conditions of its incident, always include a remediation, and prune on a schedule. Review rules quarterly on how often they fired, their true-positive rate, whether the code area still exists, and whether the incident is still representative, roughly 15% retire each quarter. Over-broad rules like 'never use raw SQL' get switched off by the team and take real rules down with them.
Do incident-derived rules actually reduce production bugs?
Yes. In one team, a first-quarter set of 23 rules fired 412 times in AI-generated PRs the next quarter, blocking or escalating each before it reached production, an estimated 4 to 7 incidents avoided based on the team's historical incident-per-PR rate. Each incident stops being a sunk cost and starts paying a dividend. This is part of a broader learning loop described in how self-improving AI learns from code reviews.
EnsureFix Engineering Team
The EnsureFix engineering team designs and operates the multi-agent pipeline that turns tickets into production-ready pull requests. They write about architecture, model routing, safety validation, and what actually ships in enterprise codebases.