IRCNF
Claude Opus 4.7 (recommended for nuanced blameless rewrites); Claude Sonnet 4.6 or GPT-4o work well for straightforward incidentsYour team resolved a P0 outage two hours ago. The on-call engineer has a Slack thread, a PagerDuty timeline, and some runbook notes — but nothing structured. You need a complete post-mortem document ready for the engineering all-hands tomorrow morning, and you have thirty minutes.Developer Tools

Generate a complete blameless post-mortem from a raw incident timeline

Share:
Generate a complete blameless post-mortem from a raw incident timeline

Why this prompt matters

Post-mortems that skip the root cause / contributing factors distinction produce action items that fix symptoms, not systems. Teams that consistently write shallow post-mortems repeat the same incident categories — different service, same missing circuit breaker, same absent alert. The blameless framing is also not optional: post-mortem cultures where engineers feel implicitly blamed for outages produce teams that under-report near-misses, which means preventable P0s go undetected until they actually happen.

What we use it for

Your team resolved a P0 outage two hours ago. The on-call engineer has a Slack thread, a PagerDuty timeline, and some runbook notes — but nothing structured. You need a complete post-mortem document ready for the engineering all-hands tomorrow morning, and you have thirty minutes.

Prompt

Act as a senior site reliability engineer with extensive experience writing blameless post-mortems for high-traffic systems.

Context:
The following incident has been resolved. You have been given a raw timeline of events, the contributing factors identified during the retrospective, and the remediation steps the team took.

Incident details:
- Service affected: [SERVICE NAME, e.g., "Payment API", "User Authentication Service"]
- Severity: [P0/P1/P2]
- Duration: [START TIME] to [END TIME] ([TOTAL DURATION])
- Customer impact: [DESCRIBE IMPACT, e.g., "100% of checkout requests failed for 47 minutes"]
- Raw timeline / notes: [PASTE YOUR INCIDENT NOTES, SLACK THREAD, OR RUNBOOK ENTRIES HERE]

Task:
Write a complete, professional blameless post-mortem document following Google's SRE post-mortem culture principles. The document must identify system failures and process gaps — never individual blame.

Constraints:
- Use blameless language throughout. Say "the deployment pipeline did not have a gate for X" not "the engineer forgot to check X"
- Distinguish between root cause (the fundamental system or process failure) and contributing factors (conditions that allowed the root cause to have impact)
- Action items must be specific, ownable, and measurable — not vague ("improve monitoring")
- Do not pad the timeline. Only include events that affected the incident trajectory
- If the raw notes contain blame language, neutralize it in the post-mortem

Output Format:
**Incident Post-Mortem: [INCIDENT TITLE]**
**Date:** [DATE]  **Severity:** [P0/P1/P2]  **Duration:** [X hours Y minutes]  **Status:** Resolved

**Executive Summary**
[2-3 sentences: what failed, for how long, customer impact, and status]

**Timeline** (all times in [TIMEZONE])
[Chronological bullet list: time → what happened / who detected it / what action was taken]

**Root Cause**
[Single paragraph identifying the fundamental system or process failure. No blame language.]

**Contributing Factors**
[Bulleted list: each factor that allowed the root cause to cause impact]

**Impact**
[Quantified: % of users affected, requests failed, revenue exposure if known, SLA breach if applicable]

**What Went Well**
[Honest assessment: detection speed, communication, rollback success, etc.]

**What Went Poorly**
[Honest, blameless assessment: slow escalation paths, missing runbooks, unclear ownership, etc.]

**Action Items**
| Priority | Action | Owner Role | Due |
|---|---|---|---|
| P0 | [Specific action] | [Role, not name] | [DATE] |

**Lessons Learned**
[2-3 sentences: the systemic lesson this incident teaches, framed for future decision-making]

Result

**Incident Post-Mortem: Payment API Outage — Connection Pool Exhaustion**

**Date:** 2026-05-14 **Severity:** P0 **Duration:** 1 hour 23 minutes **Status:** Resolved

**Executive Summary**

A deployment of Payment API v4.17.2 introduced a database query that did not release connections under timeout conditions, exhausting the PostgreSQL connection pool within 34 minutes of deploy. Checkout requests failed for 100% of users between 14:02 and 15:25 UTC. No payment data was lost; all in-flight transactions were rolled back cleanly by PostgreSQL.

**Timeline** (all times UTC)

- 13:31 — Payment API v4.17.2 deployed to production via automated pipeline. Health checks passed.

- 13:58 — First spike in payment_api_p99_latency (180ms → 2.4s). Alert fires to on-call channel.

- 14:02 — db_pool_available_connections hits zero. Checkout endpoint returns 503 for all requests.

- 14:09 — On-call engineer identifies connection pool exhaustion in Datadog. Escalates to database team.

- 14:17 — Database team identifies new query in v4.17.2 as source of leaked connections. Initiates rollback.

- 14:31 — Rollback to v4.17.1 begins after second-engineer sign-off.

- 14:37 — Rollback complete. Connection pool begins recovering.

- 15:25 — All metrics return to baseline. Incident resolved.

**Root Cause**

The new getPaymentMethodDetails() query introduced in v4.17.2 opened a database transaction but did not include a timeout handler for the network I/O path. Under elevated latency from the payment provider API (which was experiencing independent slowdowns), the transaction blocked indefinitely without releasing its connection back to the pool. With no connection limit per query type and no circuit breaker on the I/O path, the pool exhausted completely within 34 minutes of the deploy.

**Contributing Factors**

- The CI pipeline had no static analysis check for connection lifecycle management in database queries

- The staging environment uses a connection pool 10× smaller than production, masking this failure mode at staging load

- The rollback approval process requires a second engineer sign-off with no defined fast-path escalation, adding 14 minutes to recovery time

- No alert existed for connection pool availability below 20%; the first alert fired only after the pool was already fully exhausted

**Impact**

- 100% of checkout requests failed for 83 minutes

- ~14,200 failed checkout attempts (at pre-incident rate of 171/min)

- No SLA breach (threshold: 120 minutes for P0 incidents)

- No payment data loss; all in-flight transactions rolled back cleanly

**What Went Well**

- Latency alerting fired within 27 minutes of deploy — well within the team's 30-minute detection target

- Database team identified the specific query responsible within 8 minutes of escalation

- The rollback itself executed in 6 minutes with no secondary failures or data inconsistencies

**What Went Poorly**

- The rollback approval gate added 14 minutes with no fast-path for P0 incidents during peak hours

- No connection pool headroom alert meant the service transitioned from 100% available to 0% with no intermediate warning

- The PR review checklist does not include connection lifecycle verification for database query changes

**Action Items**

| Priority | Action | Owner Role | Due |

|---|---|---|---|

| P0 | Add alert for db_pool_available_connections below 30% | On-Call Infrastructure | 2026-05-21 |

| P0 | Define P0 fast-path rollback approval (max 5 min, auto-escalate) | Engineering Manager | 2026-05-21 |

| P1 | Add CI linting rule for unguarded database transactions | Backend Platform | 2026-05-28 |

| P1 | Scale staging connection pool to 1:3 ratio of production | Database Reliability | 2026-06-04 |

| P2 | Add timeout handler to all payment provider I/O paths | Payments Team | 2026-06-11 |

**Lessons Learned**

Connection lifecycle management requires both explicit code review verification and automated tooling — relying on developer attention alone is insufficient at current deployment frequency. The contributing factors identified here (missing alerts, undersized staging, slow rollback gates) are systemic patterns worth auditing across all critical services, not just the payment stack.

<p>Post-mortems written under time pressure — or days after the adrenaline fades — tend to be shallow. They miss contributing factors, skip the systemic patterns that made the incident possible, or quietly assign blame through word choice. This prompt forces the structure that makes post-mortems actually useful: blameless language, root cause separated from contributing factors, and action items specific enough to close.</p>

<h2>The prompt</h2> <p>Copy this in full. Replace every bracketed field with your actual incident details before running it.</p>

<pre><code>Act as a senior site reliability engineer with extensive experience writing blameless post-mortems for high-traffic systems.

Context: The following incident has been resolved. You have been given a raw timeline of events, the contributing factors identified during the retrospective, and the remediation steps the team took.

Incident details: - Service affected: [SERVICE NAME, e.g., "Payment API", "User Authentication Service"] - Severity: [P0/P1/P2] - Duration: [START TIME] to [END TIME] ([TOTAL DURATION]) - Customer impact: [DESCRIBE IMPACT, e.g., "100% of checkout requests failed for 47 minutes"] - Raw timeline / notes: [PASTE YOUR INCIDENT NOTES, SLACK THREAD, OR RUNBOOK ENTRIES HERE]

Task: Write a complete, professional blameless post-mortem document following Google's SRE post-mortem culture principles. The document must identify system failures and process gaps — never individual blame.

Constraints: - Use blameless language throughout. Say "the deployment pipeline did not have a gate for X" not "the engineer forgot to check X" - Distinguish between root cause (the fundamental system or process failure) and contributing factors (conditions that allowed the root cause to have impact) - Action items must be specific, ownable, and measurable — not vague ("improve monitoring") - Do not pad the timeline. Only include events that affected the incident trajectory - If the raw notes contain blame language, neutralize it in the post-mortem

Output Format: **Incident Post-Mortem: [INCIDENT TITLE]** **Date:** [DATE] **Severity:** [P0/P1/P2] **Duration:** [X hours Y minutes] **Status:** Resolved

**Executive Summary** [2-3 sentences: what failed, for how long, customer impact, and status]

**Timeline** (all times in [TIMEZONE]) [Chronological bullet list: time → what happened / who detected it / what action was taken]

**Root Cause** [Single paragraph identifying the fundamental system or process failure. No blame language.]

**Contributing Factors** [Bulleted list: each factor that allowed the root cause to cause impact]

**Impact** [Quantified: % of users affected, requests failed, revenue exposure if known, SLA breach if applicable]

**What Went Well** [Honest assessment: detection speed, communication, rollback success, etc.]

**What Went Poorly** [Honest, blameless assessment: slow escalation paths, missing runbooks, unclear ownership, etc.]

**Action Items** | Priority | Action | Owner Role | Due | |---|---|---|---| | P0 | [Specific action] | [Role, not name] | [DATE] |

**Lessons Learned** [2-3 sentences: the systemic lesson this incident teaches, framed for future decision-making]</code></pre>

<h2>Why each section is there</h2> <p>The distinction between <strong>root cause</strong> and <strong>contributing factors</strong> is the most important structural decision in this prompt. Without it, teams write "the database fell over" as the root cause and call it done. The prompt forces you to go deeper: what property of the system made "the database falling over" possible? Was there no circuit breaker? No connection pool alert? An undersized staging environment that masked the failure? Those are the contributing factors — and they're what you actually fix.</p>

<p><strong>Blameless language</strong> is a constraint, not a tone preference. The prompt explicitly tells the model to neutralize blame language if it appears in the raw notes you paste in. This matters because raw Slack threads and incident channels are full of statements like "John pushed without reviewing the runbook" — which, reframed systemically, becomes "the deployment checklist did not include a mandatory runbook review step for this service type." The fix is the same, but one version builds a culture where engineers report near-misses and the other one doesn't.</p>

<p>The <strong>action items table</strong> is formatted with Role (not name) and Due Date deliberately. Post-mortems that assign action items to named individuals fail when that person leaves the team. Assigning to a role makes the item durable through org changes. Due dates force prioritization; without them, P1 items sit in a backlog for months.</p>

<h2>Example output</h2> <p>Here is the output this prompt produces when given a connection pool exhaustion incident as input:</p>

<p><strong>Incident Post-Mortem: Payment API Outage — Connection Pool Exhaustion</strong><br/> <strong>Date:</strong> 2026-05-14 &nbsp; <strong>Severity:</strong> P0 &nbsp; <strong>Duration:</strong> 1 hour 23 minutes &nbsp; <strong>Status:</strong> Resolved</p>

<p><strong>Executive Summary</strong><br/> A deployment of Payment API v4.17.2 introduced a database query that did not release connections under timeout conditions, exhausting the PostgreSQL connection pool within 34 minutes of deploy. Checkout requests failed for 100% of users between 14:02 and 15:25 UTC. No payment data was lost; all in-flight transactions were rolled back cleanly.</p>

<p><strong>Root Cause</strong><br/> The new <code>getPaymentMethodDetails()</code> query opened a database transaction but did not include a timeout handler for the network I/O path. Under elevated latency from the payment provider API, the transaction would block indefinitely without releasing its connection. With no connection limit per query type and no circuit breaker, the pool exhausted completely within 34 minutes.</p>

<p><strong>Contributing Factors</strong></p> <ul> <li>The CI pipeline had no static analysis check for connection lifecycle management in database queries</li> <li>The staging environment uses a connection pool 10× smaller than production, preventing this failure mode from appearing in load testing at staging scale</li> <li>The rollback approval process required a second engineer sign-off, adding 14 minutes to recovery time with no defined escalation if the approver was unavailable</li> <li>No alert existed for connection pool availability below 20% — the first alert fired only after the pool was already exhausted</li> </ul>

<p><strong>Action Items (excerpt)</strong></p> <ul> <li>P0: Add connection pool headroom alert at 30% remaining — On-Call Infrastructure — 2026-05-21</li> <li>P0: Define fast-path rollback approval for P0 incidents (max 5 min) — Engineering Manager — 2026-05-21</li> <li>P1: Add linting rule to CI for unguarded database transactions — Backend Platform — 2026-05-28</li> </ul>

<h2>Getting the most out of the prompt</h2> <p>The quality of the output scales directly with the quality of your raw input. A Slack thread with timestamps and technical details will produce a more accurate timeline than a paragraph summary. If your incident notes are thin, add a second pass: after the initial post-mortem output, follow up with "Now audit the action items — are any of them vague or unownable? If so, rewrite them to be specific." The model will tighten them.</p>

<p>For recurring incident types (deploy-related outages, database issues, third-party API failures), you can build a library of past post-mortems and ask the model to compare the new one: "Here are three previous post-mortems from our payment service. What contributing factors appear in this new incident that also appeared in previous ones?" This surfaces systemic patterns that individual post-mortems miss.</p>

<p>Claude Opus 4.7 produces the most nuanced blameless rewrites — it reliably catches subtle blame language that GPT-4o sometimes leaves in. For straightforward incidents with clean raw notes, Claude Sonnet 4.6 or GPT-4o are fast and sufficient.</p>

productivityincident-responsepost-mortemsredevopsblameless-culture
Share: