Lesson 1 of 4 · Free to read
Session lesson
On-call, Postmortems & Chaos
Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-07-perf-and-sre
Why this matters (5 minutes)
Alerts without process are noise. Chaos without a steady-state hypothesis is a stunt. MNCs formalize on-call, blameless postmortems, and declarative fault injection.
Learning outcomes
- Design escalation and reduce alert fatigue
- Run a game day with a written timeline
- Write a blameless postmortem
- Run Chaos Mesh experiments against SLO hypotheses
Shared lab constants
| System | api (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07) |
| Stack | Grafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy |
| Rule | Extend the same repo every phase — never start a new toy app. |
What MNCs do in 2026
Grafana OnCall / PagerDuty-style routing, blameless postmortems, game days, and Chaos Mesh (PodChaos/NetworkChaos/StressChaos) are common reliability practices.
Teachable chunks (compressed from days 71–80)
On-call is a product
Schedules, escalation, handoffs, and health metrics (pages/week, MTTA).
Alert fatigue
Symptom alerts, inhibition, and deleting noisy rules are engineering work.
Blameless postmortems
Timeline, contributing factors, action items with owners — no villains.
Chaos Mesh loop
Hypothesis → inject real-world fault → measure SLO → minimize blast radius → catalog.
Mental model
Watch & learn
Course video
Course video
Worked example
Write hypothesis.md tied to phase-07 p95/error series. Install Chaos Mesh. Run PodChaos on worker; confirm self-heal within error budget. Add NetworkChaos delay to LocalStack path; observe Tempo + p95.
Hands-on path
Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.
Phase checklist
- On-call doc exists
- Postmortem written
- Chaos Mesh hypothesis loop demonstrated
Key takeaways
- Keep the same order pipeline — depth beats new toy apps
- Correlate signals before you SSH
- Write policies and catalogs your future on-call self can use
Credit
Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.
Observability phase lesson — On-call, Postmortems & Chaos.