CompCode Academy

Lesson 1 of 4 · Free to read

Session lesson

On-call, Postmortems & Chaos

Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-07-perf-and-sre

Why this matters (5 minutes)

Alerts without process are noise. Chaos without a steady-state hypothesis is a stunt. MNCs formalize on-call, blameless postmortems, and declarative fault injection.

Learning outcomes

  • Design escalation and reduce alert fatigue
  • Run a game day with a written timeline
  • Write a blameless postmortem
  • Run Chaos Mesh experiments against SLO hypotheses

Shared lab constants

Systemapi (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07)
StackGrafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy
RuleExtend the same repo every phase — never start a new toy app.

What MNCs do in 2026

Grafana OnCall / PagerDuty-style routing, blameless postmortems, game days, and Chaos Mesh (PodChaos/NetworkChaos/StressChaos) are common reliability practices.

Teachable chunks (compressed from days 71–80)

On-call is a product

Schedules, escalation, handoffs, and health metrics (pages/week, MTTA).

Alert fatigue

Symptom alerts, inhibition, and deleting noisy rules are engineering work.

Blameless postmortems

Timeline, contributing factors, action items with owners — no villains.

Chaos Mesh loop

Hypothesis → inject real-world fault → measure SLO → minimize blast radius → catalog.

Mental model

Watch & learn

Course video

Open on YouTube
eBPF for Observability (CNCF) — What eBPF can and cannot see — zero-code signals vs SDK instrumentation.

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Worked example

Write hypothesis.md tied to phase-07 p95/error series. Install Chaos Mesh. Run PodChaos on worker; confirm self-heal within error budget. Add NetworkChaos delay to LocalStack path; observe Tempo + p95.

Hands-on path

Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.

Phase checklist

  • On-call doc exists
  • Postmortem written
  • Chaos Mesh hypothesis loop demonstrated

Key takeaways

  • Keep the same order pipeline — depth beats new toy apps
  • Correlate signals before you SSH
  • Write policies and catalogs your future on-call self can use

Credit

Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.


Observability phase lesson — On-call, Postmortems & Chaos.

New free lessons and course updates. Unsubscribe from any email.