Lesson 2 of 4 · Free to read
Exercises
Exercises — On-call, Postmortems & Chaos
Watch & learn
Course video
Course video
Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.
How to use this file
- Work Lab A → B → C on the same
orders-labrepo - Prefer Docker Desktop; use Kind when the lab says so
- Definition of Done at the bottom is the phase gate
Lab A — On-call design
Time: 45 min
Goal: Escalation doc + alert review.
Setup
Existing alerts from earlier phases.
Steps
- Draft primary/secondary escalation and handoff checklist.
- Delete or silence one noisy alert with justification.
- Define MTTA/MTTR tracking fields for your team.
Success criteria
-
oncall.mdcommitted - Noise reduction note written
If stuck
If you lack a real pager, simulate with a Slack webhook or email sink.
Lab B — Game day + postmortem
Time: 60–90 min
Goal: Controlled incident with blameless write-up.
Setup
Dashboards + known fault script.
Steps
- Pick a fault (kill worker, block S3, raise latency).
- Run game day with timekeeper; note detections and actions.
- Write postmortem with action items.
Success criteria
- Postmortem merged
- At least two action items assigned
If stuck
Keep blast radius to local Kind/compose — never production.
Lab C — Chaos Mesh
Time: 60–90 min
Goal: Declarative faults with SLO hypothesis.
Setup
Kind cluster; Helm.
Steps
- Helm install Chaos Mesh; verify CRDs.
- Write steady-state hypothesis using phase-07 series.
- Apply PodChaos, NetworkChaos, StressChaos once each.
- Create
chaos-experiment-catalog.mdreframing past ad-hoc breaks.
Snippet — PodChaos sketch
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: kill-worker
spec:
action: pod-kill
mode: one
duration: "30s"
selector:
namespaces: ["orders"]
labelSelectors:
app: worker
Success criteria
- Three CRDs ran
- Catalog has ≥4 entries
- Hypothesis documented
If stuck
Chaos daemon needs privileged access — Kind on Docker Desktop may need docs notes.
Stretch goals
- Schedule CRD for a nightly NetworkChaos in staging-shaped namespace
Definition of Done
- On-call doc exists
- Postmortem written
- Chaos Mesh hypothesis loop demonstrated
Exercises — On-call, Postmortems & Chaos.