CompCode Academy

Lesson 2 of 4 · Free to read

Exercises

Exercises — On-call, Postmortems & Chaos

Watch & learn

Course video

Open on YouTube
eBPF for Observability (CNCF) — What eBPF can and cannot see — zero-code signals vs SDK instrumentation.

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.

How to use this file

  • Work Lab A → B → C on the same orders-lab repo
  • Prefer Docker Desktop; use Kind when the lab says so
  • Definition of Done at the bottom is the phase gate

Lab A — On-call design

Time: 45 min
Goal: Escalation doc + alert review.

Setup

Existing alerts from earlier phases.

Steps

  1. Draft primary/secondary escalation and handoff checklist.
  2. Delete or silence one noisy alert with justification.
  3. Define MTTA/MTTR tracking fields for your team.

Success criteria

  • oncall.md committed
  • Noise reduction note written

If stuck

If you lack a real pager, simulate with a Slack webhook or email sink.


Lab B — Game day + postmortem

Time: 60–90 min
Goal: Controlled incident with blameless write-up.

Setup

Dashboards + known fault script.

Steps

  1. Pick a fault (kill worker, block S3, raise latency).
  2. Run game day with timekeeper; note detections and actions.
  3. Write postmortem with action items.

Success criteria

  • Postmortem merged
  • At least two action items assigned

If stuck

Keep blast radius to local Kind/compose — never production.


Lab C — Chaos Mesh

Time: 60–90 min
Goal: Declarative faults with SLO hypothesis.

Setup

Kind cluster; Helm.

Steps

  1. Helm install Chaos Mesh; verify CRDs.
  2. Write steady-state hypothesis using phase-07 series.
  3. Apply PodChaos, NetworkChaos, StressChaos once each.
  4. Create chaos-experiment-catalog.md reframing past ad-hoc breaks.

Snippet — PodChaos sketch

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-worker
spec:
  action: pod-kill
  mode: one
  duration: "30s"
  selector:
    namespaces: ["orders"]
    labelSelectors:
      app: worker

Success criteria

  • Three CRDs ran
  • Catalog has ≥4 entries
  • Hypothesis documented

If stuck

Chaos daemon needs privileged access — Kind on Docker Desktop may need docs notes.


Stretch goals

  • Schedule CRD for a nightly NetworkChaos in staging-shaped namespace

Definition of Done

  • On-call doc exists
  • Postmortem written
  • Chaos Mesh hypothesis loop demonstrated

Exercises — On-call, Postmortems & Chaos.

New free lessons and course updates. Unsubscribe from any email.