CompCode Academy

Lesson 2 of 4 · Free to read

Exercises

Exercises — Observability: logs, metrics, traces

Watch & learn

Course video

Open on YouTube
GitHub Actions Tutorial — CI/CD Pipeline — Pipelines that gate AI-generated changes.

Course video

Open on YouTube
System Design Basics: Horizontal vs Vertical Scaling — Architecture trade-offs before you pick a stack.

Keep a prompt log while you work. Speed without a trail is not a skill — it is luck.

How to use this file


Exercise 1 — Spin up the stack

Goal: get the Compose stack running and verify all three pillars produce data. Time: 20 minutes Level: foundational Modules touched: [14-observability]

Setup

  • cd code-examples/python/observability_stack.

Steps

  1. docker compose up -d.
  2. Wait ~30 s. Then curl http://127.0.0.1:8000/health a few times.
  3. Open http://127.0.0.1:3000 (Grafana, admin/admin).
  4. Confirm: a "service health" dashboard shows non-zero traffic; Loki shows recent log lines; Tempo shows a trace for /health.

Success criteria

  • All three pillars visible in Grafana for the lab service.

AI prompt hints

  • "Ask Cursor: 'Diagnose why localhost:3000 returns connection refused after docker compose up. List 3 likely causes.'"

Solution sketch

Common issue: port already in use. docker compose down and try alternative ports.

Stretch

  • Change Grafana's port to 3001 in docker-compose.yml. Re-up. Verify.

Exercise 2 — Add a custom metric

Goal: add a counter tasks_processed_total{result=...} to the service; verify it appears in Prometheus. Time: 30 minutes Level: intermediate Modules touched: [14-observability]

Setup

  • The lab service has a /work endpoint that simulates work and randomly fails.

Steps

  1. In app/work.py, increment the counter inside the handler on each success / failure.
  2. Restart: docker compose restart app.
  3. Generate load: for i in $(seq 1 100); do curl -s http://127.0.0.1:8000/work > /dev/null; done.
  4. In Grafana → Explore → Prometheus: query rate(tasks_processed_total{result="failure"}[1m]).

Success criteria

  • Query returns non-zero on the failure series.
  • A graph panel shows success vs failure rates over time.

AI prompt hints

  • "Ask Cursor: 'Add a Counter to this FastAPI handler that records success/failure outcomes. Use the OTel Python SDK. Output a diff only.'"

Solution sketch

The trap: high-cardinality labels (e.g., adding user_id). Keep it to bounded label values.

Stretch

  • Add a histogram for /work latency. Plot p50, p95, p99.

Exercise 3 — Define an SLI / SLO / error budget

Goal: write a one-page SLO document for the /work endpoint and a Grafana alert when the burn rate exceeds 2x. Time: 30 minutes Level: intermediate Modules touched: [14-observability]

Setup

  • Create learner-artifacts/<your-name>/slo.md.

Steps

  1. Pick the SLI: e.g., "fraction of /work requests that succeed".
  2. Pick the SLO: e.g., "99% over 30 days rolling".
  3. Compute the error budget: e.g., "1% = 432 minutes / month".
  4. Write the alert rule (Grafana or Prometheus alertmanager). Target: page if the burn rate over the last hour would exhaust the budget in <14 days.
  5. Test by deliberately spiking failures with ?force_fail=true (the lab supports this).

Success criteria

  • slo.md covers SLI / SLO / budget / alert / how-to-respond.
  • Alert fires when you spike failures; clears when failures stop.

AI prompt hints

  • "Ask Cursor: 'Write a Prometheus alert rule that fires when the 1-hour burn rate would exhaust a 99% SLO within 14 days. Use the math from Google's SRE workbook.'"

Solution sketch

The discipline is anchoring SLO in user observation, not in arbitrary "5 nines". Strong learners pick a less-than-100% SLO and defend it.

Stretch

  • Add a second alert at burn rate consistent with "exhausted in <3 days" — for paging vs ticketing.

Exercise 4 — Incident triage from data (the lab)

Goal: triage the synthetic incident using only logs / metrics / traces. No source code. Time: 50 minutes Level: intermediate Modules touched: [14-observability, 02-ai-assisted-development]

Setup

  • Trigger the incident: curl -X POST http://127.0.0.1:8000/admin/trigger-incident (the script silently changes behaviour for ~10 minutes).
  • Open INCIDENT.md for the alert text.

Steps

  1. Read the alert. Note the time.
  2. Open Grafana. Use the dashboards. Use Explore for ad-hoc queries.
  3. For each observation, log in TRIAGE.md: T+N, pillar used, query / panel, conclusion.
  4. Hypothesise a root cause. Propose a mitigation.
  5. After 50 minutes, open the source. Compare to your hypothesis. Reflect.

Success criteria

  • TRIAGE.md has ≥5 timestamped entries.
  • Hypothesis matches or near-matches the actual cause (acceptable: "right service, near-right root cause").
  • Reflection (3 sentences) on what AI would have helped with.

AI prompt hints

  • "Ask Cursor: 'Given these p95 latency, error rate, and trace samples, list the 3 most likely root causes. Rank by likelihood.'"
  • (The AI helps frame the hypothesis space; you fetch the evidence.)

Solution sketch

The planted incident is a downstream dependency timeout (the /work endpoint calls a flaky simulated DB). Strong learners triangulate: error rate up + latency p95 up + traces showing time in the DB span = "downstream slow / failing".

Stretch

  • Re-trigger; this time also use AI live during triage. Compare time-to-hypothesis vs solo run.

Exercise 5 — Write a postmortem (advanced)

Goal: write a one-page postmortem of the Exercise 4 incident using the Google SRE postmortem template. Time: 45 minutes Level: advanced Modules touched: [14-observability, 16-pm-and-agile]

Setup

  • Use your TRIAGE.md from Exercise 4.

Steps

  1. Open learner-artifacts/<your-name>/postmortem.md.
  2. Fill: summary, impact, timeline, root cause, what went well, what went badly, action items.
  3. Tag every action item with an owner (you, if solo) and a deadline.
  4. Optional: ask AI to critique the postmortem as a senior SRE.

Success criteria

  • All 7 sections present.
  • Timeline references the timestamps from TRIAGE.md.
  • Action items have owners + deadlines.
  • "What went badly" includes at least one process issue (e.g., alert was too noisy).

AI prompt hints

  • "Ask Cursor: 'Critique this postmortem from a senior SRE perspective. List the 3 weakest sections.'"
  • "Ask Cursor: 'Suggest 3 process action items based on what went badly.'"

Solution sketch

The discipline is separating root cause (technical) from contributing factors (process). Weak postmortems only fix the technical; strong ones fix both.

Stretch

  • Read 3 public postmortems (resources.md). Compare structure. Adopt one practice you like.

New free lessons and course updates. Unsubscribe from any email.