Lesson 2 of 4 · Free to read
Exercises
Exercises — Observability: logs, metrics, traces
Watch & learn
Course video
Course video
Keep a prompt log while you work. Speed without a trail is not a skill — it is luck.
How to use this file
- All exercises use
code-examples/python/observability_stack/unless noted. - AI prompts logged in
PROMPTS.md. - The lab in the lesson plan is Exercise 4; the others scaffold up to it.
Exercise 1 — Spin up the stack
Goal: get the Compose stack running and verify all three pillars produce data. Time: 20 minutes Level: foundational Modules touched: [14-observability]
Setup
cd code-examples/python/observability_stack.
Steps
docker compose up -d.- Wait ~30 s. Then
curl http://127.0.0.1:8000/healtha few times. - Open
http://127.0.0.1:3000(Grafana, admin/admin). - Confirm: a "service health" dashboard shows non-zero traffic; Loki shows recent log lines; Tempo shows a trace for
/health.
Success criteria
- All three pillars visible in Grafana for the lab service.
AI prompt hints
- "Ask Cursor: 'Diagnose why localhost:3000 returns connection refused after docker compose up. List 3 likely causes.'"
Solution sketch
Common issue: port already in use. docker compose down and try alternative ports.
Stretch
- Change Grafana's port to 3001 in
docker-compose.yml. Re-up. Verify.
Exercise 2 — Add a custom metric
Goal: add a counter tasks_processed_total{result=...} to the service; verify it appears in Prometheus.
Time: 30 minutes
Level: intermediate
Modules touched: [14-observability]
Setup
- The lab service has a
/workendpoint that simulates work and randomly fails.
Steps
- In
app/work.py, increment the counter inside the handler on each success / failure. - Restart:
docker compose restart app. - Generate load:
for i in $(seq 1 100); do curl -s http://127.0.0.1:8000/work > /dev/null; done. - In Grafana → Explore → Prometheus: query
rate(tasks_processed_total{result="failure"}[1m]).
Success criteria
- Query returns non-zero on the failure series.
- A graph panel shows success vs failure rates over time.
AI prompt hints
- "Ask Cursor: 'Add a Counter to this FastAPI handler that records success/failure outcomes. Use the OTel Python SDK. Output a diff only.'"
Solution sketch
The trap: high-cardinality labels (e.g., adding user_id). Keep it to bounded label values.
Stretch
- Add a histogram for
/worklatency. Plot p50, p95, p99.
Exercise 3 — Define an SLI / SLO / error budget
Goal: write a one-page SLO document for the /work endpoint and a Grafana alert when the burn rate exceeds 2x.
Time: 30 minutes
Level: intermediate
Modules touched: [14-observability]
Setup
- Create
learner-artifacts/<your-name>/slo.md.
Steps
- Pick the SLI: e.g., "fraction of
/workrequests that succeed". - Pick the SLO: e.g., "99% over 30 days rolling".
- Compute the error budget: e.g., "1% = 432 minutes / month".
- Write the alert rule (Grafana or Prometheus alertmanager). Target: page if the burn rate over the last hour would exhaust the budget in <14 days.
- Test by deliberately spiking failures with
?force_fail=true(the lab supports this).
Success criteria
slo.mdcovers SLI / SLO / budget / alert / how-to-respond.- Alert fires when you spike failures; clears when failures stop.
AI prompt hints
- "Ask Cursor: 'Write a Prometheus alert rule that fires when the 1-hour burn rate would exhaust a 99% SLO within 14 days. Use the math from Google's SRE workbook.'"
Solution sketch
The discipline is anchoring SLO in user observation, not in arbitrary "5 nines". Strong learners pick a less-than-100% SLO and defend it.
Stretch
- Add a second alert at burn rate consistent with "exhausted in <3 days" — for paging vs ticketing.
Exercise 4 — Incident triage from data (the lab)
Goal: triage the synthetic incident using only logs / metrics / traces. No source code. Time: 50 minutes Level: intermediate Modules touched: [14-observability, 02-ai-assisted-development]
Setup
- Trigger the incident:
curl -X POST http://127.0.0.1:8000/admin/trigger-incident(the script silently changes behaviour for ~10 minutes). - Open
INCIDENT.mdfor the alert text.
Steps
- Read the alert. Note the time.
- Open Grafana. Use the dashboards. Use Explore for ad-hoc queries.
- For each observation, log in
TRIAGE.md: T+N, pillar used, query / panel, conclusion. - Hypothesise a root cause. Propose a mitigation.
- After 50 minutes, open the source. Compare to your hypothesis. Reflect.
Success criteria
TRIAGE.mdhas ≥5 timestamped entries.- Hypothesis matches or near-matches the actual cause (acceptable: "right service, near-right root cause").
- Reflection (3 sentences) on what AI would have helped with.
AI prompt hints
- "Ask Cursor: 'Given these p95 latency, error rate, and trace samples, list the 3 most likely root causes. Rank by likelihood.'"
- (The AI helps frame the hypothesis space; you fetch the evidence.)
Solution sketch
The planted incident is a downstream dependency timeout (the /work endpoint calls a flaky simulated DB). Strong learners triangulate: error rate up + latency p95 up + traces showing time in the DB span = "downstream slow / failing".
Stretch
- Re-trigger; this time also use AI live during triage. Compare time-to-hypothesis vs solo run.
Exercise 5 — Write a postmortem (advanced)
Goal: write a one-page postmortem of the Exercise 4 incident using the Google SRE postmortem template. Time: 45 minutes Level: advanced Modules touched: [14-observability, 16-pm-and-agile]
Setup
- Use your
TRIAGE.mdfrom Exercise 4.
Steps
- Open
learner-artifacts/<your-name>/postmortem.md. - Fill: summary, impact, timeline, root cause, what went well, what went badly, action items.
- Tag every action item with an owner (you, if solo) and a deadline.
- Optional: ask AI to critique the postmortem as a senior SRE.
Success criteria
- All 7 sections present.
- Timeline references the timestamps from
TRIAGE.md. - Action items have owners + deadlines.
- "What went badly" includes at least one process issue (e.g., alert was too noisy).
AI prompt hints
- "Ask Cursor: 'Critique this postmortem from a senior SRE perspective. List the 3 weakest sections.'"
- "Ask Cursor: 'Suggest 3 process action items based on what went badly.'"
Solution sketch
The discipline is separating root cause (technical) from contributing factors (process). Weak postmortems only fix the technical; strong ones fix both.
Stretch
- Read 3 public postmortems (resources.md). Compare structure. Adopt one practice you like.