Lesson 2 of 4 · Free to read
Exercises
Exercises — Metrics & Logs: PromQL, Alerting, Loki
Watch & learn
Course video
Course video
Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.
How to use this file
- Work Lab A → B → C on the same
orders-labrepo - Prefer Docker Desktop; use Kind when the lab says so
- Definition of Done at the bottom is the phase gate
Lab A — PromQL basics on /orders
Time: 45–60 min
Goal: Graph traffic, errors, and p95 latency for the order API.
Setup
Phase 01 stack with metrics exposed.
Steps
- Scrape api
/metricsand worker Prometheus exporter. - In Explore, run rate queries for request counters and histogram_quantile for latency.
- Save a Grafana dashboard panel for RED metrics.
Snippet — p95 latency
histogram_quantile(0.95,
sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)
Success criteria
- Dashboard shows live RED panels under load
If stuck
If histogram_quantile is empty, confirm _bucket series exist and le labels are present.
Lab B — Alerts + Alertmanager
Time: 60–90 min
Goal: Page on symptom burn, not CPU.
Setup
Prometheus + Alertmanager in compose.
Steps
- Add a recording rule
job:http_error_ratio:ratio5m. - Add an alert when that series exceeds a threshold for 5m.
- Configure Alertmanager grouping by
serviceand a null receiver for local demos. - Force errors and watch the alert fire then resolve.
Success criteria
- Alert transitions firing → resolved
- Grouping prevents duplicate pages
If stuck
Check prometheus.yml rule_files path and Alertmanager --config.file.
Lab C — LogQL triage
Time: 45–60 min
Goal: From a bad request, find logs and jump to a trace.
Setup
Structured JSON logs with trace_id field.
Steps
- Emit JSON logs including
trace_idandorder_id. - Query Loki for error lines in the last 15m.
- Configure Grafana derivedFields from Loki → Tempo.
- Write a short incident note: alert → series → log → trace.
Success criteria
- Clickable jump log → Tempo
- Incident note committed
If stuck
Derived field regex must match your JSON key exactly.
Stretch goals
- Remote-write a subset of metrics to Mimir and query via Grafana Mimir datasource
Definition of Done
- RED dashboard exists
- At least one symptom-based alert works
- Log→trace jump demonstrated
Exercises — Metrics & Logs: PromQL, Alerting, Loki.