Lesson 1 of 4 · Free to read
Session lesson
Metrics & Logs: PromQL, Alerting, Loki
Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-01-foundations-lgtm
Why this matters (5 minutes)
A spike hits /orders. Without PromQL and structured logs you guess. With recording rules + LogQL you jump from alert → series → log line in one pane.
Learning outcomes
- Write PromQL for rate, histogram_quantile, and recording rules
- Ship Alertmanager routes with grouping and inhibition
- Query Loki with LogQL tied to trace_id
- Remote-write mental model for Mimir long-term storage
Shared lab constants
| System | api (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07) |
| Stack | Grafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy |
| Rule | Extend the same repo every phase — never start a new toy app. |
What MNCs do in 2026
Platform teams standardize Prometheus recording/alerting rules as code, Alertmanager routing/inhibition, and Loki label budgets — not full-text ELK for every line.
Teachable chunks (compressed from days 11–20)
PromQL that production uses
rate, irate, histogram_quantile, recording rules for expensive queries, alert on symptoms not causes.
Alertmanager is a product
Grouping, inhibition, and silences prevent page storms — same pattern MNCs use with PagerDuty/Opsgenie.
Loki indexes labels, not every token
Keep label cardinality tiny; put high-cardinality fields in the line or structured metadata.
Logs capstone triage
Given only telemetry, reconstruct an incident timeline without SSH.
Mental model
Watch & learn
Course video
Course video
Worked example
Add http_request_duration_seconds histogram on api. Create a recording rule for 5m error ratio and an alert when ratio > 1%. In Loki, query {service="api"} |= "order" and link to Tempo via derived fields.
Hands-on path
Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.
Phase checklist
- RED dashboard exists
- At least one symptom-based alert works
- Log→trace jump demonstrated
Key takeaways
- Keep the same order pipeline — depth beats new toy apps
- Correlate signals before you SSH
- Write policies and catalogs your future on-call self can use
Credit
Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.
Observability phase lesson — Metrics & Logs: PromQL, Alerting, Loki.