CompCode Academy

Lesson 1 of 4 · Free to read

Session lesson

Metrics & Logs: PromQL, Alerting, Loki

Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-01-foundations-lgtm

Why this matters (5 minutes)

A spike hits /orders. Without PromQL and structured logs you guess. With recording rules + LogQL you jump from alert → series → log line in one pane.

Learning outcomes

  • Write PromQL for rate, histogram_quantile, and recording rules
  • Ship Alertmanager routes with grouping and inhibition
  • Query Loki with LogQL tied to trace_id
  • Remote-write mental model for Mimir long-term storage

Shared lab constants

Systemapi (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07)
StackGrafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy
RuleExtend the same repo every phase — never start a new toy app.

What MNCs do in 2026

Platform teams standardize Prometheus recording/alerting rules as code, Alertmanager routing/inhibition, and Loki label budgets — not full-text ELK for every line.

Teachable chunks (compressed from days 11–20)

PromQL that production uses

rate, irate, histogram_quantile, recording rules for expensive queries, alert on symptoms not causes.

Alertmanager is a product

Grouping, inhibition, and silences prevent page storms — same pattern MNCs use with PagerDuty/Opsgenie.

Loki indexes labels, not every token

Keep label cardinality tiny; put high-cardinality fields in the line or structured metadata.

Logs capstone triage

Given only telemetry, reconstruct an incident timeline without SSH.

Mental model

Watch & learn

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Course video

Open on YouTube
Grafana Loki: Centralized Logging Without the Cost — Label-indexed logs — the Loki mental model for cheap, queryable logs.

Worked example

Add http_request_duration_seconds histogram on api. Create a recording rule for 5m error ratio and an alert when ratio > 1%. In Loki, query {service="api"} |= "order" and link to Tempo via derived fields.

Hands-on path

Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.

Phase checklist

  • RED dashboard exists
  • At least one symptom-based alert works
  • Log→trace jump demonstrated

Key takeaways

  • Keep the same order pipeline — depth beats new toy apps
  • Correlate signals before you SSH
  • Write policies and catalogs your future on-call self can use

Credit

Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.


Observability phase lesson — Metrics & Logs: PromQL, Alerting, Loki.

New free lessons and course updates. Unsubscribe from any email.