CompCode Academy

Lesson 2 of 4 · Free to read

Exercises

Exercises — Metrics & Logs: PromQL, Alerting, Loki

Watch & learn

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Course video

Open on YouTube
Grafana Loki: Centralized Logging Without the Cost — Label-indexed logs — the Loki mental model for cheap, queryable logs.

Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.

How to use this file

  • Work Lab A → B → C on the same orders-lab repo
  • Prefer Docker Desktop; use Kind when the lab says so
  • Definition of Done at the bottom is the phase gate

Lab A — PromQL basics on /orders

Time: 45–60 min
Goal: Graph traffic, errors, and p95 latency for the order API.

Setup

Phase 01 stack with metrics exposed.

Steps

  1. Scrape api /metrics and worker Prometheus exporter.
  2. In Explore, run rate queries for request counters and histogram_quantile for latency.
  3. Save a Grafana dashboard panel for RED metrics.

Snippet — p95 latency

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)

Success criteria

  • Dashboard shows live RED panels under load

If stuck

If histogram_quantile is empty, confirm _bucket series exist and le labels are present.


Lab B — Alerts + Alertmanager

Time: 60–90 min
Goal: Page on symptom burn, not CPU.

Setup

Prometheus + Alertmanager in compose.

Steps

  1. Add a recording rule job:http_error_ratio:ratio5m.
  2. Add an alert when that series exceeds a threshold for 5m.
  3. Configure Alertmanager grouping by service and a null receiver for local demos.
  4. Force errors and watch the alert fire then resolve.

Success criteria

  • Alert transitions firing → resolved
  • Grouping prevents duplicate pages

If stuck

Check prometheus.yml rule_files path and Alertmanager --config.file.


Lab C — LogQL triage

Time: 45–60 min
Goal: From a bad request, find logs and jump to a trace.

Setup

Structured JSON logs with trace_id field.

Steps

  1. Emit JSON logs including trace_id and order_id.
  2. Query Loki for error lines in the last 15m.
  3. Configure Grafana derivedFields from Loki → Tempo.
  4. Write a short incident note: alert → series → log → trace.

Success criteria

  • Clickable jump log → Tempo
  • Incident note committed

If stuck

Derived field regex must match your JSON key exactly.


Stretch goals

  • Remote-write a subset of metrics to Mimir and query via Grafana Mimir datasource

Definition of Done

  • RED dashboard exists
  • At least one symptom-based alert works
  • Log→trace jump demonstrated

Exercises — Metrics & Logs: PromQL, Alerting, Loki.

New free lessons and course updates. Unsubscribe from any email.