CompCode Academy

Lesson 1 of 4 · Free to read

Session lesson

Observability Foundations: LGTM + LocalStack Order Pipeline

Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: Module 14-observability (pillars intro) recommended; Docker Desktop required.

Why this matters (5 minutes)

At 3am an order API returns 202 but nothing lands in S3. Fixed dashboards say “CPU fine.” Observability means you can ask a new question — which hop broke? — without shipping code.

Learning outcomes

  • Explain monitoring vs observability and the four signals (metrics, logs, traces, profiles)
  • Stand up an LGTM + Collector + Alloy stack on Docker Compose
  • Drive POST /orders through SNS→SQS→worker→S3 on LocalStack
  • Apply golden signals / RED / USE and avoid a cardinality trap

Shared lab constants

Systemapi (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07)
StackGrafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy
RuleExtend the same repo every phase — never start a new toy app.

What MNCs do in 2026

MNCs run vendor-neutral OTLP into Grafana LGTM (Loki, Grafana, Tempo, Mimir) plus Prometheus scrape and Pyroscope profiles. LocalStack lets you rehearse SNS→SQS→S3 without a cloud bill.

Teachable chunks (compressed from days 1–10)

Questions first, then instrumentation

Name user-facing SLIs, list incident questions, then derive the minimum telemetry. “Collect everything” burns budget and signal-to-noise.

Four signals, four jobs

Metrics: is it broken / how bad. Traces: where. Logs: why. Profiles: which code. Correlate by trace_id in Grafana.

Cardinality is a cost decision

Never put unbounded IDs (order_id, user_id) on metric labels — put them on span attributes or log fields.

One lab bench for every later phase

Build day01-style compose once. Every later phase extends this tree.

Mental model

Watch & learn

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Course video

Open on YouTube
OpenTelemetry Deep Dive (Big Tent) — Standards, tracing, and why OTel is the 2026 vendor-neutral data plane.

Worked example

Scaffold orders-lab/, add docker-compose for Grafana/Prometheus/Loki/Tempo/Mimir/Pyroscope/Collector/Alloy/LocalStack/api/worker. Provision SNS topic + SQS subscription + S3 bucket. POST an order and confirm S3 object + a Tempo trace spanning the async hop.

Hands-on path

Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.

Phase checklist

  • You can say out loud what each of the four signals answers
  • Order pipeline works end-to-end on LocalStack
  • Grafana datasources are connected

Key takeaways

  • Keep the same order pipeline — depth beats new toy apps
  • Correlate signals before you SSH
  • Write policies and catalogs your future on-call self can use

Credit

Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.


Observability phase lesson — Observability Foundations: LGTM + LocalStack Order Pipeline.

New free lessons and course updates. Unsubscribe from any email.