Lesson 1 of 4 · Free to read
Session lesson
Observability Foundations: LGTM + LocalStack Order Pipeline
Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: Module 14-observability (pillars intro) recommended; Docker Desktop required.
Why this matters (5 minutes)
At 3am an order API returns 202 but nothing lands in S3. Fixed dashboards say “CPU fine.” Observability means you can ask a new question — which hop broke? — without shipping code.
Learning outcomes
- Explain monitoring vs observability and the four signals (metrics, logs, traces, profiles)
- Stand up an LGTM + Collector + Alloy stack on Docker Compose
- Drive POST /orders through SNS→SQS→worker→S3 on LocalStack
- Apply golden signals / RED / USE and avoid a cardinality trap
Shared lab constants
| System | api (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07) |
| Stack | Grafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy |
| Rule | Extend the same repo every phase — never start a new toy app. |
What MNCs do in 2026
MNCs run vendor-neutral OTLP into Grafana LGTM (Loki, Grafana, Tempo, Mimir) plus Prometheus scrape and Pyroscope profiles. LocalStack lets you rehearse SNS→SQS→S3 without a cloud bill.
Teachable chunks (compressed from days 1–10)
Questions first, then instrumentation
Name user-facing SLIs, list incident questions, then derive the minimum telemetry. “Collect everything” burns budget and signal-to-noise.
Four signals, four jobs
Metrics: is it broken / how bad. Traces: where. Logs: why. Profiles: which code. Correlate by trace_id in Grafana.
Cardinality is a cost decision
Never put unbounded IDs (order_id, user_id) on metric labels — put them on span attributes or log fields.
One lab bench for every later phase
Build day01-style compose once. Every later phase extends this tree.
Mental model
Watch & learn
Course video
Course video
Worked example
Scaffold orders-lab/, add docker-compose for Grafana/Prometheus/Loki/Tempo/Mimir/Pyroscope/Collector/Alloy/LocalStack/api/worker. Provision SNS topic + SQS subscription + S3 bucket. POST an order and confirm S3 object + a Tempo trace spanning the async hop.
Hands-on path
Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.
Phase checklist
- You can say out loud what each of the four signals answers
- Order pipeline works end-to-end on LocalStack
- Grafana datasources are connected
Key takeaways
- Keep the same order pipeline — depth beats new toy apps
- Correlate signals before you SSH
- Write policies and catalogs your future on-call self can use
Credit
Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.
Observability phase lesson — Observability Foundations: LGTM + LocalStack Order Pipeline.