Lesson 2 of 4 · Free to read
Exercises
Exercises — Observability Foundations: LGTM + LocalStack Order Pipeline
Watch & learn
Course video
Course video
Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.
How to use this file
- Work Lab A → B → C on the same
orders-labrepo - Prefer Docker Desktop; use Kind when the lab says so
- Definition of Done at the bottom is the phase gate
Lab A — Bring up the stack
Time: 45–60 min
Goal: All observability containers healthy; Grafana datasources green.
Setup
Docker Desktop running; empty orders-lab/ folder.
Steps
- Create directories:
api/,worker/,otel/,alloy/,grafana/provisioning/datasources/. - Add docker-compose services for Grafana (:3000), Prometheus (:9090), Loki (:3100), Tempo (OTLP 4317/4318), Mimir, Pyroscope (:4040), OTel Collector, Alloy, LocalStack (:4566).
docker compose up -dand wait until healthchecks pass.- Open Grafana and confirm Prometheus, Loki, Tempo, Mimir, Pyroscope datasources connect.
Success criteria
-
docker compose psshows healthy LocalStack + backends - Grafana lists 5 datasources as OK
If stuck
If Tempo datasource is red, check OTLP ports and that Collector exports to tempo:4317.
Lab B — Wire the order pipeline
Time: 60–90 min
Goal: POST /orders returns 202 and writes orders/<id>.json to S3 via SNS→SQS→worker.
Setup
Stack from Lab A up.
Steps
- Implement
apiExpress route that validates an order and publishes to SNS (AWS SDK v3, LocalStack endpoint). - Implement
workerthat long-polls SQS, processes the envelope, PutObject to S3. - Run init script (
awslocal) to create topic, queue+subscription, bucket. - POST a sample order; verify S3 listing and worker logs.
Snippet — Smoke the pipeline
curl -s -X POST http://localhost:8080/orders \
-H 'content-type: application/json' \
-d '{"sku":"SKU-1","qty":1,"region":"IN"}'
awslocal s3 ls s3://order-results/orders/
Success criteria
- HTTP 202 from api
-
awslocal s3 ls s3://order-results/orders/shows an object
If stuck
SNS-to-SQS delivers an envelope — parse Message JSON inside the body.
Lab C — First correlated view + cardinality trap
Time: 45–60 min
Goal: See one trace across api→worker; prove why order.id must not be a metric label.
Setup
Instrumented api/worker emitting OTLP to Collector.
Steps
- Enable auto-instrumentation for Express + AWS SDK; export OTLP to Collector.
- In Grafana Explore → Tempo, find a trace that spans SNS/SQS.
- Temporarily add
order_idas a Prometheus label, send 200 orders, watch series explode — then revert. - Write a 5-line golden-signals checklist for
/orders(latency, errors, traffic, saturation).
Success criteria
- One Tempo trace covers async hop
- Cardinality experiment documented and reverted
- Golden-signals checklist saved in repo
If stuck
If trace breaks at SQS, ensure MessageAttributeNames includes All and propagator inject/extract runs.
Stretch goals
docker compose up -d --scale worker=3and confirm exactly one S3 object per order- Stop Tempo and note the app still works — observability down ≠ app down
Definition of Done
- You can say out loud what each of the four signals answers
- Order pipeline works end-to-end on LocalStack
- Grafana datasources are connected
Exercises — Observability Foundations: LGTM + LocalStack Order Pipeline.