CompCode Academy

Lesson 2 of 4 · Free to read

Exercises

Exercises — Observability Foundations: LGTM + LocalStack Order Pipeline

Watch & learn

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Course video

Open on YouTube
OpenTelemetry Deep Dive (Big Tent) — Standards, tracing, and why OTel is the 2026 vendor-neutral data plane.

Keep a lab notebook: hypothesis → command → result. Speed without evidence is not a skill.

How to use this file

  • Work Lab A → B → C on the same orders-lab repo
  • Prefer Docker Desktop; use Kind when the lab says so
  • Definition of Done at the bottom is the phase gate

Lab A — Bring up the stack

Time: 45–60 min
Goal: All observability containers healthy; Grafana datasources green.

Setup

Docker Desktop running; empty orders-lab/ folder.

Steps

  1. Create directories: api/, worker/, otel/, alloy/, grafana/provisioning/datasources/.
  2. Add docker-compose services for Grafana (:3000), Prometheus (:9090), Loki (:3100), Tempo (OTLP 4317/4318), Mimir, Pyroscope (:4040), OTel Collector, Alloy, LocalStack (:4566).
  3. docker compose up -d and wait until healthchecks pass.
  4. Open Grafana and confirm Prometheus, Loki, Tempo, Mimir, Pyroscope datasources connect.

Success criteria

  • docker compose ps shows healthy LocalStack + backends
  • Grafana lists 5 datasources as OK

If stuck

If Tempo datasource is red, check OTLP ports and that Collector exports to tempo:4317.


Lab B — Wire the order pipeline

Time: 60–90 min
Goal: POST /orders returns 202 and writes orders/<id>.json to S3 via SNS→SQS→worker.

Setup

Stack from Lab A up.

Steps

  1. Implement api Express route that validates an order and publishes to SNS (AWS SDK v3, LocalStack endpoint).
  2. Implement worker that long-polls SQS, processes the envelope, PutObject to S3.
  3. Run init script (awslocal) to create topic, queue+subscription, bucket.
  4. POST a sample order; verify S3 listing and worker logs.

Snippet — Smoke the pipeline

curl -s -X POST http://localhost:8080/orders \
  -H 'content-type: application/json' \
  -d '{"sku":"SKU-1","qty":1,"region":"IN"}'
awslocal s3 ls s3://order-results/orders/

Success criteria

  • HTTP 202 from api
  • awslocal s3 ls s3://order-results/orders/ shows an object

If stuck

SNS-to-SQS delivers an envelope — parse Message JSON inside the body.


Lab C — First correlated view + cardinality trap

Time: 45–60 min
Goal: See one trace across api→worker; prove why order.id must not be a metric label.

Setup

Instrumented api/worker emitting OTLP to Collector.

Steps

  1. Enable auto-instrumentation for Express + AWS SDK; export OTLP to Collector.
  2. In Grafana Explore → Tempo, find a trace that spans SNS/SQS.
  3. Temporarily add order_id as a Prometheus label, send 200 orders, watch series explode — then revert.
  4. Write a 5-line golden-signals checklist for /orders (latency, errors, traffic, saturation).

Success criteria

  • One Tempo trace covers async hop
  • Cardinality experiment documented and reverted
  • Golden-signals checklist saved in repo

If stuck

If trace breaks at SQS, ensure MessageAttributeNames includes All and propagator inject/extract runs.


Stretch goals

  • docker compose up -d --scale worker=3 and confirm exactly one S3 object per order
  • Stop Tempo and note the app still works — observability down ≠ app down

Definition of Done

  • You can say out loud what each of the four signals answers
  • Order pipeline works end-to-end on LocalStack
  • Grafana datasources are connected

Exercises — Observability Foundations: LGTM + LocalStack Order Pipeline.

New free lessons and course updates. Unsubscribe from any email.