Lesson 1 of 4 · Free to read
Session lesson
Distributed Tracing & TraceQL
Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-02-metrics-and-logs
Why this matters (5 minutes)
Metrics say p95 is high. Traces show which span — SNS publish vs S3 put — ate the time. TraceQL lets you ask about the shape of the request tree, not just attributes.
Learning outcomes
- Instrument SpanKinds correctly across api and worker
- Propagate context across SNS→SQS message attributes
- Choose a sampling strategy and explain what TraceQL cannot see
- Write TraceQL attribute and structural queries
Shared lab constants
| System | api (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07) |
| Stack | Grafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy |
| Rule | Extend the same repo every phase — never start a new toy app. |
What MNCs do in 2026
OpenTelemetry semantic conventions + W3C tracecontext across async messaging is table stakes. Tempo + TraceQL replace “SSH and guess” for latency RCAs.
Teachable chunks (compressed from days 21–30)
A trace is a tree
Parent/child/descendant relationships answer “was this error inside a user request or a background retry?”
Async is where traces break
HTTP headers don’t exist on SNS→SQS — inject/extract via message attributes (or OTel AWS instrumentation).
Sampling is upstream of queries
If the sampler dropped a trace, TraceQL returns empty — that is not “no bug.”
TraceQL vs PromQL/LogQL
Same { } filter feel, plus > / >> structural operators.
Mental model
Watch & learn
Course video
Course video
Worked example
Find slow S3 writes: { resource.service.name = "worker" && span.aws.s3.bucket = "order-results" && duration > 300ms }. Confirm the matching span highlights inside the full waterfall.
Hands-on path
Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.
Phase checklist
- Async hop stays in one trace
- You can explain sampling vs “no data”
- At least one structural TraceQL query works
Key takeaways
- Keep the same order pipeline — depth beats new toy apps
- Correlate signals before you SSH
- Write policies and catalogs your future on-call self can use
Credit
Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.
Observability phase lesson — Distributed Tracing & TraceQL.