CompCode Academy

Lesson 1 of 4 · Free to read

Session lesson

Distributed Tracing & TraceQL

Self-paced phase · ~8–12 focused hours across Lab A/B/C
Audience: Final-year students & early-career engineers
Prerequisites: obs-02-metrics-and-logs

Why this matters (5 minutes)

Metrics say p95 is high. Traces show which span — SNS publish vs S3 put — ate the time. TraceQL lets you ask about the shape of the request tree, not just attributes.

Learning outcomes

  • Instrument SpanKinds correctly across api and worker
  • Propagate context across SNS→SQS message attributes
  • Choose a sampling strategy and explain what TraceQL cannot see
  • Write TraceQL attribute and structural queries

Shared lab constants

Systemapi (Express) → SNS → SQS → worker → S3 (+ Postgres/Redis from phase 07)
StackGrafana · Prometheus/Mimir · Loki · Tempo · Pyroscope · OTel Collector · Alloy
RuleExtend the same repo every phase — never start a new toy app.

What MNCs do in 2026

OpenTelemetry semantic conventions + W3C tracecontext across async messaging is table stakes. Tempo + TraceQL replace “SSH and guess” for latency RCAs.

Teachable chunks (compressed from days 21–30)

A trace is a tree

Parent/child/descendant relationships answer “was this error inside a user request or a background retry?”

Async is where traces break

HTTP headers don’t exist on SNS→SQS — inject/extract via message attributes (or OTel AWS instrumentation).

Sampling is upstream of queries

If the sampler dropped a trace, TraceQL returns empty — that is not “no bug.”

TraceQL vs PromQL/LogQL

Same { } filter feel, plus > / >> structural operators.

Mental model

Watch & learn

Course video

Open on YouTube
OpenTelemetry Deep Dive (Big Tent) — Standards, tracing, and why OTel is the 2026 vendor-neutral data plane.

Course video

Open on YouTube
How Prometheus Monitoring works — Prometheus scrape, TSDB, and alert architecture used in MNCs.

Worked example

Find slow S3 writes: { resource.service.name = "worker" && span.aws.s3.bucket = "order-results" && duration > 300ms }. Confirm the matching span highlights inside the full waterfall.

Hands-on path

Open Exercises next. Complete Lab A → B → C in order. Stretch is optional.

Phase checklist

  • Async hop stays in one trace
  • You can explain sampling vs “no data”
  • At least one structural TraceQL query works

Key takeaways

  • Keep the same order pipeline — depth beats new toy apps
  • Correlate signals before you SSH
  • Write policies and catalogs your future on-call self can use

Credit

Curriculum themes inspired by the public Observability100 course (https://observability100.vercel.app/course/index.html); lessons and labs rewritten for AI-Tutor.


Observability phase lesson — Distributed Tracing & TraceQL.

New free lessons and course updates. Unsubscribe from any email.