We spent a week hunting a checkout outage the tracing dashboards could not explain. The gateway started a span, then the trace went dark: payment logged an empty trace ID, and every downstream service minted its own root span. The p99 of the system looked fine; the p99 of the request was buried under three disconnected span trees. The root cause was not in a hot path. It was propagation: one service read the wrong header, one service spoke B3 while the rest spoke W3C traceparent, and one hop rebuilt its outgoing request from scratch and dropped every header.

Propagation is the plumbing that makes distributed tracing work, and in Go that plumbing is context.Context. It is the only value that follows a goroutine, passes through http.Handler boundaries, and survives pool reuse and retries. The trap is that context is also where developers shove everything from DB rows to user objects, which turns the tracing value into a mutable global and a memory leak. Our rule is boring and absolute: contexts carry telemetry, deadlines, and cancellation — nothing else.

Let the propagator do the header dance

The fastest way to break tracing is to hand-roll header extraction. You will forget traceparent was renamed from the old X-B3-TraceId scheme, you will mix header names across services, and you will add a custom propagation format because “it’s only for internal calls” — and then a partner system connects and you own a second format forever. The OpenTelemetry propagation.TraceContext propagator implements the W3C traceparent/tracestate spec and is the only thing we use at the edge:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package httpmw

import (
	"log/slog"
	"net/http"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/trace"
)

var propagator = otel.GetTextMapPropagator()

func Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
		ctx, span := otel.Tracer("edge").Start(ctx, "ingest")
		defer span.End()

		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

Extraction is what restores the parent link; Start only works correctly when given the extracted context. Get that order wrong and every service becomes a root span again — our exact outage. The same pattern applies on the way out: propagator.Inject(ctx, propagation.HeaderCarrier(req.Header)). One propagator, both directions, every service. We set OTEL_PROPAGATORS=tracecontext at the platform level so nothing in the application layer can deviate.

The gRPC side is the same idea, no headers

gRPC does not expose headers to your handlers, so propagating by hand means intercepting client and server streams and stashing metadata into context yourself. That is what otelgrpc already does: two interceptors, and the metadata-carrier wiring is handled for you. We wrap every client with otelgrpc.UnaryClientInterceptor() and every server with otelgrpc.UnaryServerInterceptor(). The subtle part is metadata-only cancellation: when a client cancels, the server gets the grpc-status trailer, but the span must be ended by the interceptor, not by your handler — otherwise you leak spans on every canceled request.

Make logs inherit the trace ID

Correlating logs to traces is what turns a tracing story into an on-call story. Go 1.21’s slog gave us the hook: wrap the handler so WithContext reads the trace and span IDs out of the context and attaches them to every record.

1
2
3
4
5
6
7
8
9
10
11
12
13
type traceHandler struct{ slog.Handler }

func (h traceHandler) WithContext(ctx context.Context) slog.Handler {
	sc := trace.SpanContextFromContext(ctx)
	if !sc.IsValid() {
		return h.Handler
	}
	attrs := []slog.Attr{
		slog.String("trace_id", sc.TraceID().String()),
		slog.String("span_id", sc.SpanID().String()),
	}
	return h.Handler.WithAttrs(attrs)
}

We pay one SpanContextFromContext lookup per log line — sub-nanosecond in practice and far cheaper than any JSON serialization — and the payoff is that a single trace_id=... grep in Loki replaces twenty pagerduty thread replies.

Production lessons

• Use an unexported key type. context.WithValue with a string key invites collision across packages. Define type key struct{} in one internal package and expose WithTrace(ctx, t)/TraceFrom(ctx) accessors; the compiler then enforces that only your package can read the value.
• Propagate deadlines and cancellation, not just trace IDs. A trace without a deadline is a story about a request that never ended. Copy ctx deadlines across the outbound call and let context.DeadlineExceeded be the signal that fails fast instead of retrying a payment that already committed.
• Never stash business state. Session user, request body, DB rows: all of it will be read somewhere you cannot audit and will hold the entire object graph alive for the duration of the request.
• Sampling is an edge decision. Decide keep/drop once, at the ingress service, and propagate the decision via traceparent flags. Sampling independently at every service is how you get 40% of spans with no parents and no way to query a whole trace.
• Watch the third-party boundary. Your client to a database or cache is a propagation boundary too. If the driver does not accept a context, that hop is blind and that latency is invisible. We instrument every client or we do not call it from a traced path.

Propagation is unglamorous, and it is the difference between a tracing system that answers “which service is slow?” in five seconds and one that answers “all of them, probably.” Get the headers, deadlines, and log correlation right once, at the edge, and every service you add for the next two years inherits it for free.