Skip to main content

Excessive Short-Duration Spans

Normal Signal: Traces

Problem

Traces from a service contain large numbers of spans that finish almost instantly: cache hits, in-memory lookups, per-item spans inside loops. An operation that finishes in under five milliseconds rarely needs its own span; at volume, recording it costs more than the operation itself.

Insights flags a trace once it carries more than 20 spans shorter than 5 milliseconds.

Impact

  • Tracing overhead grows in the hot path, where the spans say the least.
  • Trace volume and cost inflate with detail nobody drills into.
  • Meaningful spans share screen space with thousands of near-zero bars.

Example

Flooded:

HTTP GET /products (240 spans)
├─ cache.get (0.3ms) × 183
├─ flag.check (0.2ms) × 55
└─ DB query products (12ms)

Focused:

HTTP GET /products (3 spans)
├─ DB query products (12ms)
└─ HTTP GET pricing-service (8ms)
# cache.get and flag.check now emit metrics

How to fix

  1. Remove span creation from tight loops and per-item processing; wrap the batch, not each element.
  2. Replace spans on sub-millisecond operations with a counter and a latency histogram.
  3. Keep fast spans that can fail in interesting ways; errors are worth a span at any duration.

Expected gains

Lower tracing overhead in your hottest code, smaller bills, and traces where every span earns its place.

See the Insights catalog for everything Insights can detect.