Noisy Service (Internal Spans)

Problem

A service is adding a large number of internal-kind spans to its traces: cache checks, helper functions, tight-loop iterations. Each one seemed worth tracing when it was written, but together they turn every trace into a wall of detail where the actual request flow is hard to see.

Insights flags a service once a single trace carries more than 10 internal spans from it.

Impact

• The critical path, the calls that explain latency, is buried in noise.
• Every trivial span costs ingest and storage like a meaningful one.
• Reading a trace during an incident takes minutes instead of seconds.

Example

Noisy:

HTTP GET /checkout (98 spans)
  ├─ cache.check
  ├─ cache.check
  ├─ feature_flag.read
  ├─ cache.check
  ├─ ... 92 more internal spans
  └─ DB query orders

Readable:

HTTP GET /checkout (4 spans)
  ├─ DB query orders
  ├─ HTTP POST payment-service
  └─ DB insert confirmations

How to fix

1. Keep spans for operations that cross a boundary (network calls, database queries, queue operations) or that meaningfully explain latency.
2. For hot in-process operations (cache hits, flag checks), use metrics: a counter and a histogram carry the aggregate signal at a fraction of the cost.
3. Check auto-instrumentation settings; some libraries trace internals aggressively by default and can be scoped down.

Expected gains

Traces show the request’s actual shape at a glance, and trace volume drops without losing anything you’d miss.

See the Insights catalog for everything Insights can detect.