Skip to main content
On this page

Log Storm

Critical Signal: Logs

Problem

A service is emitting the same log message over and over at an extreme rate, typically a tight error loop retrying a failed dependency without backoff. The message was useful the first time; the next hundred thousand copies only cost money and bury everything else.

Insights groups log records by service, fingerprints the message body, and counts duplicates in a five-minute sliding window, flagging a storm once one message’s rate tops 100 duplicates per second.

Impact

  • Ingest and storage costs spike while the storm lasts.
  • Real signals drown: the warning that explains the outage is buried under the message that reacts to it.
  • Log-based alerting turns into noise exactly when you need it most.

Example

Storming:

14:02:01.001 ERROR connection refused: payments-db
14:02:01.003 ERROR connection refused: payments-db
14:02:01.004 ERROR connection refused: payments-db
... 6,000 identical lines per minute

Calm:

14:02:01 ERROR connection refused: payments-db (retrying with backoff)
14:02:31 ERROR still unreachable: payments-db (attempt 2, next retry 60s)

How to fix

  1. Find the loop producing the message: the storm usually points straight at a dependency failure being retried in a hot loop.
  2. Add backoff (and ideally a circuit breaker) so retries slow down instead of logging at CPU speed.
  3. For messages that legitimately repeat, log a periodic summary with a count instead of every occurrence.

Expected gains

Log volume drops back to what the workload justifies, the message that matters becomes visible again, and the next incident isn’t a storage bill.

See the Insights catalog for everything Insights can detect.