Noise is a symptom, not the disease
Ten thousand alerts a day is not a monitoring problem. It is an accumulation of thresholds that made sense individually and were never revisited collectively, layered over a topology nobody has modeled.
Correlation engines help, but pointing one at an unmodeled estate mostly produces correlated noise. The prerequisite is a dependency map good enough that the system can tell a cause from its forty downstream effects.
A sequence that works
- Model the topology — service dependencies, not just host inventory
- Instrument consistently with OpenTelemetry so signals are comparable across teams
- Define SLOs, and demote every alert that does not threaten one
- Introduce correlation once there is a topology for it to reason over
- Automate remediation only for failure modes with an existing, tested runbook
Measuring it honestly
Track alerts per incident, not alerts per day — the former tells you whether correlation is working. Track mean time to acknowledge separately from mean time to resolve, because they fail for different reasons. And track how often an automated remediation was later reverted; a high revert rate means the automation is running ahead of its evidence.