Revisited from the Foresaw archive. The argument and structure have been updated for this edition.
Operations teams work with streams of metrics, logs, events, and alerts. Machine learning can be explored as a way to find patterns in that information, but a detected pattern is not automatically an actionable incident.
The engineering question is whether a signal helps someone make a better operational decision.
Start with the response
Decide what should happen when the system flags something unusual. Who reviews it? What context do they need? Which action is safe, and which needs approval?
If there is no useful response, adding another detection mechanism may increase the alert burden rather than reduce it.
Establish a baseline
Compare the model with a straightforward threshold or existing operational rule. Use historical events where the team understands what happened, and keep the evaluation separate from the data used to develop the model.
Count false alarms and missed incidents in terms of their operational cost. A high-level score can hide a system that repeatedly interrupts the wrong person.
Account for changing behavior
New releases, traffic changes, and infrastructure migrations can alter what “normal” looks like. Record the context around the signals and define when the evaluation should be revisited.
Keep automation bounded
Begin with explanation and decision support before delegating consequential actions. If a remediation step is automated, define its limits, observability, and recovery path.
The most useful outcome may be a clearer queue for a human operator. Automation should follow the quality of the evidence and the reversibility of the action.
Adapted from “The Role of Machine Learning in IT Operations: Improving Efficiency and Performance.”
First published on Foresaw. Reworked for this edition.