What happened
Atlassian’s central monitoring team published an account on the CNCF blog of how they rebuilt the system that automatically raises high-severity incidents. They moved from a Node.js aggregator on roughly 90 virtual machines to Apache Flink on Kubernetes, fed by Kafka and emitting metrics through OpenTelemetry. Event-to-metric time had been over 40 seconds, and the target for the new design was under 10. They also shared honest numbers: recall of real incidents rose from about 60% to a peak of 86%, then dipped to 64% in one bad month.
Why it matters
You almost certainly do not run billions of events a day. The lessons still carry over, because the failure modes are common.
- Detection that shares infrastructure with everything else can fail with it. Two of their incidents in a year came from another tenant’s lag on a shared queue.
- Cost followed the number of things onboarded, not traffic. Their run cost roughly doubled as products were added.
- Silence is not health. Client-side telemetry only exists when a page loads, so a database shard that is completely down produced no errors at all. A check that only looks for failures reads that as fine.
- Slow detection compounds. Their old pipeline added more than 40 seconds before the detectors even started collecting their own windows of data.
For a small company, the question from the article applies directly: after your last outage, did your monitoring notice first, or your customers?
What to do this week
- Pull up your last two incidents and write down who noticed first, and how many minutes passed between the start and the first alert.
- List the alerts that fire on errors only. For each, ask what happens if the service produces no traffic at all, and add a check for a drop in volume or a missing heartbeat.
- Check whether your alerting path shares a queue, cluster or account with the workloads it watches. If it does, note what would silence it.
- Confirm you can replay or backfill events after a restart without double-counting. Ask whether your pipeline would survive that.
- Track recall and precision informally: how many real incidents were caught, and how many pages were noise. Atlassian found the numbers more useful than slogans, and so will you.
Need a hand?
Fast detection is only half of the job. The other half is knowing what you do when a region or database really is down. Our Disaster Recovery service covers failure scenarios, recovery plans and the monitoring that tells you early. The first call is free and it is with an engineer, not a salesperson.
Related service: Disaster Recovery & Business Continuity