Monitoring & Observability

The two failure modes

Either nothing alerts and you learn about outages from customers, or everything alerts and the on-call engineer has muted the channel. The second is more common and more dangerous, because it looks like you have monitoring.

What we change

We start from the user’s experience and work inwards. Can people log in? Can they complete the core action? How long does it take? Those become your SLOs, and your error budget becomes the thing that decides whether next sprint is features or reliability.

Then we cut. Most alerting setups can lose over half their rules without losing coverage, because they alert on causes — high CPU, a restarted pod — that frequently do not affect anyone. CPU at 90% is not an incident. Checkout failing is.

Keeping the bill sane

Observability platforms are priced to grow faster than your traffic. We set retention tiers by value — full-fidelity for recent data, sampled or aggregated beyond that — and drop the high-cardinality labels nobody queries. It is common to cut ingest substantially while improving what you can actually answer.