Observability

Observability is three separate feeds. Errors tell you something threw and where. Traces tell you what a single request did across the services it touched. Metrics tell you how the system behaves in aggregate over time. Sentry does the first well and some of the second, OpenTelemetry is the vendor-neutral way to produce traces, and Prometheus scrapes and stores metrics. You need them in proportion to how much of your system runs when nobody is looking at it.

How we use it

DMARC Engine is the case in point. Most of its work is asynchronous: aggregate reports arrive by email from other organisations' mail servers, get parsed, unzipped and written, alert rules are evaluated on a queue, and a scheduled worker republishes flattened SPF records. Nobody is sitting in front of any of that. A parse failure on a malformed report is silent by construction, so Sentry is what turns it into something a person finds out about, and the enforcement journey from p=none to p=reject depends on the data arriving intact.

Starterflare sits at the other end. It carries OpenTelemetry tracing and Prometheus metrics because it is a modular monolith with Postgres, Redis, a message broker and a search cluster behind it, and in that architecture a slow request has five places to hide. Worth saying plainly: that codebase has never been released, so its telemetry has never had production traffic through it. The instrumentation is designed rather than proven, and we would not claim otherwise.

What it costs you

Error reporting is billed by event volume, and event volume is not something you control on your worst day. One bad deploy in a retry loop will spend a month's quota before lunch, so sampling, rate limits and sensible grouping are part of the setup, not a later tuning exercise. Payloads need scrubbing too: an error from a mail parser can carry someone else's message headers, and an error from a checkout can carry an email address, so the redaction rules matter before the first event is sent.

A self-hosted metrics stack is infrastructure. Prometheus wants storage, retention rules and somewhere to run, a collector is another process to keep alive, and both of them run whether or not the product has users, which is the same arithmetic we set out in server stack versus serverless cost. On Kubernetes that is often already there, which is one of the few things that makes a cluster easier rather than harder.

The failure that actually happens is not missing data. It is alerts nobody reads. Dashboards accumulate, thresholds get set optimistically, and within a quarter a team is muting a channel. Fewer alerts, each one pointing at a specific action, is the version that survives.

When we would choose something else

For a small edge application, the platform's own request analytics and log tail cover most of the ground for nothing extra, and we would start there rather than instrument a build that gets forty requests a day; that is part of what running on Workers already gives you. Add error reporting when there is an asynchronous path with no user watching it, and add tracing when there are enough hops that you cannot guess where the time went. Before all of it, structured logs with a request ID solve more problems than most people expect, and cost a morning.