Skip to main content

Observability

Honest performance measurement: separating real telemetry from your own dev traffic, and turning budgets into tests.

The dev server was writing telemetry into the production analytics database. That is a common mistake and an unusually nasty one, because of which direction it skews the numbers.

Contamination that flatters

Dev samples come from localhost: no network, no TLS, no cold start. They do not add symmetric noise around the truth — they drag every latency percentile downward. Over one 30-day window, 72% of collected samples were dev rows, and first-byte latency read far better than reality.

Reported (all samples) 1051 ms
Actual (production lane) 1431 ms

The discriminator

No extra column and no backfill were needed. The attribution build records the CSS selector of the element it blamed, and Svelte's scope-class format differs between builds — so every sample already carries a fingerprint of the build that produced it.

Dev server nav.flex-1.overflow-y-auto.p-2.s-Xv7_7mcdkQaC.scrollbar-nav
Production nav.flex-1.overflow-y-auto.p-2.scrollbar-nav.svelte-1e55qdy

The dev pattern is anchored so it cannot match the production prefix: it requires a hyphen immediately after the s, and svelte- has a v there. A discriminator that matched both would filter out every real sample and show an empty production lane on a perfectly healthy deployment.

Live split

Classified across the last 30 days of this site's own telemetry. A session showing any dev marker is treated as dev — losing a real session costs one sample, keeping a dev one costs the accuracy of every percentile it lands in.

Production 65 (10 sessions) Unclassified 17 (4 sessions)

What this generalises to

Fixing the source is necessary but not sufficient. The read-side filter is what makes the fix durable:

  • Fix the source — gate telemetry on a production check, so no new bad rows are written.
  • Filter on read anyway — rows already written stay inside the retention window and keep skewing every aggregate over them.
  • Report what you excluded — a count that can rise is the only thing that notices if the source-side gate ever regresses.

Think this pattern could be better? Tell us how.

Leave feedback