External service: a separately deployed backend behind a typed client

How a SvelteKit app calls a backend that is its own process, built, deployed and scaled on its own — through one authoritative contract, one server-only service client, and a failure model that keeps the rest of the app alive when that process is not. The reference is a small Rust/Axum service, the runtime lab (services/runtime-lab/, an order-lifecycle rules engine), the service client is src/lib/server/showcases/external-service/, and the live proof is /showcases/external-service.

This is a pattern for a boundary, not a migration: v10r stays one SvelteKit app, and Rust is incidental. The contract, service client, failure model and container would be the same for a Go or Python service.

External service (docs/naming.md) means any separately deployed process the app calls over HTTP, first-party or a vendor's. This doc is about the first-party case — the one where you own both ends of the wire and can therefore make the contract executable.

When to use it — and when not

Reach for a separate service when one responsibility genuinely needs a different runtime, release cadence or resource profile from the app: a long-lived worker, a CPU-heavy engine, a component another team or product also calls, a language-specific library.

Do not, when the work fits in a #lib/server/[domain]/ module. A second deployable doubles the things that can be down, the configuration that can drift, and the places a credential lives. In-process alternatives are cheaper, in order: a domain module (multi-client-core), a Web Worker or wasm for client-side compute (architecture-wasm-kernel), a queue worker on the same host (workers.md). "Rust is faster" is not a reason by itself — see Measurement.

Architecture

browser ── form POST ──▶ +page.server.ts ──▶ callService() ── HTTPS (+ Bearer) ──▶ runtime-lab (Rust)
           (ids only)    (honeypot, IP +      (deadline, validate,                (auth, bounds, rules,
                          global limit,        classify, time)                     Server-Timing, log)
                          sub-deadline)

The browser never talks to the service: no CORS, no public proxy, no credential in the page. The page sends ids from closed lists (target, preset, drill, patience); the server owns every URL, credential and request body.

services/ is new on purpose: a separately deployed process is not a library. crates/kernel is compiled into the app; services/runtime-lab runs beside it. The directory says which (docs/codebase-organization.md).

In a product, the service client would be a domain named for what the service does — src/lib/server/order-rules/, say — holding client.ts, contract.ts, targets.ts, targets.adapter.ts and config.ts — reached through its barrel like any other domain. Here it sits under server/showcases/ because the showcase is its only caller, and isolation.gate.test.ts proves that.

The API

Liveness and identity are separate on purpose: "it is up" is public, "what it is and how long it has run" is not. A refusal is an answer, not an error — 200 with outcome: "refused" means the request was valid and a rule said no; 400 means the request was not valid. Errors use v10r's own envelope { error: { code, message } } (http/response.ts), so the boundary introduces no second API convention.

The rules: an order is placed, becomes partially_paid and paid by payments (never above the total), ships only when fully paid, is delivered after shipping, can be cancelled before shipping, and can be refunded — never above what was paid — after delivery or after a cancellation; the last refunded cent makes it refunded. Evaluation stops at the first refused event.

The contract chain

One authority, and every derivation checked:

contract.rs ──cargo test──▶ openapi.json ──external-service:contract──▶ contract.generated.ts ──svelte-check──▶ contract.ts (valibot)

1. Rust → spec. utoipa derives the OpenAPI 3.1 document from the wire types; paths come from the router itself (utoipa_axum::routes!), so a route and its spec entry cannot be registered apart. openapi_json_matches_the_code fails until UPDATE_OPENAPI=1 cargo test rewrites the committed file.
2. Spec ↔ serde. A schema generator reads attributes, not behaviour, and the two disagreed twice here: utoipa ignores rename_all_fields (it published amount_cents; serde reads amountCents) and does not close internally tagged enum variants (the spec admitted {"type":"ship","amountCents":5}, which serde refuses). Two tests now hold the published schema to serde in both directions — every response the service sends conforms, and the request schema admits exactly what serde accepts — and contract::complete() closes the variants.
3. Spec → TypeScript. bun run external-service:contract (openapi-typescript, types only — nothing ships to the browser) writes contract.generated.ts; contract-freshness.test.ts fails when it is stale.
4. Types → runtime. Types vanish at runtime and a response is untrusted input, so contract.ts holds valibot mirrors. Exact<InferOutput<…>, components['schemas'][…]> pins fail svelte-check on any field difference, and contractDrift() compares each mirror's JSON Schema with openapi.json — names, required, enums, closed objects and every bound keyword — and reports any keyword it does not model rather than skipping it. contract.test.ts proves it catches fourteen realistic breaking changes at the right path, and pins the client's operation table (path, method, auth, request/response components, error statuses) to paths.
5. Rust edits without Rust in the gate. bun run validate never runs cargo. scripts/external-service/check.sh hashes the crate's sources, its build definition and itself, runs fmt, clippy (-D warnings) and the tests, refuses to stamp if any input changed while they ran, and writes check-manifest.json with the runner it actually used. contract-freshness.test.ts recomputes the hashes and pins the stamped check list to the script's. A Rust edit without a passing check run turns validate red. Same method and the same honest limit as the wasm kernel's build-manifest.json: it proves "these files passed when the script last ran", not that they would on another toolchain.

Requests are validated closed (strictObject, like the service); responses open (object): a field the service adds later is ignored — additive change stays compatible — while a removed or renamed one fails both the drift test and, at runtime, validation.

Evolving the contract

The service is deployed by hand and the app from main, so for a while after every change the two run different code. The drift tests prove each side matches its own copy of the contract; what they cannot see is the pair actually running. Three rules keep that pair safe, and one diagnosis says when it is not.

1. The release names the contract. release — the Cargo version, which is also the spec's info.version and what /v1/status reports — follows semver for the contract: releases sharing major.minor share one contract; a minor bump is additive from 1.0 on; a major bump (in 0.x, a minor one) may break. services/runtime-lab/contract-releases.json records release → digest of the contract — the spec without info, prose (description, summary, title, examples) and tags, keys sorted — so a doc-comment edit or a generator that reorders keys is not a contract change, while a field, bound, enum value or route is. bun run external-service:contract appends a new release, never rewrites one, and checks the rule before writing, so a release that breaks it never becomes history; a missing record is refused unless --init asks for a new one. contract-freshness.test.ts fails when the spec changed under a recorded release (a forgotten bump) or two releases of one major.minor carry different contracts (a patch that changed the contract), and accepts a minor or major release that changes it — all tested. Honest limit: the test sees the record as it is, not its history; a hand edit that rewrites or deletes a row passes, so review diffs to that file. A recorded release is spent even if it was never deployed: a second contract change before deploying needs another bump. It is a record, not a classifier: whether a change is additive or breaking stays the author's call, made with rule 2.

The order, every time: bump version in Cargo.toml → UPDATE_OPENAPI=1 bash scripts/external-service/check.sh → bun run external-service:contract (types + record) → deploy the service → deploy the app.

2. Additive is directional — deploy in expand/contract order. The app sends requests validated closed and reads responses open for fields — but closed for enum values (order states, refusal reasons, error codes, operations): a value it does not know is off-contract. So:

3. A breaking change gets a new major (or 0.x minor) and a new path (/v2/…) only when the old one must keep serving deployed callers. One first-party caller usually can follow rule 2 instead; no versioning framework.

Keep /v1/status frozen. Skew is read from it, so its shape must outlive every other change: no new required fields, no new enum values in operations without a major. When status itself answers off-contract, skew reads unknown rather than nothing.

The diagnosis. contractSkew(service release, CONTRACT_RELEASE) → none · service_ahead · service_behind · breaking · unknown. The lab's check shows it beside the release and verify.ts prints it. No call is refused on it — validation on both sides decides; skew explains a malformed_response (the app cannot read the answer) or an invalid_request (the service refuses what a newer app sends) before anyone reads a diff. Verified on 2026-10-23 with a 0.1.0 service under a 0.1.1 app: none (a patch apart, one digest).

While the service is 0.x there is no additive level: patch means "no contract change" and minor "may break", so every contract change reads as breaking, including the safe service-first kind. Releasing 1.0.0 gives minor bumps their additive meaning; that is a decision about the service's stability promise, left to its owner.

Considered and not adopted: oasdiff (breaking-change detection, supports 3.1) would need a Go binary or Docker inside validate; Pact fits several consumers that both evolve, not one first-party caller; a version header or contract field on every response would itself be a contract change and is redundant with release.

The service client

callService(target, operation, input, { deadline, fetch?, requestId? }) returns a value for everything the network or the service can do:

What it guarantees, and where the code for each lives:

• Bounded. Each attempt runs inside the request's Deadline (http/deadline.ts); the route gives the call child(8 s, reserve 0.5 s) of the 10 s request budget. The body is read with readTextBounded (64 KiB). No new timeout system.
• Retried only when safe. retryWithin (resilience/retry.ts) runs a second attempt only for operations declared idempotent, and only when no response arrived. Once a response head is in, nothing is retried — the service has the request. A timeout is never retried: the request may have been processed, and on a waking host a retry only spends the rest of the budget waiting for the same cold start. evaluate is a POST but a pure function of its input, so it is declared idempotent — the declaration is per operation, not per method.
• Least privilege. The credential is sent only to operations marked authenticated; liveness gets none.
• Not redirected. redirect: 'manual': a fixed origin stays fixed, and a redirect is classified, not followed with the credential attached.
• Rate-limited in front, deliberately. Every lab bucket states its failure policy instead of inheriting defaults (LAB_LIMITER_POLICY, config.ts): closed when the limiter store is slow, and when it is unavailable. The buckets share one store, so a slow store slows them all and opening one would only delay the next one's refusal; and they are the only cap on outbound calls, so on the free host's monthly hours. A wake check — up to 75 s of a function and of the host — also passes two buckets of its own (2 a minute per IP, 10 overall). config.test.ts runs the real createLimiter against a slow, a failing and an absent store. The visitor then reads "rate limited": imprecise, and the honest short answer.
• Correlated. Each call sends x-request-id — a UUID, or the caller's id when it satisfies the service's own rule (1–64 of [A-Za-z0-9._-]), so the id in the result always matches the service's log line. Render adds its own Rndr-Id; ours is independent of the host. A request id is a correlation aid, not evidence: the service adopts any well-formed id, so anyone who can reach a public route can write a log line carrying an id they have seen.
• Timed three ways. roundTripMs (fetch start → body read), serviceMs (the service's own Server-Timing: app;dur=, read with readServerTimingMetric beside v10r's own writer in http/request-timing.ts), clientMs (everything else). The route records them as ext_* spans in locals.timing, so they appear in v10r's own Server-Timing for admins (latency-tracing) and nowhere public.
• Quiet. The credential is a header and nothing else. Results carry no origin and no credential (toPublicServiceCall), log lines carry target id, kind, status, request id and duration, and error messages are fixed strings — except a 400's, which is the service's own field-level validation message, length-capped.
• No demo switches. callService has no option to override the credential or skip validation. The lab's drills pass ordinary values instead — a spent deadline, a target with a wrong credential, an operation without a request schema — so the reference client stays the thing to copy.

Security

A static bearer credential, server to server, compared in constant time (subtle::ConstantTimeEq; only the length can leak), printable ASCII without spaces and at least 32 characters, required by the service at startup (a trailing newline from echo would otherwise make every caller's credential mismatch while liveness stays green). It is enough for this reference because exactly one trusted server calls the service, over TLS on the public hop, and the service holds no data. Its limits, stated plainly:

• It is shared and static: anyone holding it is "the app"; it carries no user identity and no per-request binding, so a captured request can be replayed until the credential is rotated.
• Rotation needs an overlap window, and has one — see Rotating the credential. Named after v10r's key ring (ENCRYPTION_KEY + ENCRYPTION_KEY_PREVIOUS): SERVICE_TOKEN + SERVICE_TOKEN_PREVIOUS.
• It authenticates the caller, not the end user. Authorization of what a user may do stays in v10r; the service trusts its one caller.

Production upgrades, in rising order of effort: OIDC workload identity (Vercel issues short-lived OIDC tokens to functions on every plan — issuer https://oidc.vercel.com/<team>, subject owner:<team>:project:<project>:environment:<env>; the service checks iss, aud, sub and the signature against the issuer's JWKS — no shared secret at all), HMAC request signing (binds method, path, body and a timestamp; stops replay and tampering), mTLS, and private networking where both ends share a provider. Do not build a general auth framework into a service like this one; pick the smallest of these that matches the threat. When to move past the static bearer: when the credential guards writes, data or spend; when there is more than one caller (each needs its own identity and revocation); or when a party in the path (a proxy you do not run) must not be able to replay or alter requests — that last one is HMAC's case. None holds for the reference today.

Rotating the credential

The service accepts SERVICE_TOKEN and, when set, SERVICE_TOKEN_PREVIOUS — held to the same rules, refused at startup when equal to the current one, and always compared too (both in constant time, so whether the second comparison runs does not depend on the first). Zero-downtime rotation, in order:

1. Generate the new credential (openssl rand -hex 32).
2. Service: SERVICE_TOKEN = new, SERVICE_TOKEN_PREVIOUS = old; deploy. Both are accepted; the listening log line says "previousToken": true.
3. App: EXTERNAL_SERVICE_REMOTE_TOKEN = new; redeploy.
4. Wait for the old credential to stop arriving. Every authenticated access-log line names the credential used — "credential":"current" or "previous", a name, never the value — so the log shows when nothing presents the old one any more.
5. Service: remove SERVICE_TOKEN_PREVIOUS; deploy. The old credential is refused from here. A previous credential left set is a second valid credential indefinitely.

Locally: EXTERNAL_SERVICE_LOCAL_TOKEN_PREVIOUS in .env feeds the compose service's SERVICE_TOKEN_PREVIOUS.

Against OWASP API Security Top 10 (2023)

API1/3/5 (object, property and function-level authorization) are the app's: the service has no users and no objects. API6 (sensitive business flows) does not apply to a pure function.

Residual risk, accepted for the reference: the 256-connection cap is a shared resource. A client that reconnects every 5 s, or one that sends pipelined requests and never reads the answers (hyper has no write timeout, so a pending write holds its connection), can keep the slots full and new connections — the host's health check included — are closed at accept. On Render the attacker never holds a TCP connection to the container: Render's proxy does, and it reads its answers. Exposed directly, add a write timeout around the connection's IO and a per-address cap, or put a proxy in front.

The service also protects itself at the connection level, because its public URL is reachable without the credential: an unfinished request head — and an idle keep-alive connection — is closed after 5 s (slowloris), a request head may be at most 16 KiB (431 past it — hyper's default would buffer ~400 KiB per connection first), at most 256 connections are open at once, a request has 5 s from head to response, and everything it logs from the request (method, path, serde's messages) is clipped by characters, never bytes — a byte cut inside a multi-byte character panicked the request task, unauthenticated, until review caught it.

Invariants, each with its proof:

Failure model — the service is optional

Only /showcases/external-service depends on the service, and isolation.gate.test.ts makes that a property of the code: nothing outside the showcase's own directories imports the service client, statically or dynamically. A target is resolved per call from the environment, so a missing variable is an unconfigured target, never a module that fails to load. The page's load makes no network call — a crawler visit must not wake a sleeping host — so the page renders identically with the service up, down or absent.

The lab distinguishes four situations a single "error" would blur: not configured (this deploy has no such target — muted, not red), misconfigured (set but refused, with the rule that refused it), unavailable (unreachable / timed_out after a real attempt), and ready (liveness and authenticated status both answered within contract).

The page says when an action did not land. An enhanced action can end without a result: Vercel's own error page (a 504 at maxDuration is HTML; since SvelteKit 3 it arrives as an error carrying that response's status), a SvelteKit error result (with its status), a redirect (no lab action redirects, so one is reported, not followed), or a fetch that got no answer (a rejected fetch, which reaches the page as SvelteKit's unexpected-error default). Each shows a dismissible notice — "the server sent no result" or "never got an answer" — the spinner clears and the session history stays (tests/browser/external-service.spec.ts drives both by intercepting the action POST). Before 2026-10-23 these results were dropped silently.

Cold starts. A free host that idles takes about a minute to start, which no ordinary request budget can hold. The lab's Check uses the ordinary 8 s ceiling and reports a timeout honestly; Wake and check gives that one action its own 75 s deadline (the route declares maxDuration: 90), so the cold start itself is observed — and the service's uptimeMs, smaller than the check's own duration, says this check started the process.

Logging never stops the service

A log reader is a dependency too. println! takes stdout's lock and blocks while the pipe is full, and panics when the write fails. Reproduced against the released binary (2026-10-23, Measurement): a reader that stalled — a container log driver in its default blocking mode, a slow collector — stopped every Tokio worker, the accept loop and /health/live included, and SIGTERM went unanswered for over 15 s; a reader that went away turned every request into a reset connection while the process stayed up.

log.rs is the one writer: a request appends its line to a bounded buffer under a short lock and never wakes anything; one thread owns stdout, drains the buffer every 20 ms and writes the batch in one write, outside the lock, ignoring write errors; a line that does not fit (1024 held — a few hundred KiB at most) is dropped and counted, and the count is written as {"event":"log_dropped","lines":n} once the writer catches up; every exit path stops the runtime's remaining connections, then wakes the writer and flushes for at most 1 s, so shutdown cannot hang on a reader that is gone — the 8 s drain plus that second stays inside Docker's and Podman's default 10 s stop grace. std only. What the count cannot cover: a batch whose write fails (the reader is gone — no one is left to tell), lines emitted after the final flush, and a last line cut short when the process exits with the writer blocked.

The first version used a channel that woke the writer for every line. Measured under the free-tier envelope, that wake-up — paid from the same 0.1 CPU quota as the requests — cost a fifth of burst throughput and tripled p95 against plain println!; writing in batches did not help, removing the wake did. The design above is the measured winner (Measurement): on a fractional CPU, a per-event cross-thread wake-up is not free. When to use it elsewhere: any service that logs from its request path to a pipe it does not control. When not: a CLI whose output is the product (the openapi subcommand keeps println!), or a service whose log lines are an audit record that must not be dropped — that one needs durable, blocking storage and back-pressure on purpose.

Idle connections and the host's proxy

Since hyper 1.6 the 5 s header-read timeout also closes an idle keep-alive connection. A proxy that reuses an upstream connection the service is closing at that moment can answer 502. Render's upstream idle timeout is undocumented and nothing like it has been observed; raising the timeout would weaken the slowloris bound, so it is measured before it is changed: verify.ts --idle-gap-ms 6000 --idle-samples 5 sends calls after quiet gaps and prints each outcome.

Container

One Containerfile for every host:

• Build context is the service directory, never the repo root; an allowlist .dockerignore admits only Cargo.toml, Cargo.lock and src/.
• Builder: the same digest-pinned rust:1.97-slim (Debian 13) as scripts/wasm/build.sh — one Rust builder for the repo, contract-freshness.test.ts asserts the pins agree. A RUST_IMAGE build arg lets a registry mirror holding the same digest substitute when Docker Hub rate-limits (mirror.gcr.io/library/rust@sha256:… — the digest still pins the bytes).
• Dependency layer via a manifest-only stub build: a source-only change rebuilds the crate, not its dependencies. Plain layers, no BuildKit cache mounts — those are not persisted by every builder (Render's are assumed cold).
• Runtime: gcr.io/distroless/cc-debian13:nonroot (digest-pinned; matches the builder's glibc; only -debian13 distroless tags are still maintained), uid 65532, no shell. The binary is PID 1 and handles SIGTERM itself; no init process.
• Reproducible binary. Two --no-cache builds of commit 32b7f3e on one builder produced a bit-identical binary (sha256 d994cda4…, 2026-10-23; also held for an earlier revision). The image digest differs between them (layer timestamps); not pursued — Render builds the image itself and no image is published.
• Run hardened. The process needs a socket and nothing else: compose runs it read_only, with cap_drop: [ALL] and no-new-privileges; smoke.sh runs the same flags, so a write or a capability the image starts to need fails there first. Resources stay unlimited for development.
• No HEALTHCHECK. OCI-format images (Podman's default) ignore it, distroless has no shell to run one, and Render uses its own healthCheckPath.
• PORT defaults to 8080; hosts that assign one (Render: 10000) set it. Binds 0.0.0.0 (IPv4, as Render requires).

Local development

The service is optional and behind a compose profile, so plain podman compose up is unchanged.

openssl rand -hex 32          # put it in .env as EXTERNAL_SERVICE_LOCAL_TOKEN
# .env: EXTERNAL_SERVICE_LOCAL_URL=http://runtime-lab:8080
podman compose --profile external-service up -d runtime-lab
podman compose up -d --force-recreate app   # env_file is read when the container is created
podman compose stop runtime-lab             # the lab reports it unreachable; nothing else changes

Rust checks never run in the app container (the wasm precedent: it stays Rust-free):

verify.ts runs real src/lib/server modules under bare Bun. They reach SvelteKit's virtual $app/env/private through the resilience barrel, so the script installs a two-module Bun plugin mapping $app/env/private (every variable src/env.ts declares) and $app/env to the process environment — the counterpart of the mock src/lib/server/test/vitest.setup.ts installs for tests (recorded in docs/codebase-organization.md).

Hosting on Render

The remote target runs on Render Free: a Docker web service in Frankfurt (next to v10r's Vercel functions in fra1), https://v10r-runtime-lab.onrender.com, created by the owner on 2026-10-23 from the feature branch. Creating or changing it is the owner's decision; a payment method on the Render account is the line not to cross (see the cost checklist).

What Render Free gives, per its docs (fetched 2026-10-22 through search summaries — the sandbox could not open render.com directly; re-check in the dashboard): 750 free instance-hours per workspace per month (suspended, not billed, when used up); spin-down after 15 minutes without inbound traffic and a cold start of about a minute; 500 build-pipeline minutes per month (overage is billed only with a payment method on file — without one, builds stop); no persistent disk, no SSH, single instance; must bind 0.0.0.0:$PORT (default 10000); TLS terminated at Render's proxy; SIGTERM then SIGKILL after 30 s on deploys. Unverified: the Free instance's CPU/RAM (a Render article says 0.1 CPU / 512 MB), Render's request timeout, Free availability per region, and whether Render's proxy forwards raw non-ASCII bytes in paths (the service is safe either way since review).

Exact setup (dashboard → New → Web Service)

Set the root directory. The first deploy left it empty and gave repo-root paths for the Dockerfile and context; Render then sent the whole repository (8.87 MB) as the build context and the build failed at COPY Cargo.toml — the Containerfile expects the service directory as its context. services/runtime-lab/render.yaml holds the same settings as a Blueprint, for reference or for a Blueprint created with that file path. Then, in Vercel (server-only env vars, Production): EXTERNAL_SERVICE_REMOTE_URL=https://<service>.onrender.com and EXTERNAL_SERVICE_REMOTE_TOKEN=<same value>.

Cost checklist

• The Render workspace has no payment method; the plan is Hobby; the instance type reads Free.
• No other paid resource is attached (no disk, no custom-domain upgrade, no paid workspace plan).
• Auto-Deploy is Off and the root directory is set, so unrelated pushes spend no pipeline minutes.
• The workspace runs no other free service: anyone can keep this one awake by hitting its public URL every <15 minutes (~744 of the 750 monthly hours, shared across the workspace) — v10r's limiters cannot see those requests. Watch instance-hours in the dashboard.
• Teardown is known: Settings → Delete Web Service; then remove the two Vercel env vars.
• The vendor checklist in vendors.md: Render publishes a GDPR DPA and holds SOC 2 Type II; the compliance row, gdpr.md and the /showcases/privacy/data disclosure name it. The service receives only server-owned synthetic order events and stores nothing; Render sees request metadata (the IPs of Vercel's functions, timing), never a visitor's.

Security checklist

• The token is new, 32+ random characters (printable ASCII, no newline), used for this target only, stored only in Render and Vercel (never in the repo or a chat).
• EXTERNAL_SERVICE_REMOTE_URL is https:// and an origin only; the lab shows the target as Configured.
• curl https://<service>.onrender.com/v1/status without the token → 401; /health/live → {"status":"live"} only.
• Render's logs show one JSON line per request with no authorization value.

Hosted verification (the evidence that promotes the remote half)

1. Leave the service idle for 20+ minutes, then run EXTERNAL_SERVICE_REMOTE_URL=… EXTERNAL_SERVICE_REMOTE_TOKEN=… bun run external-service:verify -- --target remote --samples 50 --concurrency 4 --idle-gap-ms 6000 --idle-samples 5. The first call measures the cold start; uptimeMs confirms the run woke the process.
2. All negative paths and all presets must pass (401, 400, 413, client rejection, every preset's answer, spent deadline).
3. On the deployed v10r, open /showcases/external-service after another idle period: Wake and check the Remote host card (cold — the cold-start badge must read "This check started the process"), then Check again (warm); run a preset and each drill.
4. Record the figures and conditions in Measurement, then update the pattern record (architecture-external-service) — its remote verification is pending until this is done.

Measurement

What each figure is, and where it comes from — no figure here is estimated:

Local, 2026-10-22 — verify.ts against the container, cloud sandbox (Linux, 4 vCPU, Docker 29.9 rootful, loopback), Bun 1.3.12, 200 sequential evaluate calls per run. Two runs, to show the spread:

Loopback figures in a shared sandbox move by ~2× between runs; read them as orders of magnitude. Failure detection: connection refused → unreachable in 3–34 ms (two attempts, jittered backoff); spent deadline → timed_out in 0.12–0.2 ms (nothing sent); a stopped container → unreachable in 23 ms. A 1 ms budget lost the race to the warm local service 15–19 times in 20 across runs — which is why the lab's deadline drill uses a spent budget rather than a tiny one. Container start → first liveness answer: 318–362 ms (4 runs, image present). An unfinished request head and an idle keep-alive connection are both closed at 5.0 s. Image: 11.7 MB compressed, 46.4 MB unpacked. Build: 33 s with base images cached, 8 s for a source-only rebuild (dependency layer cached).

Render Free, Frankfurt, 2026-10-23 — first deploy, from the build and runtime logs: dependencies compiled in 15.5 s and the crate in 4.4 s on Render's builder; Render exports the layer cache to its registry, so the manifest-only dependency layer is reused by later builds. The process bound 0.0.0.0:10000 from Render's PORT and logged bootMs 0.30 (process start to listening). Render probes HEAD / and GET / once at deploy — two 404 log lines, expected: the service has no route there.

Production, 2026-10-23 — the lab on www.v10r.dev (Vercel function in fra1, confirmed by the log's region=fra1) against the runtime lab on Render Frankfurt, used by hand. The service client logs one line per failed call (successes log nothing), so Vercel's runtime log holds the failure paths, with each call's total client time:

The last row is the free tier's cold start meeting a normal budget: a sleeping Render Free service does not answer within 8 s, which is why Wake and check has its own 75 s deadline. The owner then confirmed the wake check, presets and drills working on the page. The 401 calls opened the sequence and carry a new TLS connection; the 400 calls that followed reused it — a cross-provider, same-city round trip of a few tens of milliseconds against ~1 ms on loopback.

Hardening pass, 2026-10-23 — cloud sandbox, host binaries (Linux, 4 vCPU, rustc 1.97.1, Bun 1.3.12, loopback; 4 concurrent evaluate clients, 3 s per request). Before = release 0.1.0 (println!), after = 0.1.1 (log.rs):

The normal-path ranges overlap — no difference is claimed on an unthrottled host. Service time (app) was 20–21 µs in both, and both wrote every line: 25 003 of 25 003.

Controlled lab, 2026-10-23 — the image under the free-tier envelope (bench.sh: Docker 29.9.1, --cpus 0.1 --memory 512m, read-only, no capabilities; 300 × evaluate in 4 lanes; 6 alternating rounds per build). Cgroup limits reproduce the CPU quota and memory ceiling, not Render's proxy, network or scheduler:

Same envelope, release 0.1.1: Tokio sees the quota ("workers": 1; 4 at --cpus 4); container run → first liveness answer 682 ms (442 ms at --cpus 4); service time 30 µs; memory 1.4–1.9 MiB idle and after a run. At 0.1 CPU the median barely moves and the tail grows: CFS throttling — 10 ms of CPU per 100 ms period — shows up as tens of ms at p95 for any build. p99 is not quoted from 300-sample runs (three values above it); verify.ts prints it only from 100 samples and marks it indicative.

Image: 11.6 MB compressed (docker save | gzip), 46.4 MB unpacked — unchanged. Build in the sandbox: 47–61 s cold (crates fetched through the sandbox's egress proxy), 12 s for a source-only change. A release build with lto = "fat", codegen-units = 1 and panic = "abort" was measured and rejected: binary 2.44 → 1.55 MB and app p50 −7 µs, for a 90 s build instead of 47 s — against a hop of tens of ms and Render's 500 free build minutes (Rejected alternatives).

Still unmeasured — hosted: the cold-start duration, warm success-path percentiles with the Server-Timing split, the cost of a new TLS connection after a quiet gap, and whether Render's proxy ever races the 5 s idle close. The sandbox's egress proxy refuses both www.v10r.dev and *.onrender.com; the verify.ts --target remote run in step 1 of Hosted verification records all four.

What these numbers do not say: that Rust is faster. The evaluation is microseconds in any language; the boundary costs ~1 ms on loopback and will cost tens of milliseconds across a network, before any cold start. A service boundary is paid for in latency and operations — it is worth it for isolation, ownership or a runtime the app cannot host, not for speed of a small computation.

Rejected alternatives

• The browser calling the service directly (CORS). Puts the credential or a public endpoint in reach of every visitor; the server-side client makes the boundary private and lets one deadline govern the whole request.
• A generic proxy route. A caller-supplied URL or path is an SSRF primitive; targets are named, paths fixed.
• A Rust toolchain in the app container. The wasm precedent: the dev container stays Rust-free and validate never needs cargo; the manifest carries the evidence across.
• One Cargo workspace with crates/kernel. Different deploy units, lockfiles and release cadences; a workspace would couple them for no gain.
• A hand-written spec, aide, progenitor. A hand-written spec is a second source of truth; aide (schemars) is a peer of utoipa with less axum integration; progenitor generates a Rust client, the wrong direction.
• openapi-fetch or a generated runtime client. The service client needs its own deadline, bounded read, classification and timing anyway; a runtime client would be a second layer around the same fetch.
• ajv/zod for runtime validation. valibot is already the repo's validator; the mirrors are pinned to the generated types instead.
• RFC 9457 problem+json. A fine standard, but v10r's envelope is { error: { code, message } }; one convention beats two.
• tracing/OpenTelemetry in the service. One JSON line per request and Server-Timing cover this service; the latency-tracing record names OpenTelemetry as the upgrade when spans must cross processes.
• Release-profile tuning (LTO, codegen-units = 1, panic = "abort") and an alternative allocator. Measured above: 7 µs off a call that crosses tens of ms of network, for double the build minutes. panic = "abort" would also turn any handler panic — the logging panic was one — into a process crash instead of one dropped connection. An allocator matters at high allocation rates; this process holds ~2 MiB.
• TCP_NODELAY on accepted sockets. hyper writes a small head and body in one flush; no Nagle/delayed-ACK stall (40 ms) appeared — loopback p99 1.5–2.2 ms. Revisit if responses become streamed or large.
• A long-lived undici Agent for connection reuse. Node's fetch keeps an idle connection 4 s, so most lab actions — seconds apart — open a new TLS connection. Same-metro, that costs a few tens of ms (the production 401/400 rows above); a module-scope Agent needs the npm undici package beside Node's bundled one and risks stale sockets after a Fluid instance suspends. The idle-gap series measures the cost first.
• Hashing the bearer before the constant-time compare. It hides the length — which, for a server-chosen random credential, is not the secret — at the price of a hash dependency.
• A breaker around the call. resilience-policy has one (defineBreaker); here it would fight cold starts — a waking free host fails the first calls by design, and an open breaker would keep refusing after it woke. Add one for a service that is expected to be always-on.
• Fault-injection endpoints or flags — in the service or the client. Anything that can make a call fail on purpose can be reached by mistake, and whatever the reference client carries gets copied. Drills change only the values a caller passes.
• A database or any persistent state in the service. Out of scope for the reference; it would make it a different pattern.

Emulating this pattern

Take the boundary, not v10r: none of this needs the rest of the repository.

1. Put the service in its own directory with its own lockfile, toolchain pin and container definition; build context = that directory.
2. Make the service's wire types the authority and generate the spec from them; snapshot-test the spec, and test it against your serializer's real behaviour in both directions.
3. Generate client types from the spec; add a runtime validator pinned to those types; prove the drift check with deliberately broken specs, and make it fail on keywords it does not understand. Let the service's release name its contract (a patch never changes it), record release → digest of the canonical contract (no prose, sorted keys) append-only, report skew between the running pair as a diagnosis, and deploy additive changes service-first. Watch the closed-enum trap — a reader that validates enum values strictly turns a new response value into a breaking change — keep the endpoint skew is read from frozen, and leave 0.x once minor bumps should mean "additive".
4. One server-only service client, as a domain named for the service: fixed targets from server config, origin-only URLs, TLS off private hosts, a deadline per call, a bounded body read, one failure kind per cause, retries only for idempotent operations when no response arrived, no redirects, the credential only where needed, a request id, and the service's own timing header. No demo options in its signature.
5. The service authenticates every functional route itself, refuses to start without a well-formed credential, accepts a previous credential during a rotation, keeps liveness public and empty, bounds its input and its request heads, times out slow request heads, caps connections, clips what it logs by characters, and logs one line per request with no headers or bodies — through a writer that can neither block nor fail a request, and that does not wake a thread per line on a fractional CPU.
6. Keep the service optional in code: one importer, configuration read per call, no network work on page load.
7. Ship one image to every host; let the host set only the port and the secret. Run it read-only and capability-free; measure it under the host's CPU and memory envelope (a cgroup limit), not only on a developer machine; audit its dependencies outside the gate.
8. Measure before optimizing: here the hot path was 20–30 µs against tens of ms of network, so the release profile, allocator and Nagle were left alone, and the one change that moved the numbers was in logging.