AI Testing Infrastructure

Superseded in part. This document records the original design and the evidence behind it. Three things have since changed and strategy.md is authoritative: the suite is split into unit and db vitest projects, and the db lane restores a PGlite datadir snapshot built once per run rather than calling pushSchema per file — pushSchema measured 61 ms of a 1,435 ms per-file cost, while initdb alone was 840 ms. The automatic Claude Stop hook was also removed; agents run the explicit bun run validate gate instead.

Status: built. The Vitest + PGlite harness described here exists and works (vitest.config.ts, the createTestDb() helper in src/lib/server/test/db.ts, hundreds of *.test.ts files, in-process PGlite Postgres). Schema setup uses pushSchema from drizzle-kit/api — no migration files, no drizzle/ directory, no schema-gen step. The root AGENTS.md exists. There is no CI, by decision: the gate is bun run validate, which the agent runs (cloud sessions directly, local ones in the container).

This document describes the testing approach — what exists, why, and how.

The target is an AI agent testing loop: implement → run tests → parse output → fix failures → repeat. Claude Code does this natively. The infrastructure's job is to give the agent clear, structured feedback so that loop terminates quickly.

Execution model

Cloud sessions (the default) have no container: the same commands run directly (bun run test), on a VM the SessionStart hook prepared (development-environment.md). The rest of this section is the local model.

Locally, nothing is installed on the host machine except Podman. Claude Code runs on the host but all tools (Bun, Vitest, Biome, node_modules) live inside the v10r container. Every local command goes through podman exec.

HOST (Podman + Claude Code only)
 │
 │  edits files at /home/ad/dev/v10r/
 │  runs: podman exec v10r <cmd>
 │
 └──► CONTAINER v10r (WORKDIR /app)
      ├── Bun runtime
      ├── node_modules (named volume)
      ├── Source at /app/ (bind mount ← host)
      ├── Vitest runs here
      ├── PGlite runs here (in-process)
      └── Biome runs here

Bind mount (.:/app in compose.yaml): file edits on the host are instantly visible in the container. No copy step.

Exit codes pass through: podman exec returns whatever the inner command returns. Hooks work correctly — a failing test exits non-zero.

No path translation needed: Vitest and Biome run from the project root against src/. Host paths never enter the commands.

Container must be running: podman exec fails if v10r isn't up. If you're coding, the dev server is running — this is always true in practice.

No wrapper scripts: Use podman exec v10r <cmd> directly. Explicit is better.

Test Strategy

Two tools. No more.

Why Vitest and not Bun test

Every module in #lib/server/ eventually imports $app/env/private. It is a SvelteKit virtual module — Vite resolves it at build time. Bun test has no Vite plugin system and cannot resolve it. Tests fail immediately on import (oven-sh/bun#5541, oven-sh/bun#10712 — both still open as of early 2026). Additionally, PGlite itself produces out-of-bounds WASM errors with bun test (oven-sh/bun#15032).

Vitest with the @sveltejs/kit/vite plugin resolves the $app/* virtual modules; #lib/* is a package.json subpath import any resolver follows. Same tool handles both server-side unit/integration tests and Svelte component tests.

Vitest 4.x: Use the projects field (not workspace, deprecated in Vitest 3.2) for multi-project configuration. Browser Mode is now stable in Vitest 4.0 but not needed for this project's initial phases.

Why Playwright only for showcase specs

The Claude Chrome extension handles exploratory browser interaction, and component tests stay out of the browser: vitest-browser-svelte needs ~300MB Chromium in the container, not justified until node-side tests hit real limits. The one exception is an interaction-heavy showcase whose proof is a repeatable pointer and keyboard run — WebGL picking, gizmo drags, file round-trips. Each such showcase gets one @playwright/test spec in tests/browser/, run by bun run test:browser and kept out of validate (strategy.md, "Where UI is proven").

Why vi.mock and not dependency injection

Every query and mutation module imports db as a module-level singleton (import { db } from '../index'). Converting to DI would change 30+ function signatures across 8+ files plus all call sites — high cost, zero production benefit. The multi-client core architecture explicitly chose module-level imports over DI containers (zero-cost imports vs 50–200ms cold start on serverless). vi.mock('#lib/server/db', ...) is a one-line swap at a clean module boundary.

Directory Structure

Tests live next to source (co-located). A test file for send.ts sits at send.test.ts.

src/
  lib/
    server/
      errors/
        index.ts
        index.test.ts              ← unit test
      retrieval/
        rank.ts
        rank.test.ts               ← unit test (pure logic, highest value)
        chunk.ts
        chunk.test.ts              ← unit test (pure logic)
      db/
        notifications/
          queries.ts
          queries.test.ts          ← integration test (PGlite)
      notifications/
        send.ts
        send.test.ts               ← integration test (PGlite)
      test/                        ← shared test utilities (not a test runner target)
        db.ts                      ← PGlite setup + schema push (drizzle-kit/api)
        fixtures.ts                ← test data factories
    components/
      primitives/
        button/
          Button.svelte
          Button.test.ts           ← component test

Database Test Isolation

PGlite (unit and integration tests)

PGlite runs WASM Postgres in-process inside the v10r container. No external DB, no network, no shared state. Each test file gets its own database instance — parallel-safe by default. PGlite installs as an npm devDependency into the named node_modules volume — no host installation needed. Target version: 0.3.14+ (PostgreSQL 17.4 based).

Schema sync via pushSchema: The harness calls pushSchema(schema, db) from drizzle-kit/api. This compiles the live TypeScript schema into DDL in-process — no migration files, no drizzle/ directory, no codegen step. It matches the project's push-only workflow (drizzle-kit push against the live Neon DB), so test and production schemas come from the same source with nothing to regenerate.

pushSchema returns { statementsToExecute }. The harness executes those statements manually rather than calling the bundled apply helper — apply fails on this project's unqualified enum references. The execution order matters:

1. Run every CREATE SCHEMA statement first.
2. Set search_path so unqualified enum references resolve.
3. Run the remaining DDL.

This project defines 14 custom PostgreSQL schemas (admin, ai, analytics, app, auth, blog, dbops, desk, feedback, image, jobs, notifications, rag, showcase) plus public, with schema-scoped enums and cross-schema foreign keys. The schema-first + search_path ordering is what makes those resolve under PGlite, which starts with only public. The search_path set in createTestDb() lists 11 of the 14 — it omits dbops, feedback, and image (a known gap; those three define enums too).

Rejected: migrate with generated files. The migrate function from drizzle-orm/pglite/migrator needs SQL migration files. The project is push-only — there is no drizzle/ directory and no drizzle-kit generate step. Adding one solely for tests would split the schema source and create drift between the test schema and the live DB. pushSchema reads the same TypeScript schema the app uses, so no sync step exists to forget.

Driver difference: Production uses drizzle-orm/neon-serverless. Tests use drizzle-orm/pglite. Same query API, different connection. Swap via vi.mock.

pgvector: PGlite supports pgvector via @electric-sql/pglite/vector. Must be loaded explicitly via the extensions option — without it, any column using vector(1536) fails with "type vector does not exist." The CREATE EXTENSION must run before any DDL that references the vector type.

Custom schemas: PGlite starts with only public. pushSchema emits CREATE SCHEMA statements for every custom schema; the harness runs them first (before any table DDL), then sets search_path across them (currently 11 of the 14 — dbops, feedback, image omitted) so unqualified enum references resolve.

Cross-schema foreign keys: Almost every schema references auth.user.id. PGlite tolerates forward references within a single exec batch, but the auth schema and its tables must exist before dependent tables are created — pushSchema orders the returned statements so they do. Schemas cannot be loaded in true isolation; when bootstrapping new test coverage, expand incrementally:

Out-of-schema DDL: only the seed row in embedding_model (scripts/db/setup-retrieval.ts, phase 2) lives outside the Drizzle schema. The generated tsvector column (search_vector), its GIN index and the HNSW index on chunk.embedding are declared in schema/retrieval/chunk.ts, so pushSchema emits them — a test that needs the seed row inserts it itself.

Index support: PGlite supports GiST (for range types), GIN (for tsvector/jsonb), and B-tree indexes natively — no extensions needed. HNSW indexes (pgvector) are theoretically supported but unverified in WASM at scale — functional correctness is expected, not performance parity with native Postgres.

Snapshot/restore (future optimization): PGlite's dumpDataDir/loadDataDir API can cut test init time by creating the schema once and restoring per suite instead of re-running the push. The API is documented and stable, but the pattern is experimental at scale. Consider this if the test suite grows slow.

The real helper lives at src/lib/server/test/db.ts:

// src/lib/server/test/db.ts
import { PGlite } from '@electric-sql/pglite';
import { vector } from '@electric-sql/pglite/vector';
import { pushSchema } from 'drizzle-kit/api';
import { drizzle } from 'drizzle-orm/pglite';
import * as schema from '#lib/server/db/schema/index.ts';

export async function createTestDb() {
  const client = new PGlite({ extensions: { vector } });
  await client.exec('CREATE EXTENSION IF NOT EXISTS vector');
  const db = drizzle(client, { schema });

  // Get DDL from drizzle-kit push. Don't call apply — it fails on unqualified enums.
  const { statementsToExecute } = await pushSchema(schema, db as any);

  // Create schemas first, then set search_path, then run the rest.
  const schemaStmts = statementsToExecute.filter((s) => s.startsWith('CREATE SCHEMA'));
  const otherStmts = statementsToExecute.filter((s) => !s.startsWith('CREATE SCHEMA'));

  for (const stmt of schemaStmts) await client.exec(stmt);

  await client.exec(
    'SET search_path TO public, admin, ai, analytics, app, auth, blog, desk, jobs, notifications, rag, showcase',
  );

  for (const stmt of otherStmts) await client.exec(stmt);

  return { db, client };
}

The db as any cast is needed because PGlite's db type doesn't match pushSchema's strict PgDatabase generic. Setting search_path before the table DDL is the load-bearing step — without it, the schema-scoped enums in unqualified column definitions fail to resolve.

Inject into tests via vi.mock:

// In any test file that needs the DB
vi.mock('#lib/server/db/index.ts', async () => {
  const { createTestDb } = await import('#lib/server/test/db.ts');
  const { db } = await createTestDb();
  return { db };
});

Cleanup: Each test file creates one PGlite instance (via vi.mock factory, called once per module). Vitest runs files in parallel across workers — this is safe. Use afterAll to shut down the client:

// In test files using PGlite
let testClient: PGlite;

vi.mock('#lib/server/db/index.ts', async () => {
  const { createTestDb } = await import('#lib/server/test/db.ts');
  const { db, client } = await createTestDb();
  testClient = client;
  return { db };
});

afterAll(async () => {
  await testClient?.close();
});

Neo4j (unit and integration tests)

Mock the cypher() function. It's a single import — one mock covers all graph calls.

vi.mock('#lib/server/graph/index.ts', () => ({
  cypher: vi.fn().mockResolvedValue({ records: [] }),
}));

For tests that need specific graph responses, configure the mock per test:

vi.mocked(cypher).mockResolvedValueOnce({
  records: [{ get: (key: string) => mockData[key] }],
});

Test Aura instance: only if graph features require E2E validation. Not needed for unit or integration coverage.

Test Data Factories

Factories produce valid test objects with sensible defaults. Override only what the test cares about.

// src/lib/server/test/fixtures.ts
import type { InferInsertModel } from 'drizzle-orm';
import type { user } from '#lib/server/db/schema/auth/_better-auth.ts';
import type { notifications } from '#lib/server/db/schema/notifications/notifications.ts';

type UserInsert = InferInsertModel<typeof user>;
type NotificationInsert = InferInsertModel<typeof notifications>;

export function makeUser(overrides?: Partial<UserInsert>): UserInsert {
  const id = overrides?.id ?? crypto.randomUUID();
  return {
    id,
    name: `Test User ${id.slice(0, 6)}`,
    email: `${id.slice(0, 8)}@test.local`,
    emailVerified: false,
    createdAt: new Date(),
    updatedAt: new Date(),
    ...overrides,
  };
}

export function makeNotification(overrides?: Partial<NotificationInsert>): NotificationInsert {
  return {
    id: crypto.randomUUID(),
    userId: 'must-be-set',
    type: 'system',
    messageKey: 'notif_system',
    messageParams: {},
    isRead: false,
    createdAt: new Date(),
    ...overrides,
  };
}

Claude Code edit hook

Not configured. .claude/settings.json holds only the SessionStart hook (scripts/cloud/session-start.sh). The PostToolUse lint hook below is a design; a cloud session would run its command without podman exec.

Local hook commands run on the host machine (where Claude Code runs). Since all tools live in the container, every command uses podman exec v10r.

PostToolUse — lint only

Runs after every Edit or Write. Fast (sub-second). Biome only — no tests.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [{
          "type": "command",
          "command": "podman exec v10r bun biome check --write --reporter json src/"
        }]
      }
    ]
  }
}

No path translation needed — Biome runs against src/ from the container's /app workdir.

Why not run tests here: Running full tests after every edit causes thrashing. The agent gets failure reports for partially-complete work and starts fixing intermediate states before the feature is done.

Machine-readable output

Locally, prefix every command with podman exec v10r; cloud sessions run them as written.

Configuration Files

vitest.config.ts

import { defineConfig } from 'vitest/config';
import { sveltekit } from '@sveltejs/kit/vite';

export default defineConfig({
  plugins: [sveltekit()],
  test: {
    include: ['src/**/*.test.ts', 'src/**/*.svelte.test.ts'],
    environment: 'node',
    globals: true,
    testTimeout: 15_000,
    setupFiles: ['src/lib/server/test/vitest.setup.ts'],
    passWithNoTests: true,
  },
});

The sveltekit() plugin resolves the SvelteKit virtual modules ($app/*). Without it, every server-side test fails on import.

testTimeout: 15_000: PGlite WASM startup + schema push can take 3–5 seconds per test file. The default 5000ms timeout causes false failures. Set higher globally, keep individual tests fast.

setupFiles: Runs once per worker (per test file). Used for global mocking.

Vitest 4.x note: The workspace field is deprecated since Vitest 3.2. Use projects for multi-project configuration if needed later. The simple single-project config above works for the initial phases.

vitest.setup.ts

// src/lib/server/test/vitest.setup.ts (abridged)
import { vi } from 'vitest';

// Prevent job schedulers from starting setInterval during tests
globalThis.__v10r_scheduler = 'test' as never;
globalThis.__v10r_delivery_scheduler = 'test' as never;

// Secrets modules check at load; a real value in the environment still wins
process.env.BETTER_AUTH_SECRET ||= 'vitest-better-auth-secret-at-least-32-characters';
process.env.ALTCHA_HMAC_KEY ||= 'vitest-altcha-hmac-key-at-least-32-characters';

// Hermeticity tripwire: remote-service credentials are deleted
// (NEON_DATABASE_URL_PROD, NEO4J_*, UPSTASH_*, EXTERNAL_SERVICE_*)

// Every variable declared in src/env.ts, as a getter reading process.env live
vi.mock('$app/env/private', async () => (await import('./env-mock.ts')).envModule((name) => process.env[name]));

vi.mock('$app/env', () => ({
  building: false,
  browser: false,
  dev: true,
  version: 'test',
}));

Why the $app/env/private mock: SvelteKit validates the variables src/env.ts declares when the server starts and fixes them for the process. Tests set and delete process.env.X between cases, so envModule() (src/lib/server/test/env-mock.ts) exposes each declared variable as a getter that reads process.env on every access. A variable missing from src/env.ts is not exported at all.

Why scheduler sentinels: hooks.server.ts imports #lib/server/jobs/scheduler and #lib/server/jobs/delivery-scheduler as side effects. Both start setInterval loops unless a globalThis sentinel is set. The sentinels are the cleanest containment — no need to mock the platform module.

Why the $app/env mock: Both schedulers check building from $app/env. Other modules (sidebar.svelte.ts) check browser. Providing deterministic values prevents runtime detection logic from firing.

Load-time secrets

There is no .env.test. The $app/env/private mock reads process.env, and the load-time secrets are set there by vitest.setup.ts: #lib/server/auth/index.ts throws at module load if BETTER_AUTH_SECRET is shorter than 32 characters, and abuse/config.ts asserts ALTCHA_HMAC_KEY. #lib/server/db/index.ts reads NEON_DATABASE_URL_PROD at module load, and the setup deletes it on purpose — PGlite replaces the DB, so a test that reaches the real client fails loudly.

package.json scripts

{
  "scripts": {
    "test": "vitest run",
    "test:watch": "vitest",
    "validate": "bun run check && biome ci . && bun run test && bun run i18n:check-missing && bun run content:check"
  }
}

validate is the quality gate command. From the host: podman exec v10r bun run validate.

There is no schema-gen step. createTestDb() builds the schema in-process via pushSchema, reading the same TypeScript schema the app uses. The live Neon database is managed via drizzle-kit push (db:push); tests need nothing regenerated after a schema change.

Implementation Phases

Phase 0 — Foundation

1. Add vitest to devDependencies in package.json
2. Create vitest.config.ts with sveltekit() plugin, testTimeout, and setupFiles
3. Create src/lib/server/test/vitest.setup.ts — global mocks and scheduler sentinels
4. Set the load-time secrets in vitest.setup.ts (no .env.test; see above)
5. Add test scripts to package.json (test, test:watch)
6. Restart container so deps install: podman compose down && podman compose up -d
7. Ensure .svelte-kit/ types exist — the sveltekit() plugin needs them. The dev server creates them automatically on startup (svelte-kit sync). If running tests before the dev server has started, run podman exec v10r bun run check first (which triggers svelte-kit sync).

No tests yet. Verify from host: podman exec v10r bun run test exits cleanly with "no test files found."

Phase 1 — Pure logic tests

Start with modules that have zero external dependencies — no DB, no mocks, maximum signal. These validate the full pipeline (host → container → Vitest → results) while testing the highest-value algorithmic code.

Priority order:

1. #lib/server/retrieval/rank.ts — RRF algorithm, deduplication, fusion. Four functions, deterministic math, highest ROI. The ranking logic is non-trivial and could easily have off-by-one errors.
2. #lib/server/retrieval/chunk.ts — text splitting with overlap and hierarchy. Only needs crypto.subtle (available in Bun). Complex enough to have real bugs.
3. #lib/server/errors/index.ts — ServerError class, toStatus() mapping, JSON serialization. Pure logic, validates the pipeline.
4. #lib/schemas/showcase/ — Valibot schema rules. Test with v.safeParse(). The realtimeSchema password-confirm cross-field validation is the most interesting.

// Illustrative — `server/errors/index.ts` has no dedicated test today; its behaviour is covered through the adapters that throw ServerError
import { describe, expect, it } from 'vitest';
import { ServerError } from './index';

describe('ServerError', () => {
  it('sets kind and message', () => {
    const err = new ServerError('db', 'something failed');
    expect(err.kind).toBe('db');
    expect(err.message).toBe('something failed');
  });

  it('is an Error instance', () => {
    expect(new ServerError('ai', 'provider unavailable')).toBeInstanceOf(Error);
  });

  it('maps to 500 by default', () => {
    const err = new ServerError('unknown', 'test');
    expect(err.toStatus()).toBe(500);
  });

  it('serializes to JSON', () => {
    const err = new ServerError('db', 'connection lost', 'ECONNREFUSED');
    const json = err.toJSON();
    expect(json).toEqual({
      name: 'ServerError',
      kind: 'db',
      message: 'connection lost',
      code: 'ECONNREFUSED',
      status: 500,
    });
  });
});

Goal: verify the full pipeline works end-to-end. Claude Code (host) runs podman exec v10r bun run test → Vitest resolves #lib paths inside container → results flow back to host.

Phase 2 — DB tests

1. Add @electric-sql/pglite to devDependencies in package.json (drizzle-orm and drizzle-kit are already dependencies)
2. Restart container: podman compose down && podman compose up -d
3. Create src/lib/server/test/db.ts — PGlite setup with pushSchema (create schemas first, set search_path, run the rest)
4. Create src/lib/server/test/fixtures.ts — data factories
5. Validate incrementally — start with the jobs schema (standalone, zero cross-schema FKs) to prove the pipeline, then auth + notifications for real business logic
6. Write first DB test for notification queries

If pushSchema fails:

• Enums don't resolve: confirm search_path is set across all custom schemas before the table DDL runs, and that CREATE SCHEMA statements run first. This is the ordering in createTestDb().
• It hangs or errors on apply: don't call the bundled apply helper — execute the returned statementsToExecute yourself, as the harness does.
• Unsupported types under PGlite: load the needed extension via the extensions option (e.g. vector) before running DDL that references the type.

// Illustrative — the shipped equivalent is src/lib/server/db/notifications/queries.pglite.test.ts (PGlite lane)
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { makeUser, makeNotification } from '#lib/server/test/fixtures.ts';

vi.mock('#lib/server/db/index.ts', async () => {
  const { createTestDb } = await import('#lib/server/test/db.ts');
  const { db } = await createTestDb();
  return { db };
});

describe('getNotifications', () => {
  it('returns notifications for user', async () => {
    const { getNotifications } = await import('./queries');
    const user = makeUser();
    // seed test data, then assert
    const results = await getNotifications(user.id, 10, 0);
    expect(results).toBeInstanceOf(Array);
  });
});

Phase 3 — Expand coverage

Priority order based on codebase analysis (highest value first):

Data access layer (primary test boundary):

1. #lib/server/db/notifications/mutations.ts — race condition in getOrCreateSettings(), IDOR protection via userId scoping in markAsRead(). High bug potential.
2. #lib/server/db/notifications/queries.ts — core CRUD with auth scoping.
3. #lib/server/db/ai/mutations.ts — multi-step operations with ownership verification.
4. #lib/server/db/errors.ts — classifyDbError(), classifyCode(), safeDbMessage().

Orchestration layer:

5. #lib/server/notifications/send.ts — sendNotification() coordinates DB insert, SSE push, and external routing. Requires mocking SSE and router in addition to the DB.
6. #lib/server/retrieval/index.ts — 261 lines of pure orchestration over embedding, three retrieval tiers, and ranking. Highly testable with mocked tier functions, no DB needed.

Auth and security:

7. #lib/server/http/guards.ts — requireAuth, requireAdmin (throwing, pages) and guardApiUser (returning, endpoints). Takes App.Locals as a plain argument — mock the object, assert redirect/error behavior.
8. hooks.server.ts — security middleware (headers, CSRF, auth, rate limiting). Isolate as a separate concern: it has side-effect imports (schedulers), Redis connection, and feature logging that must be fully mocked. Mock auth.api.getSession, @upstash/ratelimit, construct RequestEvent objects. 515 lines of critical security logic.

Load functions:

9. Form showcase load functions (forms/basics/contact, forms/validation/server, etc.) — import only from sveltekit-superforms and #lib/schemas/. No DB, easiest load function tests. Call load() directly with a mock event object.

Components:

10. Alert.svelte — self-contained, internal $state(true) for visibility, dismiss flow. Good first component test.
11. Primitives (Switch, Toggle, Progress) — prop-to-DOM behavior, validates custom spacing tokens.

State machines (high value, tested as factories):

12. createRetrievalTrace() from retrieval-trace.svelte.ts — streaming annotation processing, cursor tracking, step status transitions. Pure factory, no external imports.
13. createDockState() from dock.state.svelte.ts — tab activation, panel close, split creation. Pure factory.

What NOT to test (thin adapters):

• Route handlers (+page.server.ts, +server.ts) — they are 15–25 line adapters that parse params and call domain functions. If the domain functions work, the routes work. Testing them requires constructing RequestEvent objects with high ceremony for low signal.
• Structural components (Card.svelte, BackLink.svelte, Kbd.svelte) — pure slots/snippets, no logic.
• Canvas/WebGL components (NetworkGraph.svelte, Scene.svelte) — impossible in jsdom.

Side-effect containment: The globalThis sentinels in vitest.setup.ts prevent scheduler startup. No earlier test should import anything that chains to hooks.server.ts. The vi.mock('#lib/server/db', ...) pattern is safe because query/mutation modules only import from '../index' (the DB client) — they do not chain through hooks.

Svelte 5 state testing note: All .svelte.ts state files in this project use the context factory pattern (createToastState(), createDockState(), etc.), not module-level $state singletons. This means the flushSync() requirement for external reactive state does not apply — call the factory directly in tests.

Phase 4 — AGENTS.md (done); CI (declined)

AGENTS.md is at the project root. CI was declined: a solo project with no PR-based review, where the agent runs bun run validate and the ship re-runs it with validate:build on the exact tree that reaches main (shipping.md).

AGENTS.md

The live file is the root AGENTS.md; the template that used to stand here was retired once it existed, so there is one copy.

Known Issues and Gotchas

Rejected Approaches

References

Official documentation

Community implementations

Issues and discussions

Architecture and patterns