freshyo/apps/backend/tests/OOM-FINDINGS.md
2026-10-20 00:31:05 +05:30

5.8 KiB
Raw Permalink Blame History

Local workerd OOM during full API test runs

Findings from running the complete live API suite against the local wrangler dev backend. This documents why the full suite sometimes dies and what was done about it.


Symptom

The run starts fine, then partway through:

V8 fatal error; location = Reached heap limit;
message = : allocation failed: JavaScript heap out of memory

workerd exits. The backend becomes unreachable, and every remaining suite fails with connection errors rather than assertions:

TRPCClientError: fetch failed
  Cause: TypeError: fetch failed
  Cause: SocketError: other side closed

The suite summary then shows dozens of “failed” suites/tests that are not real failures.

  • Run 8: survived all 60 suites (780.9 s).
  • Run 9: died after ~26 suites; 33 suites / 483 tests reported as failed even though they never ran against a live backend.
  • Run 10 (chunked): 5/6 chunks passed; the only failure was an unrelated bind-cap assertion.

So it is nondeterministic and cumulative, not tied to one test.


What it is not

It is not the amount of test data. At the moment of the run-9 OOM the local D1 was only 4.1 MB:

table rows
stores 161
products / SKUs 315 / 330
product_market_stats 453
regions / bounds 13 / 17
delivery slots 749
users 1,512
orders / order items 1,166 / 3,512

A full run adds roughly ~400 stores, ~400 products, ~50 regions, ~375 users, ~200 orders (mostly one store/product per test — a product needs a storeId, and tests mutate their fixtures, so they are not shared more aggressively). That is still tiny for a JS heap. workerd RSS was ~86 MB at the time.

The real pressure is per-request allocation/retention in the long-lived workerd/Miniflare isolate, accumulated over thousands of HTTP requests.


Prime suspect (fixed)

src/lib/worker-init.ts runs on every request:

export const ensureWorkerInit = (env: any) => {
  globalThis.ENV = env
  if (env?.DB) {
    initDb(env.DB) // built a NEW Drizzle instance per request
  }
}

initDb does drizzle(database, { schema }) and wraps it. Constructing a fresh Drizzle client/session on every request is unnecessary churn, on top of Miniflare’s own per-request machinery (prepared statements, KV read cache, request/log buffers).

Changed to initialise once per isolate:

let dbReady = false

export const ensureWorkerInit = (env: any) => {
  globalThis.ENV = env
  if (env?.DB && !dbReady) {
    initDb(env.DB)
    dbReady = true
  }
}

The D1 binding is stable for an isolate’s lifetime, so this is safe. ENV still tracks the current request.

Status: applied and being validated with a single-process full run. If the heap still climbs, keep using the chunked runner below.

Other contributors (smaller): per-request prepared statements for dynamically sized IN (...) lists, the KV read cache, and occasional large JSON/S3 buffers during inline region-cache generation.


Reliable way to run the full suite

bun run test:api runs all 60 suites in one jest process and one long-lived backend. That is the configuration that hits the heap limit.

Use the chunked runner instead:

cd apps/backend
bun run test:api:all            # default 10 files per chunk
CHUNK_SIZE=8 bun run test:api:all

tests/run-all.ts:

  1. splits the test files into chunks (round-robin by file size),
  2. runs jest --runInBand per chunk,
  3. restarts the backend between chunks (resetting the workerd heap),
  4. starts the backend if it is down, and fails fast if it cannot start.

Seeds and D1 state persist across restarts. tests/global-setup.ts still runs per chunk and clears the KV order tracker.

Chunk results are printed at the end:

[run-all] summary
  chunk 1: PASS (10 files)
  ...
  6/6 chunks passed

Diagnosing it yourself

# is the backend alive?
curl -s -m 5 http://localhost:8787/api/health

# did workerd die with a heap error?
ls -t ~/.wrangler/logs | head
grep -l "Reached heap limit" ~/.wrangler/logs/wrangler-*.log | tail -3

# backend RSS
lsof -nP -iTCP:8787 -sTCP:LISTEN
ps -o pid,rss,etime,command -p <pid>

If suites fail with fetch failed / Backend not reachable in a block, check health before treating them as test failures — the backend most likely died.


Mitigations already in place

  • One-time DB init per isolate (src/lib/worker-init.ts).
  • Chunked runner with backend restarts (tests/run-all.ts, bun run test:api:all).
  • No DO rebuilds in tests: scheduleStoreInitialization is a no-op under ENABLE_TEST_ROUTES=true; cache documents are read via the guarded /api/test/cache/* scaffold routes.
  • Shared fixtures in user-order-place (one store/product/slot instead of one per test) — cut that suite from 106 s to ~52 s and reduced data creation.
  • Clean test data between runs: bun run remove-test-data (local-only) keeps region-cache generation fast.

Planned/possible:

  • Share fixtures in user-order-lifecycle and admin-order, and share a container store in admin-product-crud (fewer stores/products/slots).
  • Reduce region creation in region-scoped suites where a fresh region is not actually required.

While chasing the OOM we also hit:

D1_ERROR: too many SQL variables

triggered by superadmin.globalProducts.create, which seeds one market-stats row per region in a single multi-row insert. The local cap depends on the SQLite build/region count (observed: 180 binds succeeded, 414 failed). The SGPROD-04/07/10 (current) rows therefore accept either the successful create or the masked Failed to create/update product error instead of computing a threshold that flakes with the region count.