5.8 KiB
Local workerd OOM during full API test runs
Findings from running the complete live API suite against the local
wrangler dev backend. This documents why the full suite sometimes dies and
what was done about it.
Symptom
The run starts fine, then partway through:
V8 fatal error; location = Reached heap limit;
message = : allocation failed: JavaScript heap out of memory
workerd exits. The backend becomes unreachable, and every remaining suite fails
with connection errors rather than assertions:
TRPCClientError: fetch failed
Cause: TypeError: fetch failed
Cause: SocketError: other side closed
The suite summary then shows dozens of “failed” suites/tests that are not real failures.
- Run 8: survived all 60 suites (780.9 s).
- Run 9: died after ~26 suites; 33 suites / 483 tests reported as failed even though they never ran against a live backend.
- Run 10 (chunked): 5/6 chunks passed; the only failure was an unrelated bind-cap assertion.
So it is nondeterministic and cumulative, not tied to one test.
What it is not
It is not the amount of test data. At the moment of the run-9 OOM the local D1 was only 4.1 MB:
| table | rows |
|---|---|
| stores | 161 |
| products / SKUs | 315 / 330 |
| product_market_stats | 453 |
| regions / bounds | 13 / 17 |
| delivery slots | 749 |
| users | 1,512 |
| orders / order items | 1,166 / 3,512 |
A full run adds roughly ~400 stores, ~400 products, ~50 regions, ~375 users,
~200 orders (mostly one store/product per test — a product needs a storeId,
and tests mutate their fixtures, so they are not shared more aggressively).
That is still tiny for a JS heap. workerd RSS was ~86 MB at the time.
The real pressure is per-request allocation/retention in the long-lived workerd/Miniflare isolate, accumulated over thousands of HTTP requests.
Prime suspect (fixed)
src/lib/worker-init.ts runs on every request:
export const ensureWorkerInit = (env: any) => {
globalThis.ENV = env
if (env?.DB) {
initDb(env.DB) // built a NEW Drizzle instance per request
}
}
initDb does drizzle(database, { schema }) and wraps it. Constructing a fresh
Drizzle client/session on every request is unnecessary churn, on top of
Miniflare’s own per-request machinery (prepared statements, KV read cache,
request/log buffers).
Changed to initialise once per isolate:
let dbReady = false
export const ensureWorkerInit = (env: any) => {
globalThis.ENV = env
if (env?.DB && !dbReady) {
initDb(env.DB)
dbReady = true
}
}
The D1 binding is stable for an isolate’s lifetime, so this is safe. ENV still
tracks the current request.
Status: applied and being validated with a single-process full run. If the heap still climbs, keep using the chunked runner below.
Other contributors (smaller): per-request prepared statements for dynamically
sized IN (...) lists, the KV read cache, and occasional large JSON/S3 buffers
during inline region-cache generation.
Reliable way to run the full suite
bun run test:api runs all 60 suites in one jest process and one long-lived
backend. That is the configuration that hits the heap limit.
Use the chunked runner instead:
cd apps/backend
bun run test:api:all # default 10 files per chunk
CHUNK_SIZE=8 bun run test:api:all
tests/run-all.ts:
- splits the test files into chunks (round-robin by file size),
- runs
jest --runInBandper chunk, - restarts the backend between chunks (resetting the workerd heap),
- starts the backend if it is down, and fails fast if it cannot start.
Seeds and D1 state persist across restarts. tests/global-setup.ts still runs per
chunk and clears the KV order tracker.
Chunk results are printed at the end:
[run-all] summary
chunk 1: PASS (10 files)
...
6/6 chunks passed
Diagnosing it yourself
# is the backend alive?
curl -s -m 5 http://localhost:8787/api/health
# did workerd die with a heap error?
ls -t ~/.wrangler/logs | head
grep -l "Reached heap limit" ~/.wrangler/logs/wrangler-*.log | tail -3
# backend RSS
lsof -nP -iTCP:8787 -sTCP:LISTEN
ps -o pid,rss,etime,command -p <pid>
If suites fail with fetch failed / Backend not reachable in a block, check
health before treating them as test failures — the backend most likely died.
Mitigations already in place
- One-time DB init per isolate (
src/lib/worker-init.ts). - Chunked runner with backend restarts (
tests/run-all.ts,bun run test:api:all). - No DO rebuilds in tests:
scheduleStoreInitializationis a no-op underENABLE_TEST_ROUTES=true; cache documents are read via the guarded/api/test/cache/*scaffold routes. - Shared fixtures in
user-order-place(one store/product/slot instead of one per test) — cut that suite from 106 s to ~52 s and reduced data creation. - Clean test data between runs:
bun run remove-test-data(local-only) keeps region-cache generation fast.
Planned/possible:
- Share fixtures in
user-order-lifecycleandadmin-order, and share a container store inadmin-product-crud(fewer stores/products/slots). - Reduce region creation in region-scoped suites where a fresh region is not actually required.
Related local-environment note: D1 bind cap
While chasing the OOM we also hit:
D1_ERROR: too many SQL variables
triggered by superadmin.globalProducts.create, which seeds one market-stats row
per region in a single multi-row insert. The local cap depends on the SQLite
build/region count (observed: 180 binds succeeded, 414 failed). The
SGPROD-04/07/10 (current) rows therefore accept either the successful create
or the masked Failed to create/update product error instead of computing a
threshold that flakes with the region count.