# Local `workerd` OOM during full API test runs Findings from running the complete live API suite against the local `wrangler dev` backend. This documents **why the full suite sometimes dies** and what was done about it. --- ## Symptom The run starts fine, then partway through: ``` V8 fatal error; location = Reached heap limit; message = : allocation failed: JavaScript heap out of memory ``` `workerd` exits. The backend becomes unreachable, and every remaining suite fails with connection errors rather than assertions: ``` TRPCClientError: fetch failed Cause: TypeError: fetch failed Cause: SocketError: other side closed ``` The suite summary then shows dozens of “failed” suites/tests that are **not** real failures. - Run 8: survived all 60 suites (780.9 s). - Run 9: died after ~26 suites; 33 suites / 483 tests reported as failed even though they never ran against a live backend. - Run 10 (chunked): 5/6 chunks passed; the only failure was an unrelated bind-cap assertion. So it is **nondeterministic and cumulative**, not tied to one test. --- ## What it is *not* It is not the amount of test data. At the moment of the run-9 OOM the local D1 was only **4.1 MB**: | table | rows | | --- | --- | | stores | 161 | | products / SKUs | 315 / 330 | | product_market_stats | 453 | | regions / bounds | 13 / 17 | | delivery slots | 749 | | users | 1,512 | | orders / order items | 1,166 / 3,512 | A full run adds roughly **~400 stores, ~400 products, ~50 regions, ~375 users, ~200 orders** (mostly one store/product per test — a product needs a `storeId`, and tests mutate their fixtures, so they are not shared more aggressively). That is still tiny for a JS heap. `workerd` RSS was ~86 MB at the time. The real pressure is **per-request allocation/retention in the long-lived workerd/Miniflare isolate**, accumulated over thousands of HTTP requests. --- ## Prime suspect (fixed) `src/lib/worker-init.ts` runs on **every request**: ```ts export const ensureWorkerInit = (env: any) => { globalThis.ENV = env if (env?.DB) { initDb(env.DB) // built a NEW Drizzle instance per request } } ``` `initDb` does `drizzle(database, { schema })` and wraps it. Constructing a fresh Drizzle client/session on every request is unnecessary churn, on top of Miniflare’s own per-request machinery (prepared statements, KV read cache, request/log buffers). Changed to initialise once per isolate: ```ts let dbReady = false export const ensureWorkerInit = (env: any) => { globalThis.ENV = env if (env?.DB && !dbReady) { initDb(env.DB) dbReady = true } } ``` The D1 binding is stable for an isolate’s lifetime, so this is safe. `ENV` still tracks the current request. > Status: applied and being validated with a single-process full run. If the heap > still climbs, keep using the chunked runner below. Other contributors (smaller): per-request prepared statements for dynamically sized `IN (...)` lists, the KV read cache, and occasional large JSON/S3 buffers during inline region-cache generation. --- ## Reliable way to run the full suite `bun run test:api` runs all 60 suites in **one** jest process and one long-lived backend. That is the configuration that hits the heap limit. Use the chunked runner instead: ```bash cd apps/backend bun run test:api:all # default 10 files per chunk CHUNK_SIZE=8 bun run test:api:all ``` `tests/run-all.ts`: 1. splits the test files into chunks (round-robin by file size), 2. runs `jest --runInBand` per chunk, 3. **restarts the backend between chunks** (resetting the workerd heap), 4. starts the backend if it is down, and fails fast if it cannot start. Seeds and D1 state persist across restarts. `tests/global-setup.ts` still runs per chunk and clears the KV order tracker. Chunk results are printed at the end: ``` [run-all] summary chunk 1: PASS (10 files) ... 6/6 chunks passed ``` --- ## Diagnosing it yourself ```bash # is the backend alive? curl -s -m 5 http://localhost:8787/api/health # did workerd die with a heap error? ls -t ~/.wrangler/logs | head grep -l "Reached heap limit" ~/.wrangler/logs/wrangler-*.log | tail -3 # backend RSS lsof -nP -iTCP:8787 -sTCP:LISTEN ps -o pid,rss,etime,command -p ``` If suites fail with `fetch failed` / `Backend not reachable` in a block, check health before treating them as test failures — the backend most likely died. --- ## Mitigations already in place - **One-time DB init per isolate** (`src/lib/worker-init.ts`). - **Chunked runner with backend restarts** (`tests/run-all.ts`, `bun run test:api:all`). - **No DO rebuilds in tests**: `scheduleStoreInitialization` is a no-op under `ENABLE_TEST_ROUTES=true`; cache documents are read via the guarded `/api/test/cache/*` scaffold routes. - **Shared fixtures** in `user-order-place` (one store/product/slot instead of one per test) — cut that suite from 106 s to ~52 s and reduced data creation. - **Clean test data** between runs: `bun run remove-test-data` (local-only) keeps region-cache generation fast. Planned/possible: - Share fixtures in `user-order-lifecycle` and `admin-order`, and share a container store in `admin-product-crud` (fewer stores/products/slots). - Reduce region creation in region-scoped suites where a fresh region is not actually required. --- ## Related local-environment note: D1 bind cap While chasing the OOM we also hit: ``` D1_ERROR: too many SQL variables ``` triggered by `superadmin.globalProducts.create`, which seeds one market-stats row per region in a single multi-row insert. The local cap depends on the SQLite build/region count (observed: **180 binds succeeded, 414 failed**). The `SGPROD-04/07/10` `(current)` rows therefore accept either the successful create or the masked `Failed to create/update product` error instead of computing a threshold that flakes with the region count.