198 lines
5.8 KiB
Markdown
198 lines
5.8 KiB
Markdown
# Local `workerd` OOM during full API test runs
|
||
|
||
Findings from running the complete live API suite against the local
|
||
`wrangler dev` backend. This documents **why the full suite sometimes dies** and
|
||
what was done about it.
|
||
|
||
---
|
||
|
||
## Symptom
|
||
|
||
The run starts fine, then partway through:
|
||
|
||
```
|
||
V8 fatal error; location = Reached heap limit;
|
||
message = : allocation failed: JavaScript heap out of memory
|
||
```
|
||
|
||
`workerd` exits. The backend becomes unreachable, and every remaining suite fails
|
||
with connection errors rather than assertions:
|
||
|
||
```
|
||
TRPCClientError: fetch failed
|
||
Cause: TypeError: fetch failed
|
||
Cause: SocketError: other side closed
|
||
```
|
||
|
||
The suite summary then shows dozens of “failed” suites/tests that are **not**
|
||
real failures.
|
||
|
||
- Run 8: survived all 60 suites (780.9 s).
|
||
- Run 9: died after ~26 suites; 33 suites / 483 tests reported as failed even
|
||
though they never ran against a live backend.
|
||
- Run 10 (chunked): 5/6 chunks passed; the only failure was an unrelated bind-cap
|
||
assertion.
|
||
|
||
So it is **nondeterministic and cumulative**, not tied to one test.
|
||
|
||
---
|
||
|
||
## What it is *not*
|
||
|
||
It is not the amount of test data. At the moment of the run-9 OOM the local D1 was
|
||
only **4.1 MB**:
|
||
|
||
| table | rows |
|
||
| --- | --- |
|
||
| stores | 161 |
|
||
| products / SKUs | 315 / 330 |
|
||
| product_market_stats | 453 |
|
||
| regions / bounds | 13 / 17 |
|
||
| delivery slots | 749 |
|
||
| users | 1,512 |
|
||
| orders / order items | 1,166 / 3,512 |
|
||
|
||
A full run adds roughly **~400 stores, ~400 products, ~50 regions, ~375 users,
|
||
~200 orders** (mostly one store/product per test — a product needs a `storeId`,
|
||
and tests mutate their fixtures, so they are not shared more aggressively).
|
||
That is still tiny for a JS heap. `workerd` RSS was ~86 MB at the time.
|
||
|
||
The real pressure is **per-request allocation/retention in the long-lived
|
||
workerd/Miniflare isolate**, accumulated over thousands of HTTP requests.
|
||
|
||
---
|
||
|
||
## Prime suspect (fixed)
|
||
|
||
`src/lib/worker-init.ts` runs on **every request**:
|
||
|
||
```ts
|
||
export const ensureWorkerInit = (env: any) => {
|
||
globalThis.ENV = env
|
||
if (env?.DB) {
|
||
initDb(env.DB) // built a NEW Drizzle instance per request
|
||
}
|
||
}
|
||
```
|
||
|
||
`initDb` does `drizzle(database, { schema })` and wraps it. Constructing a fresh
|
||
Drizzle client/session on every request is unnecessary churn, on top of
|
||
Miniflare’s own per-request machinery (prepared statements, KV read cache,
|
||
request/log buffers).
|
||
|
||
Changed to initialise once per isolate:
|
||
|
||
```ts
|
||
let dbReady = false
|
||
|
||
export const ensureWorkerInit = (env: any) => {
|
||
globalThis.ENV = env
|
||
if (env?.DB && !dbReady) {
|
||
initDb(env.DB)
|
||
dbReady = true
|
||
}
|
||
}
|
||
```
|
||
|
||
The D1 binding is stable for an isolate’s lifetime, so this is safe. `ENV` still
|
||
tracks the current request.
|
||
|
||
> Status: applied and being validated with a single-process full run. If the heap
|
||
> still climbs, keep using the chunked runner below.
|
||
|
||
Other contributors (smaller): per-request prepared statements for dynamically
|
||
sized `IN (...)` lists, the KV read cache, and occasional large JSON/S3 buffers
|
||
during inline region-cache generation.
|
||
|
||
---
|
||
|
||
## Reliable way to run the full suite
|
||
|
||
`bun run test:api` runs all 60 suites in **one** jest process and one long-lived
|
||
backend. That is the configuration that hits the heap limit.
|
||
|
||
Use the chunked runner instead:
|
||
|
||
```bash
|
||
cd apps/backend
|
||
bun run test:api:all # default 10 files per chunk
|
||
CHUNK_SIZE=8 bun run test:api:all
|
||
```
|
||
|
||
`tests/run-all.ts`:
|
||
|
||
1. splits the test files into chunks (round-robin by file size),
|
||
2. runs `jest --runInBand` per chunk,
|
||
3. **restarts the backend between chunks** (resetting the workerd heap),
|
||
4. starts the backend if it is down, and fails fast if it cannot start.
|
||
|
||
Seeds and D1 state persist across restarts. `tests/global-setup.ts` still runs per
|
||
chunk and clears the KV order tracker.
|
||
|
||
Chunk results are printed at the end:
|
||
|
||
```
|
||
[run-all] summary
|
||
chunk 1: PASS (10 files)
|
||
...
|
||
6/6 chunks passed
|
||
```
|
||
|
||
---
|
||
|
||
## Diagnosing it yourself
|
||
|
||
```bash
|
||
# is the backend alive?
|
||
curl -s -m 5 http://localhost:8787/api/health
|
||
|
||
# did workerd die with a heap error?
|
||
ls -t ~/.wrangler/logs | head
|
||
grep -l "Reached heap limit" ~/.wrangler/logs/wrangler-*.log | tail -3
|
||
|
||
# backend RSS
|
||
lsof -nP -iTCP:8787 -sTCP:LISTEN
|
||
ps -o pid,rss,etime,command -p <pid>
|
||
```
|
||
|
||
If suites fail with `fetch failed` / `Backend not reachable` in a block, check
|
||
health before treating them as test failures — the backend most likely died.
|
||
|
||
---
|
||
|
||
## Mitigations already in place
|
||
|
||
- **One-time DB init per isolate** (`src/lib/worker-init.ts`).
|
||
- **Chunked runner with backend restarts** (`tests/run-all.ts`,
|
||
`bun run test:api:all`).
|
||
- **No DO rebuilds in tests**: `scheduleStoreInitialization` is a no-op under
|
||
`ENABLE_TEST_ROUTES=true`; cache documents are read via the guarded
|
||
`/api/test/cache/*` scaffold routes.
|
||
- **Shared fixtures** in `user-order-place` (one store/product/slot instead of
|
||
one per test) — cut that suite from 106 s to ~52 s and reduced data creation.
|
||
- **Clean test data** between runs: `bun run remove-test-data` (local-only) keeps
|
||
region-cache generation fast.
|
||
|
||
Planned/possible:
|
||
|
||
- Share fixtures in `user-order-lifecycle` and `admin-order`, and share a
|
||
container store in `admin-product-crud` (fewer stores/products/slots).
|
||
- Reduce region creation in region-scoped suites where a fresh region is not
|
||
actually required.
|
||
|
||
---
|
||
|
||
## Related local-environment note: D1 bind cap
|
||
|
||
While chasing the OOM we also hit:
|
||
|
||
```
|
||
D1_ERROR: too many SQL variables
|
||
```
|
||
|
||
triggered by `superadmin.globalProducts.create`, which seeds one market-stats row
|
||
per region in a single multi-row insert. The local cap depends on the SQLite
|
||
build/region count (observed: **180 binds succeeded, 414 failed**). The
|
||
`SGPROD-04/07/10` `(current)` rows therefore accept either the successful create
|
||
or the masked `Failed to create/update product` error instead of computing a
|
||
threshold that flakes with the region count.
|