freshyo/apps/backend/tests/OOM-FINDINGS.md
2026-10-20 00:31:05 +05:30

198 lines
5.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Local `workerd` OOM during full API test runs
Findings from running the complete live API suite against the local
`wrangler dev` backend. This documents **why the full suite sometimes dies** and
what was done about it.
---
## Symptom
The run starts fine, then partway through:
```
V8 fatal error; location = Reached heap limit;
message = : allocation failed: JavaScript heap out of memory
```
`workerd` exits. The backend becomes unreachable, and every remaining suite fails
with connection errors rather than assertions:
```
TRPCClientError: fetch failed
Cause: TypeError: fetch failed
Cause: SocketError: other side closed
```
The suite summary then shows dozens of “failed” suites/tests that are **not**
real failures.
- Run 8: survived all 60 suites (780.9 s).
- Run 9: died after ~26 suites; 33 suites / 483 tests reported as failed even
though they never ran against a live backend.
- Run 10 (chunked): 5/6 chunks passed; the only failure was an unrelated bind-cap
assertion.
So it is **nondeterministic and cumulative**, not tied to one test.
---
## What it is *not*
It is not the amount of test data. At the moment of the run-9 OOM the local D1 was
only **4.1 MB**:
| table | rows |
| --- | --- |
| stores | 161 |
| products / SKUs | 315 / 330 |
| product_market_stats | 453 |
| regions / bounds | 13 / 17 |
| delivery slots | 749 |
| users | 1,512 |
| orders / order items | 1,166 / 3,512 |
A full run adds roughly **~400 stores, ~400 products, ~50 regions, ~375 users,
~200 orders** (mostly one store/product per test — a product needs a `storeId`,
and tests mutate their fixtures, so they are not shared more aggressively).
That is still tiny for a JS heap. `workerd` RSS was ~86 MB at the time.
The real pressure is **per-request allocation/retention in the long-lived
workerd/Miniflare isolate**, accumulated over thousands of HTTP requests.
---
## Prime suspect (fixed)
`src/lib/worker-init.ts` runs on **every request**:
```ts
export const ensureWorkerInit = (env: any) => {
globalThis.ENV = env
if (env?.DB) {
initDb(env.DB) // built a NEW Drizzle instance per request
}
}
```
`initDb` does `drizzle(database, { schema })` and wraps it. Constructing a fresh
Drizzle client/session on every request is unnecessary churn, on top of
Miniflare’s own per-request machinery (prepared statements, KV read cache,
request/log buffers).
Changed to initialise once per isolate:
```ts
let dbReady = false
export const ensureWorkerInit = (env: any) => {
globalThis.ENV = env
if (env?.DB && !dbReady) {
initDb(env.DB)
dbReady = true
}
}
```
The D1 binding is stable for an isolate’s lifetime, so this is safe. `ENV` still
tracks the current request.
> Status: applied and being validated with a single-process full run. If the heap
> still climbs, keep using the chunked runner below.
Other contributors (smaller): per-request prepared statements for dynamically
sized `IN (...)` lists, the KV read cache, and occasional large JSON/S3 buffers
during inline region-cache generation.
---
## Reliable way to run the full suite
`bun run test:api` runs all 60 suites in **one** jest process and one long-lived
backend. That is the configuration that hits the heap limit.
Use the chunked runner instead:
```bash
cd apps/backend
bun run test:api:all # default 10 files per chunk
CHUNK_SIZE=8 bun run test:api:all
```
`tests/run-all.ts`:
1. splits the test files into chunks (round-robin by file size),
2. runs `jest --runInBand` per chunk,
3. **restarts the backend between chunks** (resetting the workerd heap),
4. starts the backend if it is down, and fails fast if it cannot start.
Seeds and D1 state persist across restarts. `tests/global-setup.ts` still runs per
chunk and clears the KV order tracker.
Chunk results are printed at the end:
```
[run-all] summary
chunk 1: PASS (10 files)
...
6/6 chunks passed
```
---
## Diagnosing it yourself
```bash
# is the backend alive?
curl -s -m 5 http://localhost:8787/api/health
# did workerd die with a heap error?
ls -t ~/.wrangler/logs | head
grep -l "Reached heap limit" ~/.wrangler/logs/wrangler-*.log | tail -3
# backend RSS
lsof -nP -iTCP:8787 -sTCP:LISTEN
ps -o pid,rss,etime,command -p <pid>
```
If suites fail with `fetch failed` / `Backend not reachable` in a block, check
health before treating them as test failures — the backend most likely died.
---
## Mitigations already in place
- **One-time DB init per isolate** (`src/lib/worker-init.ts`).
- **Chunked runner with backend restarts** (`tests/run-all.ts`,
`bun run test:api:all`).
- **No DO rebuilds in tests**: `scheduleStoreInitialization` is a no-op under
`ENABLE_TEST_ROUTES=true`; cache documents are read via the guarded
`/api/test/cache/*` scaffold routes.
- **Shared fixtures** in `user-order-place` (one store/product/slot instead of
one per test) — cut that suite from 106 s to ~52 s and reduced data creation.
- **Clean test data** between runs: `bun run remove-test-data` (local-only) keeps
region-cache generation fast.
Planned/possible:
- Share fixtures in `user-order-lifecycle` and `admin-order`, and share a
container store in `admin-product-crud` (fewer stores/products/slots).
- Reduce region creation in region-scoped suites where a fresh region is not
actually required.
---
## Related local-environment note: D1 bind cap
While chasing the OOM we also hit:
```
D1_ERROR: too many SQL variables
```
triggered by `superadmin.globalProducts.create`, which seeds one market-stats row
per region in a single multi-row insert. The local cap depends on the SQLite
build/region count (observed: **180 binds succeeded, 414 failed**). The
`SGPROD-04/07/10` `(current)` rows therefore accept either the successful create
or the masked `Failed to create/update product` error instead of computing a
threshold that flakes with the region count.