165 lines
7.2 KiB
Markdown
165 lines
7.2 KiB
Markdown
# Re-creating a Table Whose Keys Are Foreign Keys Elsewhere
|
|
|
|
> **Problem:** you need to rebuild a table (drop a column that participates in an
|
|
> FK, change nullability, etc. — anything SQLite's limited `ALTER TABLE` cannot
|
|
> do), but other tables hold foreign keys into it and the database has live rows.
|
|
> A naive `DROP TABLE` fails with `FOREIGN KEY constraint failed`, on D1 no
|
|
> matter which executor you use (dashboard, CLI, `migrations apply`).
|
|
>
|
|
> Last reviewed: 2026-10-05 (written after `0007_region_scoping.sql` forced us
|
|
> to rebuild a 13-table subgraph on D1).
|
|
|
|
## 1. Why the textbook pattern dies on D1
|
|
|
|
The standard SQLite table-rebuild recipe relies on switching enforcement off
|
|
around the rebuild:
|
|
|
|
```sql
|
|
PRAGMA foreign_keys = OFF; -- or PRAGMA defer_foreign_keys = ON;
|
|
-- ... create-copy-drop-rename ...
|
|
PRAGMA foreign_keys = ON;
|
|
```
|
|
|
|
D1 **enforces foreign keys and ignores both pragmas**. `DROP TABLE parent`
|
|
runs an implicit `DELETE FROM parent` first, which trips every live child row
|
|
(`ON DELETE no action`). Verified locally: the DROP fails with exit 19 while
|
|
the identical statements pass with enforcement off. There is no executor-side
|
|
workaround — dashboard, `d1 execute --file`, and `migrations apply` all fail
|
|
the same way. (Side effect: `scripts/clear-db.sql`'s `PRAGMA foreign_keys=OFF`
|
|
likewise cannot protect a full wipe on D1 remote.)
|
|
|
|
On an **empty** database the same migration passes (no child rows → the
|
|
implicit delete violates nothing). So this failure only appears on populated
|
|
DBs, which is why it slips past dev-time testing.
|
|
|
|
## 2. The pattern: rebuild the whole FK subgraph, leaves first
|
|
|
|
Since no referenced table with live rows can be dropped, rebuild **every table
|
|
in the connected subgraph** so that at each DROP, nothing references the table
|
|
being dropped. Every step is individually FK-legal, so per-statement autocommit
|
|
(D1 dashboard/CLI) is fine — no transaction wrapping needed.
|
|
|
|
### Step 0 — Map the full subgraph (not just direct children)
|
|
|
|
Audit `schema.ts` for every `references(() => T)` where `T` is in your set,
|
|
**recursively**. Grandchildren matter: in our case `coupon_usage.order_item_id
|
|
-> order_items(id)` meant `order_items` was not a leaf and `coupon_usage` had
|
|
to join the rebuild set. Also audit for anything the schema may not know about
|
|
(legacy tables, views, triggers) — see pre-flight below.
|
|
|
|
Classify each table:
|
|
|
|
- **Leaf** — nothing references it. Rebuild directly with the final schema
|
|
(NOT NULL preserved natively, no temp columns needed).
|
|
- **Mid-level** — referenced by other tables in the set. Build the replacement
|
|
under a `__new_<t>` name first; it gets dropped/renamed later once
|
|
unreferenced.
|
|
|
|
### Step 1 — Pre-flight on the live target (read-only, mandatory)
|
|
|
|
```sql
|
|
SELECT type, name FROM sqlite_master
|
|
WHERE (sql LIKE '%<child_table>%' OR sql LIKE '%<parent_table>%')
|
|
AND name NOT LIKE 'sqlite_%';
|
|
```
|
|
|
|
Repeat the `LIKE` for every table in the set. The result must show **only** the
|
|
known tables plus their indexes. Anything else (legacy table, view, trigger)
|
|
must be folded into the plan before proceeding — the migration cannot branch
|
|
at runtime.
|
|
|
|
Also confirm no index covers a column you plan to `DROP COLUMN` (SQLite blocks
|
|
dropping indexed columns; FK-child columns are droppable, PK/UNIQUE columns
|
|
are not).
|
|
|
|
### Step 2 — Create + copy, parents before children
|
|
|
|
```sql
|
|
CREATE TABLE `__new_parent` ( ... final schema ... );
|
|
INSERT INTO `__new_parent` (...) SELECT ... FROM `parent`;
|
|
CREATE TABLE `__new_child` (
|
|
...
|
|
FOREIGN KEY (`pid`) REFERENCES `__new_parent`(`id`) ON UPDATE no action ON DELETE no action
|
|
);
|
|
INSERT INTO `__new_child` (...) SELECT ... FROM `child`;
|
|
```
|
|
|
|
- Preserve explicit `id`s so existing cross-references stay valid.
|
|
- `INSERT` validates FKs immediately under autocommit, so fill parents first.
|
|
- Point `__new_` FKs at `__new_` names. SQLite resolves FK parents **by name
|
|
at enforcement time**, and `RENAME TABLE` (3.25+) rewrites other tables' FK
|
|
clauses to the new name — so after the renames everything points at the
|
|
final tables. (Verified: post-rename `sqlite_master` shows
|
|
`REFERENCES "parent"`, and `foreign_key_check` is clean.)
|
|
- Copy **all** columns explicitly; `sqlite_sequence` stays correct
|
|
automatically since ids are preserved.
|
|
|
|
### Step 3 — Drop originals, leaves first
|
|
|
|
```sql
|
|
DROP TABLE `leaf_1`;
|
|
DROP TABLE `leaf_2`;
|
|
-- ...
|
|
DROP TABLE `mid_child`; -- now unreferenced
|
|
DROP TABLE `parent`; -- now unreferenced
|
|
```
|
|
|
|
Each DROP succeeds because nothing references the table at that moment.
|
|
|
|
### Step 4 — Rename top-down, then recreate indexes
|
|
|
|
```sql
|
|
ALTER TABLE `__new_parent` RENAME TO `parent`;
|
|
ALTER TABLE `__new_child` RENAME TO `child`;
|
|
-- ...
|
|
CREATE UNIQUE INDEX `<original_name>` ON `<table>` (...);
|
|
```
|
|
|
|
Create indexes **after** the drops: index names live in a global namespace, so
|
|
reusing the original names before dropping the old tables collides. (Non-unique
|
|
indexes too — same namespace.)
|
|
|
|
### Step 5 — Verify
|
|
|
|
```sql
|
|
PRAGMA foreign_key_check; -- must return zero rows
|
|
-- plus per-table COUNT(*) compared against pre-migration counts
|
|
```
|
|
|
|
Also exercise post-rename writes once: an INSERT violating a CHECK / FK /
|
|
NOT NULL must still fail (proves constraints survived the rename — see
|
|
pitfalls).
|
|
|
|
## 3. Pitfalls (each one bit us)
|
|
|
|
1. **CHECK constraints are NOT rewritten by RENAME.** Only other tables' FK
|
|
clauses follow a rename. A drizzle-rendered
|
|
`CHECK("old_name"."col" ...)` dangles after promotion and breaks all
|
|
subsequent writes. Write CHECKs **unqualified**: `CHECK(col >= 1 AND ...)`.
|
|
2. **No `ALTER COLUMN` in SQLite** (syntax error). Nullability can only change
|
|
via rebuild — which is why mid-level tables go through `__new_` tables
|
|
instead of `ADD/DROP COLUMN` surgery. (`ADD ... REFERENCES ... NOT NULL
|
|
DEFAULT` is explicitly rejected by SQLite, so temp FK columns would be
|
|
stuck nullable — another reason to rebuild leaves directly.)
|
|
3. **`DROP COLUMN` works on FK-child columns** (verified) but is unnecessary in
|
|
this pattern — prefer full rebuilds so NOT NULL survives everywhere.
|
|
4. **Transcribe schemas from the migration target state, not from history.**
|
|
Old migration files (e.g. `0000`) show stale shapes (integer timestamps,
|
|
pre-rename columns). The `__new_` definitions must match post-migration
|
|
reality = current `schema.ts` (we cross-checked every column/index/default
|
|
against it).
|
|
5. **`NUMERIC` affinity** (`numericText`) has no precedent in our migration
|
|
history but is valid SQLite — safe to emit.
|
|
6. **Fix the audit, not just the symptom.** The `coupon_usage -> order_items`
|
|
edge was found by grepping all `references()` targets, not by trial and
|
|
error. One missed edge = same FK failure one level down.
|
|
|
|
## 4. Proof harness
|
|
|
|
`0007`'s chain was proven before shipping by extracting the migration block
|
|
verbatim and running it statement-by-statement in autocommit with
|
|
`PRAGMA foreign_keys=ON` against an in-memory DB with all tables populated,
|
|
asserting: row counts preserved, `foreign_key_check` clean, no `__new_`
|
|
leftovers, indexes present, and post-rename CHECK/FK/NOT NULL enforcement
|
|
alive. Reuse that approach for the next rebuild: populate → extract → run →
|
|
assert. (Scratch script used: `/tmp`-adjacent `proof_0007.py`; not committed.)
|