Production learnings (running log)

Durable notes from production incidents and fixes. Each entry captures symptoms, root cause, what we changed, and how to avoid repeating the mistake. Operational how-to stays in BACKEND_OPERATIONS.md; high-level architecture will live in ARCHITECTURE.md when added.

How to add an entry: append a dated section, link the PR, keep secrets out of this file.

---

2026-10-19 — Vercel + Supabase: session pool exhaustion, migration stampede, static 500s

PR: #42 (merged 2026-10-19)

Symptoms

/api/health → 503, Explore and saves failing with generic 500s.
Supabase / logs: EMAXCONNSESSION — max clients reached in session mode — pool_size: 15.
Also: Postgres migration failed: canceling statement due to statement timeout, CONNECT_TIMEOUT to pooler.supabase.com:5432.
Even static assets (/wall.js, /remote.js) returned 500 during bad periods (when DB path was wedged or session ran before static handling).

Root cause (three layers)

1. Wrong pool mode for serverless runtime The app rewrote Supabase transaction pool URLs (:6543) to session pool (:5432). Each warm Vercel instance holds one postgres.js client (max: 1). Session mode caps concurrent backends (~15 on our tier). Traffic and cold starts → session pool exhausted → API 500s (including PUT /api/boards/..., uploads, explore).

2. Migration stampede Every cold start ran migrations inside a transaction with pg_advisory_xact_lock. Many parallel instances queued → statement timeout before any request could succeed.

3. Static traffic coupled to DB Catch-all routing sent all URLs through Node; session lookup ran before serving public/ files, so a sick DB could break JS loads.

What we changed

Area	Fix
Runtime URL (Vercel)	Use transaction pooler `:6543` for all API/query traffic (`resolveDatabaseUrl`).
Migrations	Separate short-lived session URL (`:5432` on pooler host, or `POSTGRES_URL_NON_POOLING`) only in `PostgresStore.migrate()`, then close.
postgres.js	One module-level store per instance: `max: 1`, `prepare: false`, shorter idle/`max_lifetime`.
Migrations concurrency	Fast path if `schema_migrations` complete; `pg_try_advisory_lock` + poll instead of blocking the whole fleet.
HTTP	Serve allowlisted `public/` (and Studio) before `accounts.current()`; `vercel.json` adds `{ "handle": "filesystem" }` for CDN static.

Vercel env (what we learned)

Keep integration defaults: POSTGRES_URL (transaction, 6543) + POSTGRES_URL_NON_POOLING for migration fallback.
Do not point Production `DATABASE_URL` at session :5432 for “fixing” DB unless you understand runtime vs migration split — runtime should stay on 6543 on Vercel.
Extra keys (POSTGRES_PRISMA_URL, POSTGRES_USER, …) are unused by this app; `POSTGRES_URL` is the source of truth.

Verification

GET /api/health → {"ok":true,"store":"postgres","durable":true}.
Explore loads, board save/autosave, image upload — confirmed in production after deploy.

Do not repeat

Treating session pool (5432) as the default for serverless API traffic on Supabase.
Running DDL migrations on every request path without a fast path and lock strategy.
Assuming auth works therefore Postgres is fine — session endpoint does not need DB; saves/uploads do.

Related docs

BACKEND_OPERATIONS.md — deploy contract, env vars, Output Directory.
SUPABASE_AUTH.md — auth URLs and integration keys.

---

2026-10-25 — Stale immutable wall.js (catalog sprites + Browse backgrounds)

PRs: #59 (core bust) · #60 (media cache + catalog contrast)

Symptoms

Production Icon Catalog showed broken sprite previews and old Bold labels after Option B.
Browse backgrounds looked dead on production for the same reason.
Fresh curls of /assets/... and current /wall.js looked fine on the deployment.

Root cause

servePublicFile set Cache-Control: public, max-age=31536000, immutable on all non-HTML public files, including unhashed /wall.js and /styles.css. Browsers kept a year-old JS build while HTML stayed no-store.

What we changed

Area	Fix
JS/CSS	`public, max-age=0, must-revalidate` (+ `vercel.json` headers)
HTML	Rewrite local script/link tags with `?v=<assetVersion>` (`release.assetVersion`)
Other static	`public, max-age=3600, must-revalidate` (no `immutable` on redrawn `/assets` sprites)
Catalog UI	Light label text on dark catalog cards

Do not repeat

Marking unhashed app shells (/wall.js, /styles.css) as immutable.
Assuming a production screenshot of the catalog matches the latest deploy without checking the browser’s cached wall.js.

---

2026-10-20 — Studio board images 404; fragile sprite URLs

PR: (this fix branch)

Symptoms

/studio “Walls and boards” JPGs broken on production; live soft /assets/objects/smoke-bold/*.png often OK.
/studio-library/docs/... returned 404 on production.
Catalog soft sprites could break when the page URL was /wall if relative ./assets/... resolved incorrectly.

Root cause

Studio CDN files are built into gitignored public/studio-library/. Production did not reliably expose that tree as static files, so CDN redirects failed.
Serverless includeFiles listed public/** + server/** but not docs/** / marketing/** / screenshots/**, so /studio/file/... could not read those trees from the function FS either.
Sprite URLs used ./assets/... instead of root-absolute /assets/....

What we changed

Area	Fix
Vercel bundle	Add `docs/`, `marketing/`, `screenshots/**` to serverless `includeFiles` so `/studio/file/...` works without CDN.
Sprite URLs	Absolute `/assets/objects/...` in `wall.js` and `validation.js`.

Do not repeat

Depending only on a gitignored build output (public/studio-library/) for production Studio media.
Using page-relative asset paths for a multi-route app (/, /wall, /studio).

---