Tenant isolation and fleet lifecycle

Control-plane routing, database-per-tenant isolation, provisioning, fleet migrations, and failure handling.

Basenet uses one control-plane database plus one physical PostgreSQL database per tenant. A host, token route ID, or job reference selects a candidate tenant through the control plane; tenant-local credential validation authorizes access. Routing alone never grants access.

Runtime isolation boundary

Caption: packages/tenant-runtime/src/context.ts enforces route → active-status check → tenant-local credential validation. packages/db/src/client.ts derives a physical database name only from a validated slug. Worker jobs repeat the control-plane route and active check in apps/workers/src/tenant.ts instead of trusting payload slugs.

Isolation invariants

No default-tenant fallback exists at a trusted request or job boundary.
A route ID is non-secret routing metadata, not proof of authorization.
An inactive, missing, or mismatched tenant fails before its database is opened for the use case.
openTenant() returns a handle whose table map and connection point at exactly one physical tenant database.
Tenant-scoped external resources use the validated slug/route, never a caller-provided storage prefix or index name.
Linked worktrees require distinct environment ports and databases; follow Parallel worktrees.

Provisioning state machine

beginProvisioning() creates the control-plane tenant and job rows synchronously. runProvisioning() records each step, stops at the first failure, marks both job and tenant failed, and activates only after all required steps complete. The CLI runs begin + run synchronously; the admin flow may enqueue the same payload for apps/workers/src/jobs/tenant-provision.ts.

Current source includes these steps: tenant database + migrations, identity realm/client, storage marker, search index, seed source, optional real-IdP staff linking, residency records, matter-variant entitlements, optional Cloud Foundry route, then activation. A failed run can leave physical resources from completed steps; the current runbook directs operators to use a new slug or explicitly clean up the partial tenant rather than hand-editing control-plane state.

Fleet migrations

The fleet runner is intentionally sequential for predictable shared-cluster load. Each tenant opens and closes its own driver; failure in one database does not roll another tenant back. The default halt threshold is three failures, after which unstarted targets are reported as skipped. rollbackFleet() uses the same fleet semantics, while migrateDown() performs the selected rollback range atomically within one tenant.

Do not rename a migration after it has run: the filename ID is also the _migrations ledger identity. Compatibility checks must include populated pre-migration tenants, not only fresh databases.

Failure modes

Failure	Boundary and operator signal	Source anchor
Host/token/job routes to nothing	`no_route` or worker routing error; no fallback tenant	`packages/tenant-runtime/src/context.ts`, `apps/workers/src/tenant.ts`
Tenant is provisioning, suspended, failed, or deprovisioned	Rejected before tenant-local use	`packages/tenant-runtime/src/context.ts`
Provisioning step fails	Job and tenant become `failed`; later steps and activation do not run	`apps/control-plane/src/provisioning.ts`
Migration fails for one tenant	That result is `failed`; later tenants continue until threshold	`packages/db/src/migrate/index.ts`
Three fleet targets fail	Remaining targets are `skipped`, run reports `halted`	`packages/db/src/migrate/index.ts`
Down migration rejects live data	Tenant rollback transaction aborts and retains the newer schema/ledger	`packages/db/src/migrate/index.ts`

Focused validation

pnpm --filter @blinqx/tenant-runtime test -- --maxWorkers=2
pnpm --filter @blinqx/control-plane test -- --maxWorkers=2
pnpm --filter @blinqx/db test -- --maxWorkers=2
pnpm test:fleet

pnpm test:fleet is the Docker/testcontainers path and should run in the remote/CI lane when local resources are constrained.

Source anchors

Routing and validation: packages/tenant-runtime/src/context.ts, packages/tenant-runtime/src/router.ts, packages/tenant-runtime/src/host.ts
Database boundary: packages/db/src/client.ts, packages/db/src/schema/control-plane.ts, packages/db/src/schema/tenant.ts
Provisioning: apps/control-plane/src/provisioning.ts, apps/workers/src/jobs/tenant-provision.ts
Fleet runner: apps/control-plane/src/cli.ts, packages/db/src/migrate/index.ts
Proof: packages/tenant-runtime/src/context.test.ts, apps/control-plane/src/provisioning.test.ts, packages/db/src/migrate/migrate.test.ts, apps/control-plane/src/isolation-load.pg.test.ts