---
title: Tenant isolation and fleet lifecycle
description: Control-plane routing, database-per-tenant isolation, provisioning, fleet migrations, and failure handling.
---

Basenet uses one control-plane database plus one physical PostgreSQL database per tenant. A host, token route ID, or job reference selects a candidate tenant through the control plane; tenant-local credential validation authorizes access. Routing alone never grants access.

## Runtime isolation boundary

```mermaid
flowchart LR
  signal[Trusted host / structured token / job tenant ref]
  router[Control-plane TenantRouter]
  identity[TenantIdentity routeId + slug + status]
  active{status is active?}
  validator[Tenant-local credential validation]
  handle[openTenant slug]
  database[(blinqx_tenant_slug)]
  resources[Slug-keyed search, storage, realm and providers]
  reject[Fail closed]

  signal --> router --> identity --> active
  active -->|no| reject
  active -->|yes| validator
  validator -->|invalid| reject
  validator -->|valid actor| handle --> database
  handle --> resources
```

Caption: `packages/tenant-runtime/src/context.ts` enforces route → active-status check → tenant-local credential validation. `packages/db/src/client.ts` derives a physical database name only from a validated slug. Worker jobs repeat the control-plane route and active check in `apps/workers/src/tenant.ts` instead of trusting payload slugs.

### Isolation invariants

- No default-tenant fallback exists at a trusted request or job boundary.
- A route ID is non-secret routing metadata, not proof of authorization.
- An inactive, missing, or mismatched tenant fails before its database is opened for the use case.
- `openTenant()` returns a handle whose table map and connection point at exactly one physical tenant database.
- Tenant-scoped external resources use the validated slug/route, never a caller-provided storage prefix or index name.
- Linked worktrees require distinct environment ports and databases; follow [Parallel worktrees](/developers/operations#development-and-test-environments).

## Provisioning state machine

```mermaid
stateDiagram-v2
  [*] --> provisioning: beginProvisioning
  provisioning --> database
  database --> identityRealm
  identityRealm --> storagePrefix
  storagePrefix --> searchIndex
  searchIndex --> seed
  seed --> identityLink: real IdP only
  seed --> residency: mock IdP
  identityLink --> residency
  residency --> entitlements
  entitlements --> cfRoute: CF adapter supplied
  entitlements --> active: no CF adapter
  cfRoute --> active
  active --> done

  database --> failed: step throws
  identityRealm --> failed: step throws
  storagePrefix --> failed: step throws
  searchIndex --> failed: step throws
  seed --> failed: step throws
  identityLink --> failed: step throws
  residency --> failed: step throws
  entitlements --> failed: step throws
  cfRoute --> failed: step throws
```

`beginProvisioning()` creates the control-plane tenant and job rows synchronously. `runProvisioning()` records each step, stops at the first failure, marks both job and tenant `failed`, and activates only after all required steps complete. The CLI runs begin + run synchronously; the admin flow may enqueue the same payload for `apps/workers/src/jobs/tenant-provision.ts`.

Current source includes these steps: tenant database + migrations, identity realm/client, storage marker, search index, seed source, optional real-IdP staff linking, residency records, matter-variant entitlements, optional Cloud Foundry route, then activation. A failed run can leave physical resources from completed steps; the current runbook directs operators to use a new slug or explicitly clean up the partial tenant rather than hand-editing control-plane state.

## Fleet migrations

```mermaid
flowchart TD
  select[Select control-plane tenants or explicit slugs]
  next{Next tenant?}
  open[Open that tenant DB]
  migrate[migrateUp pending files]
  close[Close driver]
  result[Record per-tenant ok / failed]
  threshold{3 failures reached?}
  skipped[Mark remaining targets skipped]
  done[Return results, failures, halted]

  select --> next
  next -->|yes| open --> migrate
  migrate -->|success| close --> result --> next
  migrate -->|failure| close --> result --> threshold
  threshold -->|no| next
  threshold -->|yes| skipped --> done
  next -->|no| done
```

The fleet runner is intentionally sequential for predictable shared-cluster load. Each tenant opens and closes its own driver; failure in one database does not roll another tenant back. The default halt threshold is three failures, after which unstarted targets are reported as `skipped`. `rollbackFleet()` uses the same fleet semantics, while `migrateDown()` performs the selected rollback range atomically within one tenant.

Do not rename a migration after it has run: the filename ID is also the `_migrations` ledger identity. Compatibility checks must include populated pre-migration tenants, not only fresh databases.

## Failure modes

| Failure | Boundary and operator signal | Source anchor |
| --- | --- | --- |
| Host/token/job routes to nothing | `no_route` or worker routing error; no fallback tenant | `packages/tenant-runtime/src/context.ts`, `apps/workers/src/tenant.ts` |
| Tenant is provisioning, suspended, failed, or deprovisioned | Rejected before tenant-local use | `packages/tenant-runtime/src/context.ts` |
| Provisioning step fails | Job and tenant become `failed`; later steps and activation do not run | `apps/control-plane/src/provisioning.ts` |
| Migration fails for one tenant | That result is `failed`; later tenants continue until threshold | `packages/db/src/migrate/index.ts` |
| Three fleet targets fail | Remaining targets are `skipped`, run reports `halted` | `packages/db/src/migrate/index.ts` |
| Down migration rejects live data | Tenant rollback transaction aborts and retains the newer schema/ledger | `packages/db/src/migrate/index.ts` |

## Focused validation

```bash
pnpm --filter @blinqx/tenant-runtime test -- --maxWorkers=2
pnpm --filter @blinqx/control-plane test -- --maxWorkers=2
pnpm --filter @blinqx/db test -- --maxWorkers=2
pnpm test:fleet
```

`pnpm test:fleet` is the Docker/testcontainers path and should run in the remote/CI lane when local resources are constrained.

## Source anchors

- Routing and validation: `packages/tenant-runtime/src/context.ts`, `packages/tenant-runtime/src/router.ts`, `packages/tenant-runtime/src/host.ts`
- Database boundary: `packages/db/src/client.ts`, `packages/db/src/schema/control-plane.ts`, `packages/db/src/schema/tenant.ts`
- Provisioning: `apps/control-plane/src/provisioning.ts`, `apps/workers/src/jobs/tenant-provision.ts`
- Fleet runner: `apps/control-plane/src/cli.ts`, `packages/db/src/migrate/index.ts`
- Proof: `packages/tenant-runtime/src/context.test.ts`, `apps/control-plane/src/provisioning.test.ts`, `packages/db/src/migrate/migrate.test.ts`, `apps/control-plane/src/isolation-load.pg.test.ts`

See also: [Operations and runbooks](/developers/operations) · [Real-tenant fixtures](/developers/tenant-fixtures) · [Testing](/developers/testing)