Workers and integrations

Background-job registry, queue semantics, tenant routing, adapter selection, retries, idempotency, and external failure boundaries.

Long-running or retryable work runs in apps/workers; external systems sit behind interfaces in packages/integrations. The worker registry describes what consumes each queue, while environment-aware factories select deterministic mocks, sandbox endpoints, or real adapters at composition roots.

Worker execution model

Caption: apps/workers/src/main.ts registers every imported defineJob() and schedules defineCron() entries. With REDIS_URL, BullMQ polls Valkey; otherwise the deterministic in-memory queue advances during the worker tick. Tenant-scoped handlers resolve the job’s route ID against the control plane and require active status before opening the tenant database.

The memory queue and BullMQ share the JobQueue contract: at-least-once delivery, delayed jobs, a configurable attempt budget (default three), and dead letters after exhaustion. BullMQ uses fixed backoff; individual protocols may add their own persisted retry policy. For example, webhook delivery stores attempts and uses exponential backoff up to five attempts while rechecking SSRF safety immediately before every fetch.

Adapter selection

Every provider exposes kind (mock, sandbox, or real) and a provider name. Contract tests always exercise supported mocks. Sandbox/testcontainer tiers activate only when their explicit environment gate is present and otherwise skip cleanly. Some tick-driven or scenario-specific mocks use focused unit tests instead of the shared harness; packages/integrations/src/contract.ts documents current adoption rather than pretending every port has identical coverage.

Factory behavior is provider-specific. Do not infer that every missing credential fails startup: many non-production factories deliberately fall back to a mock, while security-sensitive production configurations such as storage and the model gateway fail closed on mock, partial, or contradictory settings. Read and test the exact factory before changing deployment behavior.

Retry, idempotency, and failure boundaries

This is an at-least-once system, not exactly-once. A crash after enqueue but before published can cause re-enqueue; a stale outbox lease is reclaimable. Inbox keys collapse duplicate processing only for handlers wired through consumeWithInboxDedup(). Existing producers and handlers are still being migrated to the transactional outbox, as stated in packages/db/src/repos/outbox.ts; verify the actual job path before relying on outbox/inbox guarantees.

External provider calls are not part of a PostgreSQL transaction. Handlers must make replay safe through an idempotency key, a state check, a provider-supported idempotency mechanism, or a reconciliation loop. Persist enough state to distinguish retryable work from terminal failure without logging credentials or sensitive payloads.

Worker and adapter invariants

Request handlers enqueue OCR, sync, transcription, webhook, and reconciliation work; they do not perform it inline.
Cron publication enumerates only active tenants from the control plane.
A job payload cannot authorize a tenant; the worker re-routes and checks current status.
Queue delivery is at least once. Every externally visible effect needs an explicit replay strategy.
Tests use deterministic mocks; live provider calls require an explicit sandbox contract gate.
Provider selection happens at a composition root. Domain/application code depends on ports, not concrete SDK clients.
A single bad row should not poison a batch when the job is designed for row-level isolation (webhook retry is the concrete example).

Failure modes

Failure	Expected behavior	Source anchor
Valkey connection drops	Queue/worker error listener logs the named connection; BullMQ reconnects	`packages/integrations/src/queue/bullmq.ts`
Handler exhausts attempts	Job is copied to `<queue>--dead`	`packages/integrations/src/queue/bullmq.ts`, `queue.contract.test.ts`
Job contains stale/inactive tenant route	Worker rejects before opening tenant DB	`apps/workers/src/tenant.ts`
Queue publish fails	Outbox attempt/lease state remains recoverable; terminal after configured limit	`packages/db/src/repos/outbox.ts`
Duplicate envelope	Inbox returns `duplicate` for completed or still-leased processing key	`packages/db/src/repos/outbox.ts`
Search upsert fails after OCR	`indexedAt` remains null so the next batch retries	`apps/workers/src/jobs/document-index.ts`, `document-index.test.ts`
Webhook URL becomes unsafe or repeatedly fails	SSRF guard blocks fetch; persisted delivery eventually becomes `failed`	`apps/workers/src/jobs/webhook-delivery.ts`, `packages/integrations/src/webhooks/delivery.ts`
Partial real provider configuration	Exact factory either falls back only where documented or throws fail-closed	`packages/integrations/src/factory.ts`, `factory.test.ts`

Focused validation

pnpm --filter @blinqx/integrations test -- --maxWorkers=2
pnpm --filter @blinqx/workers test -- --maxWorkers=2
pnpm --filter @blinqx/integrations typecheck
pnpm --filter @blinqx/workers typecheck

The real Valkey contract is gated separately:

pnpm test:redis

Run container-backed contracts in CI or the remote test lane when local resource limits apply.

Source anchors

Worker composition: apps/workers/src/main.ts, apps/workers/src/registry.ts, apps/workers/src/tenant.ts
Representative jobs: apps/workers/src/jobs/document-index.ts, apps/workers/src/jobs/webhook-delivery.ts, apps/workers/src/jobs/outbox-dispatch.ts
Queue ports: packages/integrations/src/queue/index.ts, packages/integrations/src/queue/memory.ts, packages/integrations/src/queue/bullmq.ts
Provider conventions and selection: packages/integrations/src/contract.ts, packages/integrations/src/factory.ts
Delivery guarantees: packages/db/src/repos/outbox.ts, packages/integrations/src/webhooks/delivery.ts