---
title: Workers and integrations
description: Background-job registry, queue semantics, tenant routing, adapter selection, retries, idempotency, and external failure boundaries.
---

Long-running or retryable work runs in `apps/workers`; external systems sit behind interfaces in `packages/integrations`. The worker registry describes what consumes each queue, while environment-aware factories select deterministic mocks, sandbox endpoints, or real adapters at composition roots.

## Worker execution model

```mermaid
flowchart LR
  producer[Web / control plane / cron]
  outbox[(Optional job_outbox)]
  queue[JobQueue]
  memory[Memory queue dev/tests]
  bull[BullMQ on Valkey]
  registry[apps/workers registry]
  route[Re-route JobTenantRef in control plane]
  handler[Job handler]
  repo[(Tenant database)]
  adapter[Integration port]
  dead[Dead-letter queue / terminal row]

  producer --> outbox --> queue
  producer --> queue
  queue --> memory
  queue --> bull
  memory --> registry
  bull --> registry
  registry --> route --> handler
  handler --> repo
  handler --> adapter
  handler -->|retry budget exhausted| dead
```

Caption: `apps/workers/src/main.ts` registers every imported `defineJob()` and schedules `defineCron()` entries. With `REDIS_URL`, BullMQ polls Valkey; otherwise the deterministic in-memory queue advances during the worker tick. Tenant-scoped handlers resolve the job's route ID against the control plane and require `active` status before opening the tenant database.

The memory queue and BullMQ share the `JobQueue` contract: at-least-once delivery, delayed jobs, a configurable attempt budget (default three), and dead letters after exhaustion. BullMQ uses fixed backoff; individual protocols may add their own persisted retry policy. For example, webhook delivery stores attempts and uses exponential backoff up to five attempts while rechecking SSRF safety immediately before every fetch.

## Adapter selection

```mermaid
flowchart TD
  port[Provider interface]
  factory[Environment-aware factory]
  explicit{Configuration complete and allowed?}
  mock[Deterministic mock]
  sandbox[Sandbox adapter / provider test environment]
  real[Real adapter]
  contract[Shared contract suite]
  fail[Fail closed on unsafe or partial production config]

  port --> factory --> explicit
  explicit -->|no credentials, non-production default| mock
  explicit -->|SANDBOX variables| sandbox
  explicit -->|explicit real configuration| real
  explicit -->|partial, contradictory or forbidden production config| fail
  mock --> contract
  sandbox --> contract
  real -.selected contract suites only.-> contract
```

Every provider exposes `kind` (`mock`, `sandbox`, or `real`) and a provider name. Contract tests always exercise supported mocks. Sandbox/testcontainer tiers activate only when their explicit environment gate is present and otherwise skip cleanly. Some tick-driven or scenario-specific mocks use focused unit tests instead of the shared harness; `packages/integrations/src/contract.ts` documents current adoption rather than pretending every port has identical coverage.

Factory behavior is provider-specific. Do not infer that every missing credential fails startup: many non-production factories deliberately fall back to a mock, while security-sensitive production configurations such as storage and the model gateway fail closed on mock, partial, or contradictory settings. Read and test the exact factory before changing deployment behavior.

## Retry, idempotency, and failure boundaries

```mermaid
sequenceDiagram
  participant DB as Tenant DB
  participant D as Outbox dispatcher
  participant Q as JobQueue
  participant W as Worker
  participant I as Inbox/dedup
  participant P as External provider

  DB->>DB: commit mutation + audit + pending outbox
  D->>DB: lease pending row (SKIP LOCKED)
  D->>Q: enqueue outside DB transaction
  alt enqueue fails
    D->>DB: release lease / increment attempts / dead after limit
  else enqueue succeeds
    D->>DB: mark published
    Q->>W: at-least-once delivery
    W->>I: claim idempotency key
    alt completed or active claim
      I-->>W: duplicate; skip effect
    else claim acquired
      W->>P: execute effect
      alt effect succeeds
        W->>I: mark completed
      else effect fails
        W->>I: release claim for retry
      end
    end
  end
```

This is an at-least-once system, not exactly-once. A crash after enqueue but before `published` can cause re-enqueue; a stale outbox lease is reclaimable. Inbox keys collapse duplicate processing only for handlers wired through `consumeWithInboxDedup()`. Existing producers and handlers are still being migrated to the transactional outbox, as stated in `packages/db/src/repos/outbox.ts`; verify the actual job path before relying on outbox/inbox guarantees.

External provider calls are not part of a PostgreSQL transaction. Handlers must make replay safe through an idempotency key, a state check, a provider-supported idempotency mechanism, or a reconciliation loop. Persist enough state to distinguish retryable work from terminal failure without logging credentials or sensitive payloads.

## Worker and adapter invariants

- Request handlers enqueue OCR, sync, transcription, webhook, and reconciliation work; they do not perform it inline.
- Cron publication enumerates only active tenants from the control plane.
- A job payload cannot authorize a tenant; the worker re-routes and checks current status.
- Queue delivery is at least once. Every externally visible effect needs an explicit replay strategy.
- Tests use deterministic mocks; live provider calls require an explicit sandbox contract gate.
- Provider selection happens at a composition root. Domain/application code depends on ports, not concrete SDK clients.
- A single bad row should not poison a batch when the job is designed for row-level isolation (webhook retry is the concrete example).

## Failure modes

| Failure | Expected behavior | Source anchor |
| --- | --- | --- |
| Valkey connection drops | Queue/worker error listener logs the named connection; BullMQ reconnects | `packages/integrations/src/queue/bullmq.ts` |
| Handler exhausts attempts | Job is copied to `<queue>--dead` | `packages/integrations/src/queue/bullmq.ts`, `queue.contract.test.ts` |
| Job contains stale/inactive tenant route | Worker rejects before opening tenant DB | `apps/workers/src/tenant.ts` |
| Queue publish fails | Outbox attempt/lease state remains recoverable; terminal after configured limit | `packages/db/src/repos/outbox.ts` |
| Duplicate envelope | Inbox returns `duplicate` for completed or still-leased processing key | `packages/db/src/repos/outbox.ts` |
| Search upsert fails after OCR | `indexedAt` remains null so the next batch retries | `apps/workers/src/jobs/document-index.ts`, `document-index.test.ts` |
| Webhook URL becomes unsafe or repeatedly fails | SSRF guard blocks fetch; persisted delivery eventually becomes `failed` | `apps/workers/src/jobs/webhook-delivery.ts`, `packages/integrations/src/webhooks/delivery.ts` |
| Partial real provider configuration | Exact factory either falls back only where documented or throws fail-closed | `packages/integrations/src/factory.ts`, `factory.test.ts` |

## Focused validation

```bash
pnpm --filter @blinqx/integrations test -- --maxWorkers=2
pnpm --filter @blinqx/workers test -- --maxWorkers=2
pnpm --filter @blinqx/integrations typecheck
pnpm --filter @blinqx/workers typecheck
```

The real Valkey contract is gated separately:

```bash
pnpm test:redis
```

Run container-backed contracts in CI or the remote test lane when local resource limits apply.

## Source anchors

- Worker composition: `apps/workers/src/main.ts`, `apps/workers/src/registry.ts`, `apps/workers/src/tenant.ts`
- Representative jobs: `apps/workers/src/jobs/document-index.ts`, `apps/workers/src/jobs/webhook-delivery.ts`, `apps/workers/src/jobs/outbox-dispatch.ts`
- Queue ports: `packages/integrations/src/queue/index.ts`, `packages/integrations/src/queue/memory.ts`, `packages/integrations/src/queue/bullmq.ts`
- Provider conventions and selection: `packages/integrations/src/contract.ts`, `packages/integrations/src/factory.ts`
- Delivery guarantees: `packages/db/src/repos/outbox.ts`, `packages/integrations/src/webhooks/delivery.ts`

See also: [Application architecture](/developers/application-architecture) · [Tenant isolation](/developers/tenant-isolation) · [Operations and runbooks](/developers/operations)