Skip to main content

Background Processing

Endatix decouples fast API responses from slow or unreliable side effects (webhook HTTP calls, future integrations). When a domain event occurs, such as a completed submission, the API persists your data and schedules background work in the same database transaction. A background relay then delivers that work reliably.

This page explains the transactional outbox pattern, how Endatix applies it today, and how to configure the in-process relay via Endatix:Outbox in appsettings.json. It also covers background jobs, their Endatix:BackgroundJobs settings, and delivering outbox events through the job queue.

Why Background Processing?​

Without background processing, every side effect runs inside the HTTP request:

  • Webhook delivery waits on external HTTP latency and retries.
  • A slow or failing endpoint can delay, or even fail, the API response after your data was already saved.
  • Work is lost if the process crashes mid-request.

Endatix avoids that by enqueueing durable work when domain state is committed, then processing it asynchronously.

The Transactional Outbox Pattern​

The transactional outbox pattern solves a simple problem: how do you atomically save business data and schedule background work?

Instead of calling a message broker or HTTP endpoint directly from the request handler, Endatix writes an outbox row in the same database transaction as the domain change. A separate relay process later reads pending rows and performs the side effect.

Benefits for self-hosted and multi-instance deployments:

BenefitHow Endatix achieves it
AtomicityThe outbox row is written in the same database transaction as your form and submission data - either both are saved or neither is.
DurabilityRows survive process restarts; nothing is held only in memory.
Safe scale-outMultiple API instances can run relays; SKIP LOCKED / lease-based claiming prevents double-processing.
RetriesFailed deliveries are rescheduled with exponential backoff; permanent failures move to a dead-letter state.
At-least-once deliveryReplays are expected; webhook receivers should treat X-Endatix-Hook-Id as an idempotency key.

Endatix Outbox Flow​

At a high level:

  1. Capture - When a domain event occurs (for example, a submission is completed), Endatix appends an outbox message (event type, JSON payload, tenant) in the same database transaction that saves your data. This happens automatically - no extra configuration or code is involved.
  2. Claim - The relay (a background service named OutboxRelayBackgroundService - you will see it in the logs) polls the outbox table and claims rows with a time-bounded lease.
  3. Publish - By default the relay runs every registered outbox subscriber (IOutboxIntegrationEventHandler) itself: webhooks, Reporting, and Audience. It sends to every webhook endpoint inline, and all matched handlers must succeed before the row is marked Sent. With delivery to the job queue on, the relay runs none of them: for each subscriber it enqueues that subscriber's own job type and returns, one job per Reporting or Audience handler that needs the message and one WebHookDelivery job per distinct enabled endpoint URL. Each job loads the outbox message when it runs and does the same work. A webhook job also loads the current endpoint settings, then sends one HTTP request. A non-success response, including 4xx, fails that job so the job queue retries it. Other endpoints are separate jobs.
  4. Complete - When every subscriber returns success, the row is Sent. With DeliverToJobQueue on, success means the jobs were queued, and retries then belong to each job, not to the outbox row. With the switch off, success means the inline handlers, webhook HTTP calls included, finished. A subscriber failure reschedules the outbox row until MaxAttempts, or fails it at once when the subscriber throws OutboxPermanentFailureException. A Sent row is still read by every attempt of the jobs it queued, so any outbox retention must skip messages whose jobs have not finished, or keep a message longer than any job reading it can live: the job type's MaxAttempts × MaxRuntimeMinutes, its total backoff, and however long it waits to run (behind its MaxConcurrency cap, or while no node is up). Otherwise a retrying job finds the message gone, fails, and the event is lost.

The relay engine itself lives in the Endatix.Outbox.Engine package (published as NuGet Endatix.Outbox.Engine). Endatix consumes that package and supplies Endatix-specific subscribers (webhooks, Reporting, Audience) and EF schema mapping.

Outbox subscribers​

The relay resolves a single IIntegrationEventPublisher that delegates to all IOutboxIntegrationEventHandler registrations whose EventTypes include the message EventType. With DeliverToJobQueue on, that publisher enqueues their jobs instead. Multiple handlers can subscribe to the same event (for example, submission.completed triggers webhook delivery and BI flattening).

Durable integration events (webhooks, Reporting, Audience) are driven by the outbox relay. Slack notifications for submission.completed still use MediatR after SaveChanges (planned migration to outbox in a follow-up).

Submission lifecycle events​

A submission's collection status belongs to one of five phases. The phase decides which event a change raises and whether the submission can still move:

PhaseBuilt-in codesCan still move
pendingnot_started, viewedYes
activein_progressYes
completedcompleteNo, final
closedscreen_out, quota_full, cancelledNo, final
lapsedexpired, abandonedYes

Each lifecycle change raises exactly one outbox event, named by the phases before and after it. Each event raises the submission's revision by one, so revision orders every event of one submission. Saving answers while a submission stays in_progress raises none.

Outbox EventTypeRaised whenWebhookPayload
submission.createdA submission is created, from any entry point-Lifecycle fields, no previousCollectionStatus
submission.startedpending → active-Lifecycle fields
submission.completedpending, active, lapsed → completedsubmission_completed (setting SubmissionCompleted)Lifecycle fields, plus jsonData and every field submission_completed has always carried
submission.closedpending, active, lapsed → closedsubmission_closed (setting SubmissionClosed)Lifecycle fields
submission.lapsedpending, active → lapsed-Lifecycle fields
submission.resumedlapsed → active-Lifecycle fields
submission.collection_status_movedThe status moves to another code of the same phase: pending, active, lapsed-Lifecycle fields

The built-in endpoints raise created, started, completed and closed. The respondent's first open of a not_started submission (one made on their behalf) raises submission.collection_status_moved to viewed, and their first answer then raises started. resumed follows a save of a submission stored in a lapsed status, and nothing built in moves a submission to a lapsed status yet: submission.lapsed is reserved for host code that sets such statuses.

The lifecycle fields every payload carries are submissionId, tenantId, formId, formDefinitionId, revision, occurredAt, isTestSubmission, collectionStatus and previousCollectionStatus. Both statuses are objects with the exact code and its phase, for example { "code": "screen_out", "phase": "closed" }. The values are the ones when the event was raised.

A create that lands in an outcome raises two events in one transaction: submission.created, carrying the status the row is saved in, then the outcome's event at the next revision. A public create with collectionOutcome set to screen_out raises submission.created (screen_out), then submission.closed. A create that is complete raises submission.created, then submission.completed.

Register custom collection statuses​

A host adds its own status codes, each with a phase, under Endatix:Submissions:CollectionStatuses:

"Endatix": {
"Submissions": {
"CollectionStatuses": {
"panel_overquota": "closed",
"invite_expired": "lapsed"
}
}
}

A code is 1 to 32 characters, stored lowercase. Through the API a host code is set only as a closed collectionOutcome; a code of another phase is recognised when it is stored, and a save keeps an active one as it is. A custom code raises its phase's event: moving to panel_overquota raises submission.closed. A save whose collectionOutcome is not a registered closed code gets a 400, and so does one that sends quota_full or cancelled: only the server sets them. A stored code that is not registered (or no longer is) still loads and reads as active, so its submission can still be saved and completed; the first read of each such code logs a warning.

React to a screen-out​

A screen-out is submission.closed with collectionStatus.code set to screen_out. To run code in the API host when one happens:

  1. Implement IOutboxIntegrationEventHandler (Endatix.Outbox.Engine). Set EventTypes to SubmissionLifecycleEvent.EventTypes.Closed (submission.closed).
  2. Read message.Payload as SubmissionLifecyclePayload. It is camelCase JSON, and long ids are strings. Check collectionStatus.code for the outcome you care about; every closed outcome raises this event.
  3. Register it on the API host: builder.Services.AddScoped<IOutboxIntegrationEventHandler, ScreenOutHandler>().
  4. Keep the in-process relay on (Endatix:Outbox:RunInProcess). The handler runs after the submission transaction commits. A throw fails the outbox delivery and the relay retries it, unless it is an OutboxPermanentFailureException (see Fail a message that can never succeed).
using System.Text.Json;
using Endatix.Core.Entities;
using Endatix.Outbox.Engine;

public sealed class ScreenOutHandler : IOutboxIntegrationEventHandler
{
public IReadOnlyCollection<string> EventTypes { get; } = [SubmissionLifecycleEvent.EventTypes.Closed];

public Task HandleAsync(IOutboxMessage message, CancellationToken cancellationToken)
{
using JsonDocument document = JsonDocument.Parse(message.Payload);
string? submissionId = document.RootElement.GetProperty("submissionId").GetString();
string? code = document.RootElement.GetProperty("collectionStatus").GetProperty("code").GetString();
if (code != CollectionStatusCodes.ScreenOut)
{
return Task.CompletedTask;
}

// React to the screen-out of submissionId
return Task.CompletedTask;
}
}

INotificationHandler<SubmissionCompletedEvent> still runs in process after a completing save; the MediatR sample in samples/Endatix.Samples.CustomEventHandlers uses it. No in-process notification exists for the other lifecycle events.

A host with Endatix:Outbox:DeliverToJobQueue on refuses to start while a handler like this has no background job. To run it either way, implement IOutboxWork<ScreenOutHandler> (Endatix.Infrastructure.Features.Outbox): a static JobType string that is persisted on queued jobs (never rename it), a static HandledEventTypes, and Read(message), which only parses the message and returns OutboxWork.For(input, ProcessAsync), or OutboxWork.None when there is nothing to do. Register it with builder.Services.AddOutboxWork<ScreenOutHandler>() in place of step 3's AddScoped, which would run the handler a second time inline. Pass onlyWhen when a frequent event rarely needs the work, and appLevelTenant when the outbox row has no tenant and the job's tenant is in the payload. The job runs only where the Jobs module is on.

A system outside the API process uses the webhook SubmissionClosed instead of this handler.

Upgrading from submission.collection_status_changed​

Nothing raises submission.collection_status_changed anymore. Earlier versions raised it only for a screen-out, so for this release an outbox row of that type left from before the upgrade is delivered as submission.closed, with collectionStatus { "code": "screen_out", "phase": "closed" }, to the submission.closed handlers and the submission_closed webhook. A webhook job queued for such a row delivers the same way. A handler that subscribed to the old type must switch to the lifecycle event types above.

Stop every API instance of the previous version before you start this one; don't roll the update. An older instance drops the new submission.closed rows without a webhook, and once the migration has run it no longer finds endpoints under the renamed SubmissionClosed setting. On Kubernetes, scale the API deployment to 0 replicas before you deploy this version, or set its strategy: Recreate for this release, then go back to your usual strategy.

Fail a message that can never succeed​

Throw OutboxPermanentFailureException (Endatix.Outbox.Engine), or a type derived from it, when no retry can make the message succeed, such as a payload that can never be read. Throw any other exception for a problem that may pass, such as a timeout. Its message is the reason that is logged, so name the subscriber in it, such as ScreenOutHandler: payload has no submissionId.

  • Inline (DeliverToJobQueue off): the relay marks the row Failed on that attempt, with no next attempt. The message fails for every subscriber: subscribers after this one do not run for it, and the work of subscribers before it stays done.
  • Job queue (DeliverToJobQueue on): the row is already Sent, so only this subscriber's job fails, on that attempt, with no retry. Other subscribers' jobs run as usual. The job logs the reason; the job status shows only that the message can never be processed. An appLevelTenant reader is the exception: it runs while the relay enqueues, so an OutboxPermanentFailureException from it marks the row Failed at once and no subscriber gets a job.

In-Process Relay (Default)​

By default, the outbox relay runs inside your API host as a hosted background service. No separate worker container is required for single-instance or moderate deployments.

"Endatix": {
"Outbox": {
"RunInProcess": true
}
}

Under the hood, RunInProcess seeds the OpenFeature feature flag outbox-relay-in-process, which the relay re-evaluates every poll cycle. Today the flag value comes from configuration read at startup, so changing RunInProcess requires an application restart to take effect. When a dynamic OpenFeature provider (for example, flagd) is plugged in later, the flag will be flippable at runtime - that is the planned cutover switch to a standalone relay worker.

Prerequisite

The outbox table must exist before the relay can run. Enable automatic migrations (Endatix:Data:EnableAutoMigrations) or apply migrations manually as part of your deployment process.

What Gets Delivered Today?​

These integration events flow through the outbox. The EventType column is what you will see in the OutboxMessages table:

Outbox EventTypeWebhook eventRaised whenOther subscribers
form.createdFormCreatedA form is created-
form.updatedFormUpdatedA form is updated-
form.enabled_state_changedFormEnabledStateChangedA form's enabled state changes-
form.deletedFormDeletedA form is deleted, with all its submissionsReporting: drop the form's rows; Audience: drop the form's audience
form.definition.updated-Active form definition schema changes or a new active definition is publishedReporting: compile export schema
submission.completedSubmissionCompletedA submission is completed (first time)Reporting: flatten
submission.closedSubmissionClosedA submission ends in a closed outcome, such as a screen-out-
Other submission.* lifecycle events-See Submission lifecycle events-
submission.updated-A completed submission's answers, metadata, definition, or submitter identity change, or any submission is marked as test or realReporting: re-flatten when answers or definition change; Audience: free the personalized link of a submission marked as test
submission.deleted-A submission is deleted on its own. A form's deletion raises form.deleted only, not one per submissionReporting: sync flattened row deletion

Every submission.* payload carries isTestSubmission, the submission's test flag when the event was raised. A submission marked as test or real arrives as submission.updated with changeKind test and the new value.

Webhook-mapped events use the same webhook configuration described in the Webhooks guide: tenant- or form-level endpoints and API key auth. DeliverToJobQueue defaults to false. Turn it on to deliver each endpoint as its own background job (Endatix:BackgroundJobs:JobTypes:WebHookDelivery). Duplicate URLs in one event collapse to one job. A URL that changes after the job is queued ends that job without a call.

Job delivery needs the Background Jobs module: see Delivery to the job queue.

The outbox row Id is reused as the webhook X-Endatix-Hook-Id header so retries and duplicate deliveries are safe for idempotent receivers.

Configuration Reference​

Add or customize the Endatix:Outbox section in appsettings.json. All settings are optional - omitting the section (or individual keys) uses the defaults below:

"Endatix": {
"Outbox": {
"RunInProcess": true,
"DeliverToJobQueue": false,
"PollIntervalSeconds": 5,
"BatchSize": 50,
"LeaseSeconds": 60,
"MaxAttempts": 8,
"BackoffBaseSeconds": 5,
"BackoffCapSeconds": 300
}
}
SettingDescriptionDefault
RunInProcessWhen true, the API host runs the outbox relay loop. When false, the in-process relay is disabled so an external relay worker can take over (see the warning below).true
DeliverToJobQueueWhen true, the relay hands each message to the job queue: webhook events enqueue one WebHookDelivery job per endpoint, Reporting events their Reporting* jobs, and Audience events an AudienceDeleteFormAudience job or, for a submission marked as test or real, an AudienceReleaseLinkOnTestMark job. Each job type has its own retries, concurrency, and retention under Endatix:BackgroundJobs:JobTypes. When false, the relay runs every subscriber inline. Read at startup. See Delivery to the job queue.false
PollIntervalSecondsHow often the relay polls for pending outbox messages. Lower values reduce latency; higher values reduce database load.5
BatchSizeMaximum number of messages claimed per poll cycle.50
LeaseSecondsDuration a claimed row is locked to this relay instance. If the worker crashes, the lease expires and another instance can reclaim the row.60
MaxAttemptsMaximum delivery attempts before the message is treated as permanently failed (dead-letter).8
BackoffBaseSecondsBase delay for exponential backoff between retries.5
BackoffCapSecondsUpper bound on retry delay (seconds).300
Do not disable the relay without a replacement

Do not set RunInProcess to false unless you are running your own relay worker against the same database. No standalone worker is currently published - with the in-process relay off, outbox messages accumulate as Pending and webhooks are silently never delivered (no errors are raised anywhere).

Environment-specific tuning

Use appsettings.Production.json for production values. For example, you might increase PollIntervalSeconds on large databases or tighten MaxAttempts when endpoints are known to be best-effort.

Background jobs​

Background jobs run work that outlives a request, such as one webhook delivery or one Reporting update, each with its own retries and status. They are provided by the Jobs module, which is off unless Endatix:FeatureFlags:JobsModule is true. It runs on PostgreSQL and on SQL Server 2025 or later (or Azure SQL Database), in the same database as your data. On SQL Server, enqueueing holds the job scheduler's cluster-wide trigger lock until it commits, unless you turn off LockOnInsert.

Each job is a row in jobs."BackgroundJobs", the record GET jobs/{jobId} reads, plus, while it waits to run, one trigger in the clustered store of Quartz.NET 4.7.2, in the same jobs schema. Enqueueing writes both in one transaction. Quartz decides when and on which node a job runs, and runs a stopped node's jobs again on another node. The job's row alone decides whether a failed attempt is retried or dead-lettered; each retry is a new trigger for the job, at the time its backoff sets. A job whose handler finished but whose result could not be saved, because the database was unavailable, records that result once the database is back; it is not run again. If its host was stopping at the time, the result is not kept, and the job runs again on another node or, if that was its last attempt, is dead-lettered there.

The jobs migrations create Quartz's tables; with automatic migrations off, apply them before you start the API, or startup fails with Database schema validation failed under table prefix 'jobs.qrtz_'. A node whose database cannot be reached when it starts stays up without its scheduler and starts it once the database answers; until then its readiness probe reports the database as down. A database that answers and refuses, such as a login it does not accept, fails the start.

Deployment profiles​

Every host that registers the Jobs module can enqueue. Endatix:BackgroundJobs:RunInProcess decides whether it also executes jobs:

  • One process: keep RunInProcess at true. The API enqueues and runs jobs.
  • API and workers: set RunInProcess to false on API instances and true on worker instances of the same image, all against the same database. API instances only enqueue; workers pick jobs up within IdleWaitTimeSeconds.

Each job type is capped per node (JobTypes:{JobType}:MaxConcurrency, which defaults to the job type's built-in value), so a webhook backlog never takes a Reporting or Audience slot. A node never runs a job type it has no handler for; the job waits for a node that has it. Jobs recovered from a stopped node count against their type's cap like any other.

Nodes' clocks must agree

Every node sharing the database must keep its clock within 1 second of the others. A node judges a peer dead by comparing timestamps; a node whose clock runs behind can be presumed dead while it is healthy, and its running jobs run a second time. The first run is stopped within CancellationPollSeconds of the second starting, and records nothing.

Delivery to the job queue​

With Endatix:Outbox:DeliverToJobQueue set to true, the relay stops running subscribers inside its loop. For each message it enqueues one background job per subscriber and marks the message Sent: one WebHookDelivery job per enabled webhook endpoint, and one job per Reporting or Audience handler that needs the message. Each job reads the outbox message when it runs. A failing webhook endpoint then retries and dead-letters on its own, without re-sending to the healthy endpoints or holding up Reporting and Audience.

  • Requires the Jobs module. Turn on Endatix:FeatureFlags:JobsModule as well (Background jobs). If the switch is on and the Jobs module is not registered, the relay pauses: messages stay Pending, and it logs The outbox relay is paused: Endatix:Outbox:DeliverToJobQueue is on but the Jobs module is not registered.
  • Every inline handler needs a job. Startup fails when an inline outbox handler has no background job for one of its events, naming the handler and the event, and when a handler is registered twice. Register a custom handler with AddOutboxWork, as in React to a screen-out, or keep the switch off.
  • Relay hosts register every subscribing module. Every host that runs the outbox relay must register every module that subscribes to outbox events, with the same module feature flags. A relay host without a module marks that module's events Sent without enqueuing its jobs, and nothing can detect it.
  • Webhook attempts. Each webhook job reads the endpoint's current settings, within the event's tenant, and POSTs once per attempt, with no HTTP-level retries. Any response but success, 4xx included, fails the attempt. An endpoint therefore receives at most JobTypes:WebHookDelivery:MaxAttempts requests for one event: 8, its built-in default, on every host. The general MaxAttempts changes that only on a host that sets it. X-Endatix-Hook-Id is still the outbox message id.
  • Redelivery is safe. A message delivered twice, after a crash between enqueueing and marking it sent, enqueues nothing new: each job carries the dedup key {outboxMessageId}:{subscriber}.

Background jobs configuration reference​

All keys are under Endatix:BackgroundJobs. Per-job-type values go under JobTypes:{JobType}.

Anything set in configuration wins over code. Each per-job-type setting is looked up on its own, and the first of these that has a value is used:

  1. the job type's own key, Endatix:BackgroundJobs:JobTypes:{JobType}:{Setting};
  2. the general key, Endatix:BackgroundJobs:{Setting}, if the host set it, even to its default value;
  3. the job type's built-in default, declared in code;
  4. the general built-in default, listed with each key below.

MaxConcurrency has no general key, so it takes the job type's own key, then the job type's built-in default, then 1.

For example, a host whose appsettings.json has

"Endatix": {
"BackgroundJobs": {
"RetentionDays": 30,
"JobTypes": {
"WebHookDelivery": { "MaxAttempts": 6 }
}
}
}

keeps every finished job's row for 30 days, webhook deliveries included, and tries a webhook delivery 6 times; the other webhook settings keep their built-in defaults. The same with environment variables:

Endatix__BackgroundJobs__RetentionDays=30
Endatix__BackgroundJobs__JobTypes__WebHookDelivery__MaxAttempts=6

The host checks each range given below when it starts. A value outside its range fails startup with a message that names the key, such as Endatix:BackgroundJobs:MaxAttempts must be at most 10, but is 12. A job type's own key has the same range as the general key it replaces.

Whether this host executes jobs. false makes it enqueue only: its scheduler has no threads and never starts.

How long an idle node waits before looking for jobs another node enqueued. Range: 1 to 300.

How often a running job re-reads its row to notice it was cancelled. Range: 1 to 3600.

How long a due job may wait for a free slot before endatix.jobs.misfired counts it, again at every threshold while it waits. The backlog warning Background job {JobId} of type {JobType} waited past the misfire threshold; {Misfires} triggers of this type misfired since the last warning. is logged at most once per job type per threshold. Range: 1 to 3600.

How long a stopping host waits for running jobs. Jobs still running then are told to stop, record nothing, and run again on the next node or, if that was their last attempt, are dead-lettered there. The host's own shutdown timeout (HostOptions.ShutdownTimeout, 30 seconds by default) also bounds this wait, so a longer value needs a longer host timeout. Range: 1 to 3600.

Endatix:BackgroundJobs:LockOnInsertDefault false on PostgreSQL, true on SQL Server

Whether enqueueing holds the scheduler's cluster-wide trigger lock until its transaction commits. On SQL Server, false needs READ_COMMITTED_SNAPSHOT on for the database, or the node refuses to start, naming both settings. Read from configuration once, at startup. Read Persistence settings before you change it.

Ceiling on one attempt; the handler's token is cancelled when it passes. Per job type. A host that sets this key replaces every job type's built-in default, even when it sets it to 60; a job type's own key still wins. Range: 1 to 1440.

Attempts before a job that keeps failing is dead-lettered. Per job type. A job whose node stops during its last attempt is dead-lettered when another node recovers it, rather than run again. Each attempt runs with the value the node running it was started with, not one fixed at enqueue, so once the nodes restart with a new value it also applies to jobs already waiting to retry. A host that sets this key replaces every job type's built-in default, even when it sets it to 3; a job type's own key still wins. Range: 1 to 10.

Wait before the first retry; each later retry waits twice as long. Per job type. The wait is worked out when an attempt fails, from the value the node running it was started with, so once the nodes restart a change reaches the later retries of jobs already waiting; a retry already scheduled keeps its time. A host that sets this key replaces every job type's built-in default, even when it sets it to 30; a job type's own key still wins. Range: 1 to 86400.

Longest wait between retries. Per job type. It takes effect as BackoffBaseSeconds does. A host that sets this key replaces every job type's built-in default, even when it sets it to 900; a job type's own key still wins. Range: 1 to 86400, and not below BackoffBaseSeconds.

Days a finished job's row is kept before the retention job deletes it. Per job type. A host that sets this key replaces every job type's built-in default, even when it sets it to 7; a job type's own key still wins. Range: 1 to 3650.

Jobs of this type one node runs at once. 0 keeps the node from running the type. Unset, it is the job type's built-in default: 4 for WebHookDelivery, 2 for each Reporting and Audience type, 1 for any other. The node's thread pool is the sum of these caps (at least 1), plus one thread for the retention job. Range: 0 to 1000.

When the retention job runs, as a Quartz cron expression. It runs on one node at a time. An expression Quartz cannot parse fails startup.

Rows the retention job deletes per statement. Range: 1 to 100000.

Batches one retention run deletes at most. Range: 1 to 10000.

Serves the Quartz dashboard at /quartz and its HTTP API at /quartz-api, to platform admins only. It shows every tenant's jobs.

Lets platform admins change schedules through the dashboard. Read-only otherwise. A caller who may write can trigger, pause, reschedule, unschedule or delete any tenant's jobs. Unscheduling a job's trigger, or deleting a job type's job, leaves those jobs' rows unfinished with nothing to run them.

How often a node records that it is alive. Range: above 0, up to 300.

How late a check-in may be before the node is presumed dead; its running jobs then run again on another node. Range: above 0, up to 300.

This node's identity. Unset, it is generated from the host name and start time. Set it only to a value no other running node uses.

Built-in job type defaults​

These job types declare their own defaults in code, so every host runs them as tuned without setting anything. A value set in configuration replaces them, as described above.

  • WebHookDelivery: MaxAttempts 8, MaxRuntimeMinutes 5, BackoffBaseSeconds 10, BackoffCapSeconds 3600, RetentionDays 3, MaxConcurrency 4. Eight attempts, doubling from a 10-second wait, keep retrying for about 21 minutes.
  • Reporting and Audience outbox handlers (ReportingCompileFormSchema, ReportingFlattenSubmission, ReportingRebuildSearchCopy, ReportingSeedDefaultExportFormats, ReportingSyncFormDeletion, ReportingSyncSubmissionDeletion, AudienceDeleteFormAudience, AudienceReleaseLinkOnTestMark): MaxAttempts 5, MaxRuntimeMinutes 10, BackoffBaseSeconds 10, BackoffCapSeconds 600, RetentionDays 3, MaxConcurrency 2. Five attempts retry for about 2.5 minutes.
  • AudienceImportCheck: MaxAttempts 3, MaxConcurrency 2. Its other settings are the general values.
  • AudienceImportWrite: MaxAttempts 8, BackoffBaseSeconds 5, BackoffCapSeconds 300, MaxConcurrency 2. Its other settings are the general values.
  • Any other job type: the general values, and MaxConcurrency 1.
A general backoff value must suit every job type

A general BackoffBaseSeconds or BackoffCapSeconds is checked against each job type's built-in values. For example, a general BackoffBaseSeconds of 700 fails startup for any job type whose built-in BackoffCapSeconds is lower, such as the Reporting types (600). Set the job type's own key instead, such as JobTypes:WebHookDelivery:BackoffBaseSeconds.

Separate-Process Relay (Planned)​

A standalone relay worker - a separate process or container that claims outbox rows from the same database and publishes them via a message broker (for example, Dapr pub/sub) - is planned but not yet available. It targets multi-instance scale-out and cross-service fan-out; typical self-hosted single-node deployments will not need it.

The relay engine is already built to support this: the Endatix.Outbox.Engine package is host-agnostic, and the same claiming/lease mechanics will power both the in-process relay and the future worker. When the worker ships, the cutover will be: deploy the worker against the same database, then set RunInProcess to false on API instances.

Until then, keep RunInProcess set to true (see the warning above). Follow the endatix-relay repository for availability.

Operations​

Verifying Delivery​

The OutboxMessages table is the ground truth for background delivery. Each row's Status column tells you where it is in the pipeline:

StatusMeaning
0 (Pending)Captured, waiting for the relay to deliver it (check NextAttemptAt and Attempts if it stays here).
1 (Sent)Delivered successfully.
2 (Failed)Gave up after MaxAttempts, or failed at once by OutboxPermanentFailureException - requires operator attention.

With DeliverToJobQueue on, Sent means the message's jobs were queued, not that they ran. A webhook's retries and dead-letter are then on its WebHookDelivery job, in jobs."BackgroundJobs": read one job with GET jobs/{jobId}, or count them by status, where Status is 0 Pending, 1 Processing, 2 Retrying, 3 Completed, 4 Failed, 5 DeadLettered and 6 Canceled:

SELECT "JobType", "Status", COUNT(*) FROM jobs."BackgroundJobs" GROUP BY "JobType", "Status";

The Quartz dashboard does not show a job's status or attempts, because it reads Quartz's triggers and firings, not the job's row. A finished job has no trigger, and the dashboard's history lists a firing as successful even when its attempt failed.

A quick backlog check:

SELECT "Status", COUNT(*) FROM "OutboxMessages" GROUP BY "Status";

The relay also emits structured logs (message id, event type, attempt, next retry time) - look for the OutboxRelayBackgroundService log source. With the switch off, if a webhook did not arrive, one query tells you whether the event was never captured (no row), is still retrying (Pending with growing Attempts), or was dead-lettered (Failed) - which separates a relay problem from a receiving-endpoint problem.

Metrics​

With DeliverToJobQueue on, the relay's fan-out reports on the Endatix.Outbox meter, which the host's OpenTelemetry metrics pipeline subscribes to:

InstrumentTypeUnitTags
endatix.outbox.fanout.jobscounter{job}endatix.outbox.event_type

Each publish of a message adds the number of jobs it asked the queue for. A message published again, for example after the relay stopped before marking it Sent, is counted again, although its dedup keys keep the jobs unique. The jobs report on the Endatix.Jobs meter (endatix.jobs.*, tagged endatix.job.type).

Failure Handling​

When webhook delivery fails, expect:

  • Inline delivery (DeliverToJobQueue false): HTTP retries on the relay's client, then outbox retries until MaxAttempts.
  • Job delivery (DeliverToJobQueue true): one HTTP call per job attempt. The job type WebHookDelivery retries until its own MaxAttempts (8 by default). A failed endpoint does not retry the others.

Monitor backlog growth if endpoints are down for extended periods. Guidance on triaging and replaying Failed messages will be added in a future release.