If an API operation can outlive a normal request timeout, return an accepted job identifier and do the work outside the request path. That answer is simple. Making it safe for a customer to retry is the architecture work.
Define the contract before selecting a queue
Consider a customer portal that builds a year-end export. The browser sends one request, receives a job ID, and polls a status endpoint. The status has states such as queued, running, completed, failed and cancelled. The completed state points to a short-lived download, not a file in a public directory.
POST /exports -> 202 {"jobId":"exp_123","statusUrl":"/exports/exp_123"}
GET /exports/exp_123 -> 200 {"state":"running"}Authorize both endpoints against the same tenant and user entitlement. A random job ID is not permission to read another customer's export. Retain enough status for support after the file expires.
Commit intent before starting work
Create a durable job record with an operation key, owner, input version and state. Enqueue after the record commits, or use an outbox if losing the enqueue between database commit and queue publish would be unacceptable. A browser retry should resolve to the existing operation when the intent is unchanged.
A Redis-backed queue is useful for dispatch and worker coordination; it is not automatically the source of truth for customer-visible results. Keep the business job record in durable application storage, and set a retention policy for generated files and failed attempts.
Make failure visible, not magical
A worker can die after calling a third-party API but before recording success. The next attempt must reconcile that uncertain outcome or use the target's documented idempotency key. Blindly rerunning an invoice, payment or notification may duplicate the business action. Set retry budgets and a manual-review state for outcomes that cannot be proved.
Measure queue age, run duration, failure reason and time to completion. A full queue can be a capacity problem; a growing queue with idle workers is a deployment problem. Alert on the customer-visible delay, not just on Redis availability.
An export walkthrough
A sensible first slice is a CSV export of customer cases. The request stores the filters, requesting user and tenant, then returns a job reference. A worker reads cases in bounded pages, writes to a private temporary object and marks the job complete only after the file is closed and its checksum recorded. The download endpoint rechecks permission and uses a short-lived URL. This sequence makes a partly written file impossible to mistake for a finished export.
Think through cancellation explicitly. Stopping a worker after it has sent an external email cannot unsend the email. For read-only exports, cancellation can stop work and delete the temporary object; for operations with side effects, cancellation may only stop future steps and require reconciliation of completed ones.
Capacity and retention
Set maximum active jobs per tenant, file-size bounds and an expiration policy for result objects. A customer who requests the same huge export twenty times should not consume the whole worker fleet. Queue priority may protect urgent interactive work, but it should not starve longer jobs indefinitely. Store failure reasons that support staff can diagnose without exposing credentials or document contents.
A retry test should kill the worker immediately before and after each external side effect, then assert the final business record appears once. That is stronger evidence than a unit test that runs the happy path twice. If the target has no idempotency or lookup facility, document the manual decision point rather than promising exactly-once execution.
When a job is unnecessary
If a bounded request usually completes quickly and has no expensive side effect, a queue adds a second process and more states to operate. First identify whether the timeout comes from a slow query, a missing index or a remote service. Moving an inefficient operation into a worker can hide the problem while increasing latency.
Sources and further reading
Services This Relates To
Written by KYCONNECTS Engineering.