In short
Move work to a background job when it is slow, can fail independently of the request, or does not need to block the user. Most queues guarantee at-least-once delivery, which means every job must be safe to run twice. The failures that hurt are silent: jobs that fail without alerting, jobs stuck forever, and a queue that grows faster than it drains.
What belongs in a job
Move work out of the request when it is slow, fallible, or not needed for the response.
Sending email. Generating a PDF or a thumbnail. Calling a third-party API. Syncing to another system. Anything taking more than a moment, and anything that could fail for reasons unrelated to the user's action.
The test: if this fails, should the user's action fail? If someone books an appointment and the confirmation email bounces, the booking should still exist. That means the email belongs in a job.
What should not be a job: anything the user needs an answer about immediately, and anything where doing it twice would be worse than not doing it at all — unless you make it safe to repeat, which you should anyway.
At-least-once changes how you write code
Most queues guarantee a job runs at least once, not exactly once. That is a deliberate trade: exactly-once is extremely difficult, and at-least-once is achievable and honest.
The consequence is concrete. A worker that completes its work and then crashes before acknowledging will see the job again. So will one whose acknowledgement is lost.
Every job must be safe to run twice. Check whether the work is already done, use a unique constraint, or record processed identifiers — the same idempotency discipline that applies to APIs and webhooks.
The usual symptom of ignoring this is customers receiving the same email twice, which is embarrassing, and being charged twice, which is not survivable.
Pass identifiers, not objects
A job should carry the minimum needed to find its work — usually an ID.
Serialising a whole record into the payload means the job operates on a stale snapshot. By the time it runs, the record may have changed or been deleted, and you will act on data that is no longer true.
Passing an ID means the job loads current state, and can decide sensibly if the record is gone. It also keeps payloads small, which matters at volume.
Retries need thought
Retrying is the point of a queue, but naive retries make outages worse.
Back off exponentially. Retrying immediately against a struggling service adds load to something already failing.
Add jitter. Without it, everything that failed together retries together, producing a synchronised spike — the thundering herd.
Cap attempts, then move the job to a dead-letter queue rather than retrying forever.
Distinguish permanent from transient failures. A network timeout deserves a retry. A validation error will fail identically every time, and retrying it twenty times wastes capacity and buries real problems.
The dead-letter queue is not optional
Jobs that exhaust their retries have to go somewhere visible.
A dead-letter queue nobody looks at is the same as losing the work, but with more infrastructure. Alert on anything arriving there, keep enough context to understand why, and make replay straightforward once the underlying cause is fixed.
A weekly glance at the dead-letter queue is one of the higher-value habits in operating a system, because it is where silent data loss accumulates.
The failure modes that actually bite
Silent failure. Jobs failing without anyone knowing. The request succeeded, the user saw success, and the work never happened. This is the most common and most damaging pattern, and it is why monitoring has to cover jobs, not just requests.
Growing backlog. Jobs arriving faster than workers process them. Queue depth is the metric that predicts this, and it should be alerted on before it becomes visible to users.
Stuck jobs. A worker that dies mid-job without releasing it. Use visibility timeouts so unacknowledged work returns to the queue.
Poison messages. A job that crashes the worker every time, so the queue never progresses. Attempt limits plus a dead-letter queue prevent this.
One slow queue blocking everything. A batch of heavy jobs starving urgent ones. Separate queues by priority, and give password resets a different lane from monthly reports.
Deploy-time mismatch. Jobs enqueued by old code, processed by new code — or the reverse. Keep payloads backward-compatible across a deploy, the same way you would a database migration.
What to monitor
Queue depth, per queue. Rising steadily means you are under-provisioned.
Job age. The oldest unprocessed job matters more than the count.
Failure rate, by job type. One type failing is a bug; all types failing is infrastructure.
Dead-letter arrivals, alerted immediately.
Duration, at percentiles. A job type slowly getting slower is how backlogs begin.
Choosing infrastructure
For most teams, the right answer is the queue your framework already integrates with, backed by something you already run.
A database-backed queue is genuinely fine at small and medium scale, and has the significant advantage of transactional enqueueing — the job and the data change commit together, so you cannot enqueue work for a record that was never saved. That single property removes a class of bug.
Redis-backed queues are fast and widely used. Dedicated brokers offer stronger delivery guarantees and routing at the cost of another system to operate.
Do not start with the most capable option. Start with the one you can operate, and move when queue depth or delivery requirements force it.
If you have work happening in the background and no confidence it is completing, book a call.
Common questions
When should work go in a background job?
When it is slow, can fail independently of the user's action, or is not needed to produce the response. The test is whether a failure should fail the user's action — a booking should still exist if its confirmation email bounces, so the email belongs in a job.
What does at-least-once delivery mean for my code?
That every job must be safe to run twice. A worker that completes its work then crashes before acknowledging will see the job again, as will one whose acknowledgement is lost. Check whether the work is already done, use a unique constraint, or record processed identifiers.
Should I pass full objects or IDs to a job?
IDs. Serialising a record into the payload means the job operates on a stale snapshot — by the time it runs, the record may have changed or been deleted. Passing an ID lets the job load current state and handle a missing record sensibly, and keeps payloads small.
What is a dead-letter queue and do I need one?
It holds jobs that exhausted their retries, and yes. Without one, failed work either retries forever or disappears. Alert on anything arriving there, keep enough context to diagnose it, and make replay straightforward — a dead-letter queue nobody watches is just data loss with extra infrastructure.
What should I monitor for background jobs?
Queue depth per queue, the age of the oldest unprocessed job, failure rate by job type, arrivals in the dead-letter queue, and job duration at percentiles. Silent failure is the most damaging pattern — the request succeeded, the user saw success, and the work never happened.
