A Django project running Celery with task_always_eager=True and a single worker locally reveals almost none of the problems that show up in production: tasks lost on broker restart, tasks executed twice with nothing visible in the logs, workers killed mid-task by the OOM killer. Celery's default configuration is tuned for development comfort, not production reliability - and moving from one to the other requires explicit decisions, not just pointing CELERY_BROKER_URL at Redis.
Why the default configuration fails in production
Celery's default behavior acknowledges (acks) a task before executing it, as soon as the worker receives it (task_acks_late = False). If the worker crashes mid-execution - OOM, kill -9, a network drop to the broker - the task is lost for good, since the broker already considers it handled. That's a reasonable choice for fast, non-critical tasks, but unacceptable for sending a payment confirmation email or generating an invoice.
# settings.py - configuration tuned for reliability over raw throughput
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_BROKER_TRANSPORT_OPTIONS = {
# Must exceed the expected maximum task duration, otherwise Redis
# redelivers the task to another worker while it's still running elsewhere
'visibility_timeout': 3600,
}
task_acks_late = True flips the logic: acknowledgment only happens after success, so a task interrupted by a worker crash is automatically redelivered. The mandatory corollary is lowering worker_prefetch_multiplier to 1 - otherwise a worker can hold several reserved-but-unacknowledged tasks in memory, and all of them get requeued on a crash, not just the one actually running.
Idempotency and retries: the mandatory tradeoff
acks_late eliminates lost tasks but introduces the possibility of a task running twice - a worker that finishes processing right as the broker decides it's lost (a transient network timeout, for instance) can have the same task redelivered and run by a second worker. Without idempotency, that's the exact same double-side-effect risk as with any retry system.
from celery import shared_task
from django.db import transaction
@shared_task(
bind=True,
autoretry_for=(ConnectionError, TimeoutError),
retry_backoff=10,
retry_backoff_max=300,
retry_jitter=True,
max_retries=5,
)
def send_invoice_notification(self, invoice_id, idempotency_key):
with transaction.atomic():
invoice = Invoice.objects.select_for_update().get(id=invoice_id)
if invoice.notification_sent_at is not None:
# Already handled by a previous run, even in case of double delivery
return
notification_service.send(invoice, idempotency_key=idempotency_key)
invoice.notification_sent_at = timezone.now()
invoice.save(update_fields=['notification_sent_at'])
The select_for_update() combined with the check against already-persisted state is what actually makes the task safe to replay, independently of Celery's own retry mechanism.
Diagnosing in production
Three complementary tools, worth using before assuming an abnormal behavior is a code bug:
# Monitor queue depth in real time (Redis broker)
redis-cli -h redis-host llen celery
# Flower gives a live view of running tasks, retries and failures
celery -A myproject flower --port=5555
Worker logs showing WorkerLostError almost always point to a worker killed abruptly (the system's OOM killer, most often) rather than an application exception - a signal worth correlating with system memory metrics, not just Celery logs. A queue that keeps growing without ever draining usually means production task throughput exceeds the processing capacity of currently deployed workers, not an isolated stuck task.
The dead letter pattern: Celery doesn't provide it natively
Unlike systems like SQS or RabbitMQ natively configured with a dead letter exchange, Celery has no built-in failure queue by default. A task that exhausts its retry attempts is simply marked FAILURE and disappears from the radar unless something explicitly watches for it.
from celery.signals import task_failure
@task_failure.connect
def handle_task_failure(sender=None, task_id=None, exception=None, **kwargs):
FailedTask.objects.create(
task_name=sender.name,
task_id=task_id,
exception=str(exception),
payload=kwargs.get('kwargs', {}),
)
This signal turns permanent failures into persisted, queryable records - the functional equivalent of a dead letter queue, built by hand because Celery doesn't offer one natively. Without this kind of explicit mechanism, a critical business task failing in a loop can go completely unnoticed until a customer complains.
The fix, and its tradeoffs
acks_late and prefetch_multiplier=1 improve reliability at the cost of throughput: each worker now processes only one task at a time before reserving a new one, which reduces effective parallelism on very short tasks. Splitting tasks across dedicated queues (celery -A myproject worker -Q payments for critical tasks, a separate queue for non-urgent reporting tasks) isolates slow tasks from fast ones, preventing a heavy report job from blocking transactional emails stacked behind it in the same queue.
# Dedicated workers per queue, to isolate critical tasks from background ones
celery -A myproject worker -Q payments --concurrency=4 -n payments@%h
celery -A myproject worker -Q reports --concurrency=2 -n reports@%h
When the easy fix isn't enough
For a genuinely long-running task (a large export, video processing), acks_late alone isn't enough: if the worker gets killed mid-processing, the entire task restarts from scratch instead of resuming where it left off. The right answer is breaking the task into idempotent steps with a persisted checkpoint (progress saved to the database), rather than relying on Celery to manage a monolithic task running for tens of minutes.
Another Django-specific trap involves celery beat: if a deployment ends up with several beat instances running simultaneously (often unintentional during a rolling deployment), scheduled tasks fire twice on every interval. The fix isn't a simple retry or task-level idempotency, but an explicit distributed lock between beat instances (for example via redbeat, which uses Redis as the single source of truth for scheduling) guaranteeing that only one instance actually triggers each scheduled task.
Finally, the choice of broker carries durability implications that go beyond Celery's own configuration: Redis without persistence (appendonly no) can lose queued tasks on an abrupt restart of the Redis process, while RabbitMQ with durable queues and persistent messages survives a broker restart. That choice is made upstream, at the infrastructure level, not by tweaking Celery options.
Bounding execution time with soft_time_limit
A task that loops indefinitely (an external dependency that never responds, a poorly bounded pagination loop) blocks a worker without ever triggering a retry, since no exception is ever raised. acks_late offers no protection against this case: the worker just stays busy forever, never acknowledging and never failing the task.
@shared_task(
bind=True,
soft_time_limit=300, # raises SoftTimeLimitExceeded, catchable in code
time_limit=330, # kills the worker if the task ignores the soft limit
)
def generate_monthly_report(self, account_id):
try:
return report_service.build(account_id)
except SoftTimeLimitExceeded:
# Clean exit point: log the partial state instead of
# letting the worker get killed with no context
logger.warning('Report generation exceeded soft limit', extra={'account_id': account_id})
raise
The hard time_limit acts as a final safety net: if the code ignores or mishandles SoftTimeLimitExceeded, Celery forces the worker to stop after the configured grace period. The tradeoff is that a time_limit set too low on a task that's legitimately long on a high-load day kills it wrongly - the value has to be calibrated against duration actually observed in production, not an optimistic estimate measured in development against a small dataset.
Takeaway
Celery reliability in production is built at three simultaneous levels: late acknowledgment combined with task-level idempotency to survive worker crashes, an explicit dead-letter strategy since Celery doesn't provide one natively, and a broker and queue topology choice that genuinely matches the durability guarantees the business actually needs.
