Configuration¶

Broker¶

app.config_from_object({
    # Valkey/Redis
    "broker_url": "redis://localhost:6379/0",  # or valkey://

    # RabbitMQ (AMQP)
    # "broker_url": "amqp://guest:guest@localhost:5672//",
})

Result backend¶

app.config_from_object({
    "result_backend": "redis://localhost:6379/1",
})

Without a result backend, task return values are discarded. Set result_extended = True to store extra metadata (task name, args, kwargs, worker hostname).

Transport options¶

result_backend_transport_options is passed through to the backend. The Valkey/Redis backend reads one key of its own:

Key	Default	Description
`additional_connection_errors`	`()`	Extra exception classes, or dotted paths to them, to treat as connection errors and retry

A proxy or a managed Valkey/Redis service in front of the server raises its own exception type when it drops a connection, and the retry machinery only knows the client library's types. List those here so they are retried instead of surfacing as a hard failure. An entry that does not resolve to an exception class is skipped with a warning rather than failing backend construction:

app.config_from_object({
    "result_backend_transport_options": {
        "additional_connection_errors": ["mypackage.errors.ProxyDisconnected"],
    },
})

Pool sizing¶

The asyncio pool is sized by three settings, not by worker_concurrency:

Setting	CLI flag	Default	Description
`worker_loop_workers`	`--loop-workers`	1	Threads running an event loop
`worker_loop_concurrency`	`--loop-concurrency`	10	Concurrent async tasks per loop worker
`worker_sync_workers`	`--sync-workers`	1	Threads for sync tasks

Async task capacity is worker_loop_workers × worker_loop_concurrency. Sync tasks run in a separate pool of worker_sync_workers threads. On Python 3.14t both pools run in parallel.

worker_concurrency and -c are accepted for compatibility with upstream Celery and size the prefetch count, but the asyncio pool does not read them for capacity. Left unset, the concurrency is the pool's slots, worker_loop_workers × worker_loop_concurrency + worker_sync_workers.

Worker settings¶

Setting	Default	Description
`worker_max_tasks_per_child`	None	Restart worker after N tasks
`worker_max_memory_per_child`	None	Restart worker if RSS exceeds N KiB
`task_soft_time_limit`	None	Soft time limit in seconds
`task_time_limit`	None	Hard time limit in seconds
`task_acks_late`	False	Acknowledge tasks after execution
`worker_soft_shutdown_timeout`	0.0	Seconds to let active tasks finish before cancelling them on shutdown; 0 cancels straight away
`worker_enable_soft_shutdown_on_idle`	False	Wait out `worker_soft_shutdown_timeout` even with no active tasks
`worker_deduplicate_successful_tasks`	False	On a redelivered message, look the task id up in the result backend first and skip it if it already succeeded. Needs `task_acks_late` and a persistent backend

Acknowledgement¶

task_acks_late moves the acknowledgement to after the task has run, so a task whose worker dies mid-execution is redelivered. These settings decide what happens to a message when the task that ran late-acknowledged did not succeed:

Setting	Default	Description
`task_acks_on_failure`	None	Acknowledge the message when the task raises
`task_acks_on_timeout`	None	Acknowledge the message when the task hits its time limit
`task_acks_on_failure_or_timeout`	True	Covers both of the above; deprecated in 6.0, to be removed in 7.0

None means "fall back to task_acks_on_failure_or_timeout", so the default behaviour is unchanged from upstream: a failed or timed-out task is acknowledged and not redelivered. Set one of the two to False to have that kind of outcome redelivered instead. All three only apply to tasks that are acknowledged late; without task_acks_late the message is already gone by the time the task runs. The per-task attributes acks_on_failure and acks_on_timeout override the app setting.

Prefetch¶

Setting	CLI flag	Default	Description
`worker_prefetch_multiplier`	`--prefetch-multiplier`	4	Multiplied by the concurrency to get the prefetch count the worker asks the transport for
`worker_enable_prefetch_count_reduction`		True	After a connection loss, reconnect with a prefetch count reduced by the number of tasks still running

The prefetch count caps how many messages the worker holds unacknowledged. On AMQP, RabbitMQ enforces it for each queue the worker consumes. Valkey and Redis cannot push, so there the transport enforces it, for all of the worker's queues together: one consume round-trip claims as many messages as the cap has room for, up to 100, and a worker at the cap claims nothing more until it acks or rejects one.

A task acknowledged late holds its slot while it runs. With task_acks_late, a prefetch count below what the pool can run at once leaves the rest of the pool idle. The count follows the pool's slots unless -c is set, so only a -c below worker_loop_workers × worker_loop_concurrency + worker_sync_workers can cause it.

With worker_enable_prefetch_count_reduction on, a worker that reconnects while N tasks are still running comes back with the count lowered by one multiplier for each of them. Every task acked or rejected after that gives one multiplier back until the count is whole again, so the worker does not claim a second full window on top of the work already in hand.

Connection loss¶

Setting	Default	Description
`worker_cancel_long_running_tasks_on_connection_loss`	False	On losing the broker connection, cancel the running tasks that have `task_acks_late` set

A late-acknowledged task cannot be acknowledged once the connection is gone, so the broker redelivers it and the work is done twice. Turning this on cancels those tasks instead and lets the redelivery be the only run. It is off by default, and leaving it off logs a pending-deprecation warning on every reconnect.

Serialization¶

app.config_from_object({
    "task_serializer": "json",       # json, pickle, msgpack, yaml
    "result_serializer": "json",
    "accept_content": ["json"],
})

Compression¶

app.config_from_object({
    "task_compression": "gzip",      # gzip, bzip2, lzma, zstd
    "result_compression": "gzip",
})

brotli joins that list when the brotli extra is installed.

task_compression compresses the task message body, and the broker carries the method in a message header. result_compression compresses the stored result, which has no header, so the method is written in front of the payload instead.

A compressed result is binary, so result_compression is only honoured by a backend that hands arbitrary bytes back unchanged. The Valkey/Redis, filesystem and cache backends do, except Valkey/Redis with decode_responses in its URL, set to any value, false included. A backend that cannot ignores the setting and warns once, when it is built. A client with decode_responses cannot read a result that another worker stored compressed either, so drop the parameter from every result backend URL before turning compression on.

Reading is driven by the stored payload rather than by the setting, so results written before compression was turned on stay readable, and a reader with no compression configured can still read a compressed result. Roll the setting out to readers first if writers and readers run different versions.

An unrecognised method raises ImproperlyConfigured when the backend is built.

Task autodiscovery¶

app = Celery("myapp")
app.config_from_object({
    "include": ["myapp.tasks", "myapp.other_tasks"],
})

Or with Django:

app.autodiscover_tasks()

CLI options¶

celery -A app worker [OPTIONS]

Options:
  --loglevel=INFO          Log level (DEBUG, INFO, WARNING, ERROR)
  -E / --task-events       Enable task events for Flower
  --loop-workers=N         Number of async loop workers (default: 1)
  --loop-concurrency=N     Max concurrent async tasks per loop worker (default: 10)
  --sync-workers=N         Number of sync worker threads (default: 1)
  -c / --concurrency=N     Sizes the prefetch count, not the pool (default: the pool's slots)
  --max-tasks-per-child=N  Restart after N tasks
  --max-memory-per-child=N Restart if RSS exceeds N KiB
  --prefetch-multiplier=N  Prefetch count per concurrency slot (default: 4)