Skip to content

S3 Cache

S3Cache stores each cache key as one object in an S3 bucket. Use it for values that are expensive to build but too large to keep in Valkey or Redis in bulk, such as the rendered JSON of each item that a list endpoint reads 100 at a time with get_many(). S3 Express One Zone answers a GET in single-digit milliseconds and charges less per request than S3 Standard. S3 Standard buckets work too, with higher latency.

S3Cache has get(), set(), delete(), has_key(), get_many(), set_many(), delete_many(), clear() and incr_version() of Django's cache API, and their async twins. add(), get_or_set(), touch(), incr(), decr() and the cachex extension methods except type() raise NotSupportedError (see What it does not do).

Installation

uv add django-cachex[s3]

The s3 extra installs obstore 0.12, the Python bindings of the Rust object_store crate. It has sync and async APIs and wheels for the free-threaded build.

Configuration

from datetime import timedelta

CACHES = {
    "items": {
        "BACKEND": "django_cachex.cache.S3Cache",
        "LOCATION": "s3://my-cache--euc1-az1--x-s3/items",
        "TIMEOUT": 3600,
        "OPTIONS": {
            "region": "eu-central-1",
            "compressor": "django_cachex.compressors.zstd.ZstdCompressor",
            "client_options": {"timeout": "2s"},
            "retry_config": {"retry_timeout": timedelta(seconds=5)},
        },
    },
}

LOCATION is s3://<bucket>/<prefix>. Every object of the cache goes under the prefix. When the bucket name ends in --x-s3, the name of an S3 Express directory bucket, the cache turns on S3 Express. Set "s3_express": False in OPTIONS to turn it off.

Option Default Meaning
serializer pickle As for the Valkey/Redis backends, see Serializers.
compressor none As for the Valkey/Redis backends, see Compression.
max_concurrency 64 The requests that one get_many() or set_many() sends at the same time, see Requests per call.
client_options {"timeout": "5s", "connect_timeout": "1s"} obstore's HTTP settings. Your entries replace these per key.
retry_config {"max_retries": 2, "retry_timeout": timedelta(seconds=10)} obstore's retries of a failed request. Your entries replace these per key.

Every other entry goes to obstore's S3Store.from_url(): S3 settings such as region, endpoint, access_key_id and secret_access_key, and credential_provider. Without keys in OPTIONS, obstore reads the AWS_* environment variables and falls back to ECS/EKS container credentials and EC2 instance credentials. Put HTTP settings such as allow_http in client_options, not at the top level of OPTIONS.

obstore's own defaults give a request 30 seconds and retry it up to 10 times within 3 minutes, which would hold a web request for minutes while S3 is down. The cache's defaults above give up after about 10 seconds. A large value on S3 Standard can need a longer timeout.

Permissions

Bucket Grant
S3 Standard s3:GetObject, s3:PutObject and s3:DeleteObject on <bucket>/<prefix>/*, and s3:ListBucket on the bucket
S3 Express directory bucket s3express:CreateSession on the bucket

Without s3:ListBucket, S3 answers a GET of a missing key with 403 instead of 404, so every cache miss raises obstore's PermissionDeniedError instead of returning the default. clear() needs it too.

With the default pickle serializer, get() unpickles whatever object it finds under the prefix. Whoever can write to the prefix can then run code in your application. Grant write access to the prefix only to the application, or set the JSON or MessagePack serializer.

Values

A bytes value skips the serializer. The object holds these bytes, compressed by the configured compressor when the value is longer than the compressor's min_length. A cache that stores orjson.dumps(item) therefore holds JSON or compressed JSON, not a pickle. Every other value goes through the serializer first.

Three user metadata fields describe the object:

Field Value
x-amz-meta-cachex-expires The expiry as Unix seconds. Missing when the value does not expire.
x-amz-meta-cachex-raw 1 when the value was bytes and skipped the serializer.
x-amz-meta-cachex-compressed 1 when the compressor compressed the body.

Expiry

S3 has no TTL per key. The cache reads an object after its x-amz-meta-cachex-expires time as a miss, but the object stays in the bucket until a later write replaces it or a delete removes it. Add a lifecycle rule that deletes the objects under the prefix some days after they were written, more days than the longest timeout you set:

{
  "Rules": [
    {
      "ID": "cachex-items",
      "Status": "Enabled",
      "Filter": {"Prefix": "items/"},
      "Expiration": {"Days": 2}
    }
  ]
}
aws s3api put-bucket-lifecycle-configuration --bucket my-cache--euc1-az1--x-s3 \
    --lifecycle-configuration file://lifecycle.json

The rule also deletes a value with a longer timeout or timeout=None after those days. The cache then reads it as a miss. Directory buckets accept expiration rules by prefix. Their bucket policy must let the lifecycle service create sessions, see the AWS documentation on lifecycle for directory buckets.

Requests per call

Call Requests
get() 1 GET
get_many(keys) 1 GET per key, in parallel
set() 1 PUT
set_many(data) 1 PUT per key, in parallel
has_key() 1 HEAD
delete() 1 DeleteObjects
delete_many(keys) 1 DeleteObjects per 1000 keys
clear() 1 LIST and 1 DeleteObjects per 1000 objects

set() overwrites, and the last write wins. set() with a timeout of 0 or less and set_many() with one send a delete instead of a write. S3 does not say whether a deleted object existed, so delete() returns True for a missing key too, and delete_many() returns None.

obstore sends every delete as a DeleteObjects request, also the delete of one key. For an S3-compatible store without DeleteObjects, set "disable_bulk_delete": True in OPTIONS. obstore then sends one DELETE per key.

get_many() and set_many() send at most max_concurrency requests at the same time, so a call with more keys than that takes several round trips. With 5 ms per request, a get_many() of 100 values of 9 KB took 23 ms at 32, 14 ms at 64 and 11 ms at 128, measured against a local S3 server with a delayed network. Raise max_concurrency to the number of keys in your largest call when its latency matters more than the threads and connections.

The limit holds per call. A sync call sends its requests from a pool of max_concurrency threads that belongs to the calling thread, and an async call sends them on the event loop. No call waits for the requests of another call. A process with 8 worker threads therefore has up to 8 times max_concurrency requests in flight and up to as many pool threads. A pool stops when its calling thread ends. One process keeps one S3 client per configuration, so all calls share its connections.

Keys

The object key is <prefix>/<cache key>, where KEY_FUNCTION makes the cache key (:1:user:42 by default). The cache percent-escapes %, / and control characters, so a/b becomes a%2Fb. In a key made only of dots, it escapes the dots. S3 allows 1024 bytes of UTF-8 in an object key, prefix included. A longer key raises ValueError.

clear() deletes every object under the LOCATION prefix, also those that another alias with the same prefix wrote. Give each alias its own prefix. Without a prefix, clear() empties the bucket.

Forking

obstore does not survive os.fork(). After a process has sent a request through S3Cache, every request in a child forked from it would hang, so the cache raises RuntimeError there instead. Make no S3 cache call before the fork: with gunicorn --preload or a Celery prefork pool, keep cache calls out of module imports and the parent process.

The cache sees only its own requests. After any other obstore request in the parent, the cache calls of the child hang.

What it does not do

S3 has no atomic read-modify-write, so incr(), decr(), aincr() and adecr() raise NotSupportedError. So do set() with nx, xx or get, ttl(), expire(), keys(), the hash, list, set and sorted set methods, xlen(), locks, pipelines and Lua. The other stream methods do not exist on S3Cache and raise AttributeError. type() sends 1 HEAD and returns KeyType.STRING for a key that exists.

add(), get_or_set() and their async twins raise NotSupportedError too. On S3 Standard, a conditional write cannot tell an expired object from a live one with the same bytes, so add() could not promise that only one caller wins. Call get(), and set() on a miss.

touch() and atouch() raise it as well. S3 cannot change the metadata of an object in place, so a new expiry would mean downloading the value and writing it back. Call set() with the value and the new timeout.

The cache admin lists the alias as limited and shows no keys.