S3 Cache¶
S3Cache stores each cache key as one object in an S3 bucket. Use it for
values that are expensive to build but too large to keep in Valkey or Redis in
bulk, such as the rendered JSON of each item that a list endpoint reads 100 at
a time with get_many(). S3 Express One Zone answers a GET in single-digit
milliseconds and charges less per request than S3 Standard. S3 Standard
buckets work too, with higher latency.
S3Cache has get(), set(), delete(), has_key(), get_many(),
set_many(), delete_many(), clear() and incr_version() of Django's
cache API, and their async twins. add(), get_or_set(), touch(),
incr(), decr() and the cachex extension methods except type() raise
NotSupportedError (see What it does not do).
Installation¶
The s3 extra installs obstore 0.12,
the Python bindings of the Rust object_store crate. It has sync and async
APIs and wheels for the free-threaded build.
Configuration¶
from datetime import timedelta
CACHES = {
"items": {
"BACKEND": "django_cachex.cache.S3Cache",
"LOCATION": "s3://my-cache--euc1-az1--x-s3/items",
"TIMEOUT": 3600,
"OPTIONS": {
"region": "eu-central-1",
"compressor": "django_cachex.compressors.zstd.ZstdCompressor",
"client_options": {"timeout": "2s"},
"retry_config": {"retry_timeout": timedelta(seconds=5)},
},
},
}
LOCATION is s3://<bucket>/<prefix>. Every object of the cache goes under
the prefix. When the bucket name ends in --x-s3, the name of an S3 Express
directory bucket, the cache turns on S3 Express. Set "s3_express": False in
OPTIONS to turn it off.
| Option | Default | Meaning |
|---|---|---|
serializer |
pickle | As for the Valkey/Redis backends, see Serializers. |
compressor |
none | As for the Valkey/Redis backends, see Compression. |
max_concurrency |
64 |
The requests that one get_many() or set_many() sends at the same time, see Requests per call. |
client_options |
{"timeout": "5s", "connect_timeout": "1s"} |
obstore's HTTP settings. Your entries replace these per key. |
retry_config |
{"max_retries": 2, "retry_timeout": timedelta(seconds=10)} |
obstore's retries of a failed request. Your entries replace these per key. |
Every other entry goes to obstore's S3Store.from_url(): S3 settings such
as region, endpoint, access_key_id and secret_access_key, and
credential_provider. Without keys in
OPTIONS, obstore reads the AWS_* environment variables and falls back to
ECS/EKS container credentials and EC2 instance credentials. Put HTTP settings
such as allow_http in client_options, not at the top level of OPTIONS.
obstore's own defaults give a request 30 seconds and retry it up to 10 times
within 3 minutes, which would hold a web request for minutes while S3 is down.
The cache's defaults above give up after about 10 seconds. A large value on
S3 Standard can need a longer timeout.
Permissions¶
| Bucket | Grant |
|---|---|
| S3 Standard | s3:GetObject, s3:PutObject and s3:DeleteObject on <bucket>/<prefix>/*, and s3:ListBucket on the bucket |
| S3 Express directory bucket | s3express:CreateSession on the bucket |
Without s3:ListBucket, S3 answers a GET of a missing key with 403 instead of
404, so every cache miss raises obstore's PermissionDeniedError instead of
returning the default. clear() needs it too.
With the default pickle serializer, get() unpickles whatever object it finds
under the prefix. Whoever can write to the prefix can then run code in your
application. Grant write access to the prefix only to the application, or set
the JSON or MessagePack serializer.
Values¶
A bytes value skips the serializer. The object holds these bytes, compressed
by the configured compressor when the value is longer than the compressor's
min_length. A cache that stores orjson.dumps(item) therefore holds JSON or
compressed JSON, not a pickle. Every other value goes through the serializer
first.
Three user metadata fields describe the object:
| Field | Value |
|---|---|
x-amz-meta-cachex-expires |
The expiry as Unix seconds. Missing when the value does not expire. |
x-amz-meta-cachex-raw |
1 when the value was bytes and skipped the serializer. |
x-amz-meta-cachex-compressed |
1 when the compressor compressed the body. |
Expiry¶
S3 has no TTL per key. The cache reads an object after its
x-amz-meta-cachex-expires time as a miss, but the object stays in the bucket
until a later write replaces it or a delete removes it. Add a lifecycle rule
that deletes the objects under the prefix some days after they were written,
more days than the longest timeout you set:
{
"Rules": [
{
"ID": "cachex-items",
"Status": "Enabled",
"Filter": {"Prefix": "items/"},
"Expiration": {"Days": 2}
}
]
}
aws s3api put-bucket-lifecycle-configuration --bucket my-cache--euc1-az1--x-s3 \
--lifecycle-configuration file://lifecycle.json
The rule also deletes a value with a longer timeout or timeout=None after
those days. The cache then reads it as a miss. Directory buckets accept
expiration rules by prefix. Their bucket policy must let the lifecycle service
create sessions, see the AWS documentation on lifecycle for directory buckets.
Requests per call¶
| Call | Requests |
|---|---|
get() |
1 GET |
get_many(keys) |
1 GET per key, in parallel |
set() |
1 PUT |
set_many(data) |
1 PUT per key, in parallel |
has_key() |
1 HEAD |
delete() |
1 DeleteObjects |
delete_many(keys) |
1 DeleteObjects per 1000 keys |
clear() |
1 LIST and 1 DeleteObjects per 1000 objects |
set() overwrites, and the last write wins. set() with a timeout of 0 or
less and set_many() with one send a delete instead of a write. S3 does not
say whether a deleted object existed, so delete() returns True for a
missing key too, and delete_many() returns None.
obstore sends every delete as a DeleteObjects request, also the delete of one
key. For an S3-compatible store without DeleteObjects, set
"disable_bulk_delete": True in OPTIONS. obstore then sends one DELETE per
key.
get_many() and set_many() send at most max_concurrency requests at the
same time, so a call with more keys than that takes several round trips. With
5 ms per request, a get_many() of 100 values of 9 KB took 23 ms at 32,
14 ms at 64 and 11 ms at 128, measured against a local S3 server with a
delayed network. Raise max_concurrency to the number of keys in your
largest call when its latency matters more than the threads and connections.
The limit holds per call. A sync call sends its requests from a pool of
max_concurrency threads that belongs to the calling thread, and an async
call sends them on the event loop. No call waits for the requests of another
call. A process with 8 worker threads therefore has up to 8 times
max_concurrency requests in flight and up to as many pool threads. A pool
stops when its calling thread ends. One process keeps one S3 client per
configuration, so all calls share its connections.
Keys¶
The object key is <prefix>/<cache key>, where KEY_FUNCTION makes the cache
key (:1:user:42 by default). The cache percent-escapes %, / and control
characters, so a/b becomes a%2Fb. In a key made only of dots, it escapes
the dots. S3 allows 1024 bytes of UTF-8 in an object key, prefix included. A
longer key raises ValueError.
clear() deletes every object under the LOCATION prefix, also those that
another alias with the same prefix wrote. Give each alias its own prefix.
Without a prefix, clear() empties the bucket.
Forking¶
obstore does not survive os.fork(). After a process has sent a request
through S3Cache, every request in a child forked from it would hang, so the
cache raises RuntimeError there instead. Make no S3 cache call before the
fork: with gunicorn --preload or a Celery prefork pool, keep cache calls out
of module imports and the parent process.
The cache sees only its own requests. After any other obstore request in the parent, the cache calls of the child hang.
What it does not do¶
S3 has no atomic read-modify-write, so incr(), decr(), aincr() and
adecr() raise NotSupportedError. So do set() with nx, xx or get,
ttl(), expire(), keys(), the hash, list, set and sorted set methods,
xlen(), locks, pipelines and Lua. The other stream methods do not exist on
S3Cache and raise AttributeError. type() sends 1 HEAD and returns
KeyType.STRING for a key that exists.
add(), get_or_set() and their async twins raise NotSupportedError too. On
S3 Standard, a conditional write cannot tell an expired object from a live one
with the same bytes, so add() could not promise that only one caller wins.
Call get(), and set() on a miss.
touch() and atouch() raise it as well. S3 cannot change the metadata of an
object in place, so a new expiry would mean downloading the value and writing
it back. Call set() with the value and the new timeout.
The cache admin lists the alias as limited and shows no keys.