Unrecoverable error in Celery worker: ReadOnlyError with Redis backend

[sklyar61]

Hello,

I'm experiencing an issue with my Celery worker intermittently shutting down due to a ReadOnlyError when trying to write to my Redis instance. Here's the error message I'm seeing:

[2023-06-22 12:38:52,599: CRITICAL/MainProcess] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")

This seems to be happening when my Redis instance goes into a read-only mode during sync.

I am using the following setup:

• Python 3.8
• Celery 5.2.7
• Django-redis 5.2.0

My docker-compose.yml for Redis is as follows:

redis:
image: redis:latest
command: ["redis-server", "--maxmemory 300mb", "--maxmemory-policy allkeys-lru"]
ports:
- "6379:6379"

And my Celery configuration in Django settings is:

CELERY_BROKER_URL = 'redis://redis:6379/0'
CELERY_RESULT_BACKEND = 'redis://redis:6379/0'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

I'm currently mitigating the issue by configuring my docker-compose to restart the Celery worker when it stops, but this is not a long-term solution. Is there a way to handle this ReadOnlyError within Celery, or to prevent Redis from going into read-only mode in the first place?

Thank you in advance for your help and suggestions.