Redis cache, so fast, so easy to setup but a potential Chernobyl, lying within your system.
The problem comes when the Redis cache is used as a permanent store of values within your system, so applications and services get to use it as stable source of truth. This is an easy trap to fall into since Redis is fairly stable, fast and easy to add values to
When there is any issue with this cache systems can be brought to their knees, as there is no easy fall back to recreate the cache, leading to expensive downtime
What are ways to mitigate this:
- Regularly expire your cache keys and values – this forces applications to regularly fill the cache and be able to work without it. We are commonly using 24 hours for this so that it is regular and occurs during the week days
- Cycle your cache servers – a slow modified version of the Netflix chaos monkey, create new cache servers every 6 months or so, a little painful when you have multiple services whose cache service urls need to be updated
- Use a permanent database based cache for computed values that do not change, which is close to the source – maybe Redis is not the place to store this complex data
What are your thoughts and approaches that you have used?
