Setting Up Context Caching with Redis

Why Cache Context

Database queries add latency to every AI request. Caching frequently-accessed context in Redis can reduce retrieval time from tens of milliseconds to sub-millisecond—a significant improvement for real-time applications.

Caching Strategy

What to Cache

• User profile context (high hit rate)
• Recently accessed conversation context
• Reference data (infrequently changing)

What Not to Cache

• Rapidly changing context (high invalidation rate)
• Rarely accessed historical context (low hit rate)
• Large context collections (memory pressure)

Implementation Steps

Step 1: Set Up Redis

Deploy Redis with appropriate memory limits. Enable persistence if cache warm-up time matters. Consider Redis Cluster for high availability.

Step 2: Implement Cache Layer

import redis
import json

def get_context(user_id: str, context_type: str):
    cache_key = f"context:{user_id}:{context_type}"
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)
    
    context = fetch_from_database(user_id, context_type)
    redis_client.setex(cache_key, 3600, json.dumps(context))
    return context

Step 3: Implement Invalidation

Invalidate when context changes. Use pub/sub for distributed invalidation. Consider lazy vs eager invalidation based on consistency requirements.

Monitoring

Track hit rates, memory usage, and eviction rates. Low hit rates suggest caching wrong data. High eviction rates indicate memory pressure.