Caching
Chapter 5: Caching Strategies
Section titled “Chapter 5: Caching Strategies”Improving Performance with Caches
Section titled “Improving Performance with Caches”5.1 Understanding Caching
Section titled “5.1 Understanding Caching”Caching is the process of storing frequently accessed data in a faster storage layer to reduce access time and decrease load on the primary data source.
Without Caching With Caching =============== ==============
Request Request | | v v +------+ +------+ | DB | (100ms) |Cache | (1ms) +------+ +------+ | v (if miss) +------+ | DB | +------+
Result: Up to 100x faster response!Why Cache?
Section titled “Why Cache?”| Benefit | Impact |
|---|---|
| Reduced Latency | ms instead of seconds |
| Reduced Load | Fewer database queries |
| Improved Throughput | Handle more requests |
| Cost Savings | Fewer database resources |
5.2 Cache Locations
Section titled “5.2 Cache Locations”5.2.1 Client-Side Caching
Section titled “5.2.1 Client-Side Caching” Client-Side Cache =================
+--------------------------------------------------+ | User's Browser | +--------------------------------------------------+ | +----------+ | | | Local | - HTTP Cache | | | Storage | - LocalStorage | | +----------+ - IndexedDB | | - Service Workers | +--------------------------------------------------+
What's cached: - Static assets (CSS, JS, images) - API responses - User preferences5.2.2 CDN (Edge) Caching
Section titled “5.2.2 CDN (Edge) Caching” CDN Caching ===========
User Request | v +-----------------+ | CDN Edge | | Location | (Check cache first) +-----------------+ | | (Cache miss) v +-----------------+ | Origin | | Server | +-----------------+
Cached: - Static content - Images, videos - CSS, JavaScript5.2.3 Application-Side Caching
Section titled “5.2.3 Application-Side Caching” Application Cache =================
+--------------------------------------------------+ | Application Server | +--------------------------------------------------+ | | | +------------------+ +------------------+ | | | In-Memory | | Distributed | | | | Cache | | Cache | | | | (Local) | | (Redis/Mem- | | | | | | cached) | | | +------------------+ +------------------+ | | | +--------------------------------------------------+
Examples: - In-memory: Guava Cache, Ehcache - Distributed: Redis, Memcached5.2.4 Database Caching
Section titled “5.2.4 Database Caching” Database Caching =================
+--------------------------------------------------+ | Database | +--------------------------------------------------+ | | | +------------------+ +------------------+ | | | Query Cache | | Buffer Pool | | | | (SQL results) | | (Data pages) | | | +------------------+ +------------------+ | | | +--------------------------------------------------+
Example: MySQL InnoDB Buffer Pool5.3 Cache Strategies
Section titled “5.3 Cache Strategies”5.3.1 Cache-Aside (Lazy Loading)
Section titled “5.3.1 Cache-Aside (Lazy Loading)” Cache-Aside Pattern ===================
Application Cache Database | | | | 1. Check cache | | |------------------------>| | | 2. Cache miss | | |<------------------------| | | | 3. Query DB | | |---------------------->| | | 4. Return data | | |<----------------------| | 5. Store in cache | | |------------------------>| | | | | | 6. Return data | | |<------------------------| |
Implementation: ```python def get_data(key): data = cache.get(key) if data is None: data = db.query(key) cache.set(key, data) return data ```
Pros: Simple, Only requested data is cached Cons: Cache miss penalty, Stale data possible5.3.2 Write-Through
Section titled “5.3.2 Write-Through” Write-Through Pattern =====================
Application Cache Database | | | | 1. Write data | | |------------------------>| | | 2. Write to DB | | | |---------------------->| | 3. Confirm | | |<------------------------| | | | |
Pros: Always consistent, Fast reads after write Cons: Slower writes, Wasted resources for unwritten data5.3.3 Write-Behind (Write-Back)
Section titled “5.3.3 Write-Behind (Write-Back)” Write-Behind Pattern ====================
Application Cache Database | | | | 1. Write data | | |------------------------>| | | 2. Acknowledge | | |<------------------------| | | | (Async) | | | 3. Write to DB | | |---------------------->|
Pros: Fast writes, Batch DB operations Cons: Risk of data loss if cache fails5.3.4 Refresh-Ahead
Section titled “5.3.4 Refresh-Ahead” Refresh-Ahead Pattern =====================
TTL: 60 seconds Refresh: 80% of TTL (48 seconds)
Timeline: 0s 10s 20s 30s 40s 48s 50s 60s |-----|-----|-----|-----|-----|-----|-----| | | | | | | | | Data Data Data Data Data REFRESH Data Data cached cached
Pros: Reduces cache miss latency Cons: Complex to implement5.4 Cache Eviction Policies
Section titled “5.4 Cache Eviction Policies”Common Eviction Algorithms
Section titled “Common Eviction Algorithms”| Policy | Description | Use Case |
|---|---|---|
| LRU (Least Recently Used) | Evict least recently accessed | General purpose |
| LFU (Least Frequently Used) | Evict least frequently accessed | Popular items |
| FIFO (First In First Out) | Evict oldest | Simple cases |
| TTL (Time To Live) | Evict after expiration | Time-sensitive data |
| Random | Evict random | Testing |
LRU (Least Recently Used) ==========================
Access Order: A -> B -> C -> D -> E -> A
Before: +---+---+---+---+ | A | B | C | D | (Full) +---+---+---+---+
Access E (new): +---+---+---+---+ | B | C | D | E | (A evicted) +---+---+---+---+5.5 TTL (Time To Live)
Section titled “5.5 TTL (Time To Live)” TTL Implementation ===================
Cache Entry: +----------------------------------+ | Key: "user:123" | | Value: {name: "John", ...} | | TTL: 300 seconds | | Created: 2024-01-15 10:00:00 | +----------------------------------+
TTL Strategies: +------------------+----------+------------------------+ | Data Type | TTL | Reason | +------------------+----------+------------------------+ | User session | 24 hours | Session expiry | | Product catalog | 1 hour | Inventory changes | | Configuration | 1 day | Infrequently changes | | API response | 5 min | Real-time data needed | | Static assets | 1 week | CDN caching | +------------------+----------+------------------------+5.6 Cache Invalidation
Section titled “5.6 Cache Invalidation”The Big Challenge
Section titled “The Big Challenge” Cache Invalidation ==================
"There are only two hard things in computer science: cache invalidation and naming things." - Phil Karlton
Problem: How to keep cache consistent with database?
Scenarios: 1. Update data -> Invalidate cache 2. Delete data -> Invalidate cache 3. Data expires -> Natural expirationInvalidation Strategies
Section titled “Invalidation Strategies” Strategy 1: Delete on Write ===========================
User updates profile -> Delete cached profile | v Next read -> Cache miss -> Fetch from DB -> Cache again
Strategy 2: Event-Based Invalidation ====================================
Database -> Event -> Cache Service Update Bus | Invalidate Cache
Strategy 3: Versioning ====================
Key: "user:123:v2" Update -> Increment version (v2)
Old clients still use v1 (eventually expires) New clients use v25.7 Distributed Caching
Section titled “5.7 Distributed Caching”Redis Cluster Architecture
Section titled “Redis Cluster Architecture” Redis Cluster =============
Client | v +------------------+ | Redis Proxy | (Route requests) +------------------+ | +---+---+---+---+---+ | | | | | | v v v v v v P0 P1 P2 P3 P4 P5 (16384 hash slots) | | | | | | v v v v v v S0 S1 S0 S1 S0 S1 (Primary/Replica)
Data Distribution: Key "user:123" -> Hash -> Slot -> NodeMemcached vs Redis
Section titled “Memcached vs Redis”| Feature | Memcached | Redis |
|---|---|---|
| Data Types | Strings | Strings, Lists, Sets, Hashes, Sorted Sets |
| Persistence | No | RDB, AOF |
| Clustering | Memcached SaaS | Native Cluster |
| Performance | Very Fast | Fast |
| Memory | LRU | Custom |
| Use Case | Simple caching | Caching + more |
5.8 Cache Problems
Section titled “5.8 Cache Problems”5.8.1 Cache Miss
Section titled “5.8.1 Cache Miss” Cache Miss Types ================
1. Cold Start / Cold Cache - First request after deployment - All requests hit database - Solution: Pre-warm cache
2. Eviction - Old data removed to make room - Normal behavior - Solution: Adjust cache size
3. Key Not Found - Data never cached - Solution: Check cache logic5.8.2 Cache Stampede
Section titled “5.8.2 Cache Stampede” Cache Stampede Problem ======================
1. Many requests hit cache miss simultaneously 2. All query the database at the same time 3. Database overwhelmed
Timeline: Request1 ----> Cache Miss -> DB Query Request2 ----> Cache Miss -> DB Query Request3 ----> Cache Miss -> DB Query ... Request100 --> Cache Miss -> DB Query | v Database Crashes!
Solution: Distributed Lock or Request Coalescing ================================================
Request1 ----> Cache Miss -> Lock -> DB Query -> Cache Set -> Release Request2 ----> Cache Miss -> Wait -> Wait -> Wait -> Cache Hit! Request3 ----> Cache Miss -> Wait -> Wait -> Wait -> Cache Hit!5.8.3 Thundering Herd
Section titled “5.8.3 Thundering Herd” Thundering Herd ===============
Same as Cache Stampede but on cache EXPIRY
TTL expires for key "popular_product" All 1000 users request simultaneously All hit database at same time
Solution: Jitter + Extension ============================
TTL = 60 seconds Random jitter = 0-20 seconds Actual TTL = 60-80 seconds (random)
Each instance has different TTL Reduces simultaneous expiry5.8.4 Stale Data
Section titled “5.8.4 Stale Data” Stale Data Problem ==================
User changes password Cache still has old password User can't login until cache expires
Solutions: 1. Write-through (update cache on write) 2. Event-driven invalidation 3. Short TTL for sensitive data 4. Versioned cache keys5.9 Multi-Level Caching
Section titled “5.9 Multi-Level Caching” Multi-Level Cache Architecture ===============================
Request | v +--------------------------------------------------+ | L1 Cache (Local) | | - In-memory, fast | | - Small size (GB) | +--------------------------------------------------+ | (miss) v +--------------------------------------------------+ | L2 Cache (Distributed) | | - Redis/Memcached | | - Larger size (TB) | +--------------------------------------------------+ | (miss) v +--------------------------------------------------+ | Database | | - Persistent storage | | - Slowest | +--------------------------------------------------+
Performance: L1: 0.1ms L2: 1ms DB: 100ms5.10 Caching Best Practices
Section titled “5.10 Caching Best Practices”Do’s and Don’ts
Section titled “Do’s and Don’ts”| Do | Don’t |
|---|---|
| Use for read-heavy workloads | Use for frequently changing data |
| Set appropriate TTLs | Cache without TTL |
| Monitor cache hit rate | Cache everything |
| Handle cache failures gracefully | Let cache failure crash app |
| Pre-warm after deployment | Start with empty cache |
| Use consistent hashing in clusters | Cache without eviction policy |
Summary
Section titled “Summary”Key caching concepts:
- Choose the right strategy - Cache-aside, write-through, or write-behind
- Set appropriate TTLs - Balance freshness with performance
- Handle invalidation - Keep cache consistent with database
- Prevent stampedes - Use locks or request coalescing
- Monitor metrics - Hit rate, miss rate, latency
- Plan for failures - Cache can fail; design gracefully