Scenario: The Double-Billed Customer
A company has a microservice responsible for processing monthly subscription renewals. Because the service is horizontally scaled across 5 Kubernetes pods, they need to ensure the billing job runs strictly once per day. A developer uses Redis to implement a lock: before running the job, the pod executes SETNX billing_lock 1. If it succeeds, it bills the customers; if it fails, it skips the job.
Q:On Tuesday, Pod A acquires the lock and begins processing. Halfway through, the underlying EC2 instance suffers a hardware failure and Pod A instantly dies. What happens to the billing system on Wednesday? Reveal â–¾
The system experiences a distributed Deadlock.
Because Pod A died violently, it never executed the DEL billing_lock command. The lock remains in Redis indefinitely. On Wednesday, when all 5 pods try to acquire the lock, they will all receive a failure response. The billing job will never run again until an engineer manually SSHs into the production database and deletes the orphaned key.
Q:To fix this, the developer adds a Time-To-Live (TTL). They execute SET billing_lock 1 EX 60 NX, giving the lock a 60-second expiration. On Thursday, Pod A acquires the lock. However, the database is sluggish, and the billing job takes 75 seconds to complete. What catastrophic business event just occurred?
Reveal â–¾
The customers were billed twice.
At the 60-second mark, Redis automatically deletes the lock because the TTL expired. However, Pod A is still alive and actively processing the billing job. At second 61, Pod B wakes up, tries to acquire the lock, and succeeds (because the key was deleted). Now, Pod A and Pod B are executing the exact same critical section simultaneously. The TTL designed to prevent a deadlock just caused a massive race condition.
Q:The developer argues that they can just increase the TTL to 10 minutes. You point out that a JVM Garbage Collection pause or a network partition could arbitrarily freeze Pod A for 11 minutes. How do you definitively prevent Pod A from corrupting the database if it wakes up after its lock has expired? Reveal â–¾
You must implement Fencing Tokens.
A distributed lock cannot rely solely on time; it needs cryptographic or monotonic enforcement at the storage layer. When Pod A acquires the lock, the lock manager assigns it a monotonically increasing token (e.g., Token = 33). Pod A passes this token to the database with every UPDATE command.
If Pod A pauses and its lock expires, Pod B acquires the lock and gets Token = 34. Pod B updates the database. When Pod A finally wakes up and attempts to write with Token = 33, the database rejects the transaction because it has already processed a write with a higher token number. The lock is enforced by the database, not the application.
Q:The team implements Fencing Tokens, but the Redis Master node suddenly crashes. Redis quickly promotes a Replica to become the new Master. Pod A had the lock, but now Pod B acquires the exact same lock from the new Master. How did Redis allow this? Reveal â–¾
This is the Split-Brain problem caused by asynchronous replication.
When Pod A acquired the lock on the Master, the Master returned a success message immediately, before replicating the key to the Replica. When the Master crashed, the Replica was promoted, but it never received the lock data. It appears completely empty. When Pod B requests the lock, the new Master happily grants it.
To solve this in a distributed cache, you cannot use a single node. You must use a consensus algorithm (like Paxos or Raft) or a specialized distributed lock algorithm like Redlock, which requires the client to acquire the lock on a majority (quorum) of independent Redis nodes simultaneously.
Variations & Real-World Impact
- Chubby & Zookeeper: Big Tech companies rarely use Redis for highly critical distributed locks. Google built Chubby, and the open-source world uses Apache Zookeeper or etcd. These systems use strict consensus protocols (like Multi-Paxos or Raft) to guarantee linearizability, ensuring that a lock is never lost during leader elections, sacrificing write latency for absolute correctness.
- Clock Drift: If you attempt to build your own distributed lock using physical timestamps across nodes, NTP clock drift will inevitably cause nodes to disagree on when a TTL actually expired, silently breaking the lock’s mutual exclusion guarantees.
Discussion & Comments