System Design: The Indestructible Payment Gateway

1. Problem Statement & Scope

Building a payment gateway is fundamentally different from building a social network. If a tweet drops, nobody cares; if a user is double-charged for a massive block of Nifty 500 stocks during an IPO allotment, you face catastrophic financial and legal ruin. The engineering focus shifts entirely from eventual consistency and raw throughput to absolute, immutable correctness.

Functional Requirements

• Payment Execution: Process credit cards, net banking, and UPI transactions securely.
• Idempotency: Guarantee that a duplicated request across a flaky network never results in a double charge.
• Asynchronous Webhooks: Notify merchants reliably when a transaction clears, fails, or is refunded.

Non-Functional Requirements

• Strict ACID Compliance: Financial ledgers cannot tolerate race conditions or lost updates.
• Immutability: Data can never be UPDATED or DELETED. Errors are corrected via compensating transactions (append-only).
• High Availability: The gateway must continue accepting payment intents even if downstream banking networks are experiencing downtime.

2. Back-of-the-Envelope Estimation

• Throughput: 10,000 Transactions Per Second (TPS) peak.
• Payload Size: ~2KB per transaction record (including metadata, cryptographic signatures, and routing info).
• Storage: 10,000 TPS $\times$ 2KB $\times$ 86,400 seconds = ~1.7 TB/day. Over a year, this is roughly 600 TB. We will need cold-storage archiving for auditing, but hot storage must be exceptionally fast.
• Latency: The gateway must respond to the client in $< 200$ms, even if the actual bank settlement takes days.

3. High-Level Design (HLD)

4. Deep Dive: Component Design

The Idempotency Layer

A mobile client sits on a train edge network. They click “Buy”. The request reaches the server, the database commits the transaction, and the server replies HTTP 200 OK. However, the train goes through a tunnel, and the client never receives the response. The client reconnects and retries the exact same request. How do you prevent charging them twice?

Every request must include a unique Idempotency-Key in the HTTP header. Before the Payment Orchestrator does anything, it checks this key in a fast key-value store (Redis) or the primary database.

• If the key exists and the transaction is PENDING, the system drops the request and tells the client “Processing”.
• If the key exists and the transaction is SUCCESS, it returns the exact saved HTTP response from the first successful attempt.

Question to ponder: What happens if two identical requests arrive at the exact same millisecond before the Redis key is fully replicated? How do you architect a distributed lock on the idempotency key without bottlenecking the entire cluster?

The Immutable Double-Entry Ledger

Financial systems do not use standard CRUD (Create, Read, Update, Delete). If a transaction fails, you do not UPDATE the status to ‘FAILED’.

You implement an Append-Only Double-Entry Ledger. Every transaction involves two accounts: a debit to the buyer and a credit to the merchant. The sum of all accounts in the system must always equal zero. If a payment is refunded, you append a new, mathematically inverted transaction. This allows auditors to reconstruct the exact state of the system at any nanosecond in history.

5. Fault Tolerance & Edge Cases

The Distributed Saga (The Two-Generals Problem in Finance)

The Payment Orchestrator saves the state as PENDING in our PostgreSQL ledger, then successfully calls the HDFC bank API to deduct funds. The bank succeeds. But as the HTTP response travels back to our server, our server loses power.

Our database says PENDING, but the user’s bank account has been debited. This is the nightmare scenario of distributed systems.

To solve this, we rely on a Reconciliation Engine. A background daemon constantly sweeps the ledger for PENDING transactions older than 5 minutes. It takes the transaction ID, queries the downstream bank’s API, and asks, “Did this clear?” If the bank says yes, the worker appends a SUCCESS record to our ledger. If the bank says “Transaction Not Found”, the worker appends a FAILED record.

Question to ponder: What if the bank’s API is also down? How long do you exponentially backoff before you give up? And if you give up, how do you mathematically prove to regulators that the money isn’t stuck in digital limbo?

The Thundering Herd of High-Frequency Trading

Imagine a scenario where thousands of retail investors attempt to buy into a wildly popular Nifty 50 stock simultaneously during a flash crash. Your payment gateway is suddenly hit with an extreme spike in write requests destined for the exact same merchant account.

If your database uses pessimistic row-level locks (SELECT ... FOR UPDATE), all 10,000 transactions will queue up trying to lock the merchant’s balance row to update it. The database will deadlock or connection-pool exhaust within seconds.

Question to ponder: How do you shatter this lock contention? Do you batch the credits asynchronously? Do you shard the merchant’s balance into 100 sub-accounts and randomly route the inbound payments, summing them up only on read? What are the architectural tradeoffs of eventual consistency when dealing with human money?