Scenario: The Ghost Order
An e-commerce architecture relies on microservices. When a user clicks “Buy,” the Order Service must do two things:
- Save the new order to its local PostgreSQL database.
- Publish an
OrderCreatedevent to a Kafka topic so the Shipping Service knows to dispatch the item.
Q:The developer writes the code sequentially: first, commit the database transaction, then publish the event to Kafka. What catastrophic business failure occurs if the Kafka cluster goes offline for 5 seconds right after the database commits? Reveal â–¾
This is the classic Dual-Write Problem.
Because the database committed successfully, the user sees “Order Confirmed.” However, because Kafka was offline, the OrderCreated event was never published. The system is now in an inconsistent state: the Order exists, the user’s money is taken, but the Shipping Service is completely unaware. The item will never ship.
Q:To prevent the order from being lost, the developer reverses the logic: publish the event to Kafka first, and only if that succeeds, commit the database transaction. What goes wrong now if the database happens to hit a unique constraint violation and rolls back? Reveal â–¾
You have created a Ghost Event.
Kafka is an append-only log; once you publish a message, you cannot easily “take it back.” If the database transaction rolls back, the Order does not exist in the primary system. However, the Shipping Service has already consumed the OrderCreated event from Kafka. It will attempt to ship a product for an order that never officially existed, potentially shipping free items.
Q:You cannot use a Distributed Transaction (like Two-Phase Commit) because Kafka doesn’t support it natively with PostgreSQL. How do you guarantee absolute, atomic consistency between a relational database and a message broker? Reveal â–¾
You implement the Transactional Outbox Pattern.
Instead of talking to Kafka directly during the API request, the Order Service creates an outbox table inside its own PostgreSQL database. When a user buys an item, the application writes the order to the orders table AND writes the serialized event payload to the outbox table within a single, local ACID database transaction.
If the database commits, both exist. If it rolls back, neither exists.
Separately, a background worker (or a Change Data Capture tool like Debezium) constantly polls the outbox table. It reads the events, reliably publishes them to Kafka with retries, and then marks them as processed in the database.
Q:The Outbox Pattern is working perfectly. However, the downstream Shipping Service encounters a malformed ‘OrderCreated’ event that is missing a crucial zipcode field. Every time the Shipping Service tries to process it, it throws a NullPointerException and crashes. It restarts, pulls the same message again, and crashes again. How do you fix this? Reveal â–¾
This is a Poison Pill message causing an infinite retry loop, entirely halting the partition.
To resolve this without losing data, you must implement a Dead Letter Queue (DLQ). The Shipping Service must be wrapped in a retry policy (e.g., attempt processing 3 times). If the processing fails on the 3rd attempt, the service explicitly catches the final exception, routes the malformed message into a separate, dedicated Kafka topic (the DLQ), and crucially, acknowledges (ACKs) the original message in the main queue.
This unblocks the primary Kafka partition, allowing the rest of the legitimate orders to be shipped. Later, engineers can manually inspect the DLQ, fix the zipcode bug, and replay the message.
Variations & Real-World Impact
- Change Data Capture (CDC): Polling the
outboxtable withSELECT * FROM outbox WHERE processed = falsecan put severe read load on the database. Modern architectures use CDC tools (like Debezium) which hook directly into the database’s internal transaction log (e.g., PostgreSQL’s WAL). They stream outbox inserts directly to Kafka with near-zero overhead. - Idempotency: Because the Outbox worker might publish to Kafka, crash before marking the database as processed, and then restart, it operates on an At-Least-Once delivery guarantee. Downstream services MUST be idempotent (e.g., checking if an
order_idwas already shipped before processing) to handle identical duplicate messages safely.
Discussion & Comments