All Posts
System Design 11 min read

Lessons from designing an event-driven transaction pipeline with Kafka

#kafka#distributed-systems#event-driven

Moving from synchronous REST APIs to an asynchronous event-driven architecture using Apache Kafka is like trading a bicycle for a jet engine: incredible throughput, but if something goes wrong, you crash at 500 miles per hour. Here are the hard lessons from processing 40 million financial transaction events per day.

The partition key mistake and message ordering

Kafka guarantees strict message ordering only within a single partition. Early on, a developer partitioned events by random transaction UUID instead of `account_id`.

As a result, a "Deposit $100" event and a "Withdraw $80" event for the same customer landed on different partitions and were processed concurrently. The withdrawal worker processed first and rejected the transaction for insufficient funds. Always partition by the entity ID whose business invariants must remain strictly ordered.

Surviving poison pill messages with Dead Letter Queues (DLQ)

A malformed event containing unparseable characters once caused a consumer to crash. Because the offset was never committed, the consumer restarted, read the same poison pill, and crashed again in an infinite loop, halting all transaction processing.

We implemented a resilient retry pattern:

- Retry processing transient errors up to three times with exponential backoff - If deserialization or business invariants fail permanently, commit the offset and route the failed record to a Dead Letter Topic (DLT) with error metadata for manual triage

Consumer group rebalancing hell

If a consumer takes longer than `max.poll.interval.ms` to process a batch of events (e.g. during a slow third-party API call), the Kafka broker assumes the consumer died and triggers a group rebalance. This stops the world for all other consumers in the group.

We decoupled message fetching from heavy processing using an in-memory worker pool, ensuring the poll loop always commits heartbeats on time.

Have a similar challenge?

We build production-grade software for companies that need it done right.

Let's Talk