Event-driven with Kafka: what you gain, and what you lose

On MissionMatch, Kafka event streaming decouples publishing a mission from every process that needs to react to it: notification, indexing, statistics. The decoupling benefit is real. What you lose in exchange is just as real, and it's rarely discussed with the same clarity.

The problem Kafka actually solves

Without event streaming, every new process triggered by a business event (a mission being published, for instance) requires modifying the service that publishes that event, to add a direct call to it. After a few processes, that service knows about all its consumers, which runs directly against the single responsibility principle. Publishing an event to a topic, and letting each consumer subscribe independently, reverses that dependency: the producing service doesn't know who consumes it, or how many do.

                          topic: mission.published
                    ┌────────────────────────────────┐
Mission Service ──▶ │  ████  ████  ████  ████  ████  │
 (producer)         └────────────────────────────────┘
                        │           │           │
                        ▼           ▼           ▼
                 Notification    Indexing     Statistics
                  (consumer)    (consumer)    (consumer)

Each consumer moves at its own pace, replays independently from the others on failure, and can be added or removed without touching the producing service or any other consumer.

What that decoupling costs you

The price of that flexibility is losing guarantees you took for granted with a direct synchronous call. A classic function call fails visibly and immediately. A Kafka message can be processed late, duplicated (at-least-once, not exactly-once, absent specific configuration), or processed out of order depending on the chosen partitioning. These aren't bugs: they're part of the system's contract, but they need to be designed for explicitly, not discovered in production.

What actually changes, property by property

Property	Direct synchronous call	Kafka event
Delivery	Guaranteed or immediate exception	At least once (the consumer must handle duplicates)
Ordering	Calling code's execution order	Guaranteed per partition, not globally
Perceived latency	Immediate, blocking	Asynchronous, the producer doesn't know the processing outcome
Startup coupling	The called service must be available	The consumer can be stopped, it catches up on return

Every row in that table is an explicit trade-off, not a hidden flaw in the system: accepting "at least once" over "exactly once" is a design choice that shifts the responsibility for idempotence onto the consumer, not an implementation oversight.

The question that determines whether Kafka is justified

Before introducing Kafka on a project, the question to ask isn't "could this be useful", it's "do I actually have several independent consumers that need to react to the same event, at different processing rhythms". If the answer is a single synchronous consumer, a classic function call remains simpler, easier to test, and easier to evolve. Kafka is justified when decoupling solves a real organizational or scale problem, not as a default choice for "modern architecture."

What this implies for testing

On MissionMatch, every event consumer is tested independently from the producer, with an event built directly in memory rather than going through a real Kafka broker in unit tests. The same hexagonal discipline applies here: the domain shouldn't know it communicates via Kafka, only that it publishes an event through an interface. The concrete broker stays an infrastructure detail, substitutable, testable in isolation.