Architecture and Scale

Event-Driven Architecture

Instead of services calling each other directly, they react to events. Event-driven architecture enables loose coupling and scale, but makes reasoning about the system harder.


The Concept Explained

In a request-driven system, a service that needs something calls the service that provides it. The caller knows the callee, waits for a response, and fails if the callee is down.

In an event-driven system, a service that has done something publishes a fact and moves on. Other services subscribe and react. The publisher does not know who is listening, does not wait, and is unaffected if a subscriber is unavailable.

The distinction between events and commands is worth getting exactly right, because it is where most confusion in this area originates.

A command is an instruction to a specific recipient: ChargePayment. It expresses intent, is directed at one handler, can be rejected, and the sender usually cares about the outcome.

An event is a statement of fact about something that already happened: OrderPlaced. It is past tense, it is addressed to nobody in particular, and it cannot be rejected because the thing already occurred. Whether anyone reacts is not the publisher's concern.

KEY CONCEPT

Event-driven architecture inverts the direction of knowledge. In request-driven systems the caller knows its dependencies, so adding a consumer means changing the caller. In event-driven systems the consumer knows the producer, so adding a consumer changes nothing upstream. That inversion is the entire source of the loose coupling, and also the entire source of the difficulty in understanding what the system does.


How It Works

An Order Flow: Direct Calls vs Published Events

Request-driven

Order service calls each dependency

Order service knowsInventory, payment, shipping, email
Adding a consumerChange and redeploy the order service
If payment is downThe order fails
AvailabilityProduct of every service in the chain
LatencySum of all downstream calls
Tracing the flowRead the order service code
Outcome knownImmediately, in the response
Event-driven

Order service publishes OrderPlaced

Order service knowsNothing about consumers
Adding a consumerDeploy the new consumer alone
If payment is downEvent waits, processed on recovery
AvailabilityOrder accepted if the broker is up
LatencyPublish and return
Tracing the flowFind every subscriber, across services
Outcome knownEventually, via further events

The Benefits Are Concrete

Adding consumers costs nothing upstream. A fraud check, an analytics pipeline, or a loyalty points service can subscribe to OrderPlaced without the order service knowing or changing. In a request-driven system each of those would be another call added to the order service and another dependency in its availability chain.

Producer availability decouples from consumer availability. If the shipping service is down, orders are still accepted; the events queue and are processed when it returns. In the request-driven version the order fails because a downstream service is unavailable.

Load absorbs into the queue. A traffic spike lengthens the queue rather than overwhelming consumers, and consumers process at their own rate. Backpressure is inherent instead of being something you build.

Availability arithmetic improves substantially. A synchronous chain multiplies availabilities: five services at 99.9% each gives 99.5%. Moving four of them off the critical path means the user-facing operation depends on the order service and the broker only. This is frequently the strongest argument for the pattern and the one candidates most often omit.

The Costs Are Equally Concrete

No single place describes the flow. The order service code shows a publish and nothing else. Understanding what happens after an order requires finding every subscriber across every repository. New engineers cannot read the flow; they have to discover it.

Debugging spans systems and time. When an order does not ship, the question is which stage failed, and answering it means correlating logs across several services and a broker. Without a correlation identifier threaded through every event this is close to impossible, so distributed tracing is a precondition rather than an enhancement.

Everything becomes eventually consistent. The order exists before the inventory is decremented. Reads in that window see an inconsistent picture, and the system must model those intermediate states honestly rather than pretending the operation was instantaneous.

Ordering is not free. Events about the same entity usually need to arrive in order, and brokers typically guarantee ordering only within a partition. Partitioning by entity id gives per-entity ordering, which is what most systems actually need, but it must be a deliberate choice.

Duplicates are guaranteed. Practical messaging is at-least-once, so consumers must be idempotent. A consumer that is not idempotent will eventually double-charge or double-ship, rarely enough to survive testing.

WARNING

Publishing an event and separately committing to your database is the dual-write problem, and it is the most common correctness bug in event-driven systems. Commit then publish, and a crash in between loses the event silently while the state change is durable. Publish then commit, and consumers react to something that never happened. The fix is the transactional outbox: write the event into a table in the same local transaction as the state change, and let a relay publish it afterwards.


System Design Implications

Designing the e-commerce order flow, which is the interview prompt, has a defensible shape.

Publish OrderPlaced from the order service, in the same transaction as the order row, via an outbox. The order service now depends only on its own database and returns to the user immediately.

Let inventory, payment, notification, and analytics subscribe independently. Each processes at its own pace and its failures do not affect order acceptance.

Publish further events as each stage completes. PaymentCaptured, InventoryReserved, OrderShipped. Downstream stages subscribe to those rather than being called.

Model the order status explicitly. Pending, confirmed, shipped, failed. These are real business states, and exposing them honestly is better than pretending the operation is atomic.

Handle the failure path deliberately. A payment failure publishes PaymentFailed, which triggers compensation: release the inventory reservation, mark the order failed, notify the customer. This is a saga, and event-driven architecture is the natural transport for one.

Carry a correlation identifier through every event, so the whole flow can be reconstructed from logs. Without it, the debugging cost described above becomes the dominant operational problem.

Use events for facts and direct calls for queries. Asking "what is this customer's current balance" is a request, not an event. Event-driven does not mean everything must be asynchronous, and systems that force queries through events become far harder than they need to be.

PRO TIP

Answer the debugging half of the question with a specific reason rather than "it is harder to trace." No single artifact describes the flow: the producer code shows only a publish, so understanding what happens next means finding every subscriber across every service. That, plus the fact that a failure may surface minutes later in a different system, is what makes correlation identifiers and distributed tracing mandatory rather than nice to have.


Tradeoffs and Decision Framework

DimensionRequest-drivenEvent-driven
CouplingCaller knows calleesProducer knows nobody
Adding a consumerChange the callerDeploy the consumer alone
AvailabilityProduct of the chainProducer plus broker
Latency to userSum of downstream callsPublish and return
ConsistencyImmediateEventual
Outcome visibilityIn the responseVia later events
DebuggingOne call pathCorrelation across systems
OrderingInherentPer partition, by design
DuplicatesRetry-dependentGuaranteed, need idempotency

The framework in four questions. Does the caller genuinely need the result to respond, since if it does, that call should stay synchronous? Will the number of interested parties grow, which is where the coupling inversion pays off most? Can the business tolerate the intermediate states, and can they be modelled explicitly? And does the operational capability exist, meaning a broker you can run, correlation identifiers, and tracing?

A sensible default is hybrid: synchronous for the query path and for the few operations whose result the user is waiting on, events for everything downstream of a completed fact. Systems that go fully asynchronous frequently rediscover that some questions really do need an immediate answer.


Common Mistakes

Publishing commands as events. SendEmail addressed to one consumer is a command wearing an event's clothes, and it recreates the coupling the pattern was meant to remove.

The dual-write problem. Committing to the database and publishing separately loses events or emits phantom ones. Use an outbox.

Non-idempotent consumers. At-least-once delivery means duplicates arrive eventually, and a consumer that cannot absorb them will double-process.

Assuming global ordering. Brokers order within a partition. Per-entity ordering requires partitioning by entity id deliberately.

No correlation identifier. Without one, reconstructing a flow across services and time is close to impossible.

Events carrying too little or too much. Too little and every consumer calls back to the producer, restoring the coupling. Too much and the event exposes internal schema and every consumer is coupled to it.

No dead letter handling. One poison message can block a partition indefinitely if there is nowhere for it to go.

Making queries asynchronous. Requests for current state should be requests. Forcing them through events adds latency and complexity for nothing.


INTERVIEW QUESTION

Design an event-driven architecture for an e-commerce order flow. What are the benefits over direct service calls, and what makes it harder to debug?