Event-driven Architecture (EDA) needs no introduction, as it’s increasingly being adopted across various domains. However, there are key enhancements that can further bolster the resilience of EDA systems.
In this blog, I will explore how the reliability and auditability of an Event-Driven Architecture (EDA) system can be enhanced by leveraging the Event Sourcing architectural pattern.
EDA : The Foundation of Reactive Systems
Event Driven Architecture(EDA) embraces the concept of events as primary communication mechanism within a systems. Instead of tightly coupled synchronous interactions, EDA relies on asynchronous messaging, where components publish and subscribe to events, reacting to changes in the system state. This approach offers several key advantages:
Decoupling : Components can evolve independently, reducing dependencies and facilitating parallel development.
Scalability : Systems can easily scale horizontally by adding more event processing nodes.
Resilience: Failures in one component have minimal impact on others, as they are loosely coupled and communicate through events.
Flexibility: New functionalities can be introduced without disrupting existing components, simply by subscribing to new events.
In Event-Driven Architecture (EDA), events are pivotal as they drive the flow and coordination of actions across the system. In some scenarios, it becomes necessary to persist events for an extended period, either for audit purposes or to reconstruct the system’s state in the event of a failure.
The Event Sourcing architectural pattern complements EDA by ensuring that every state change in the system is captured as an immutable sequence of events. This approach not only facilitates system recovery and state reconstruction but also provides a reliable audit trail for the events.
Event Sourcing Architectural Pattern
Event Sourcing is an architectural pattern where the state of a system is derived from a sequence of events rather than being stored directly. Instead of persisting the current state of an entity, every change (event) to the state is captured and stored in an immutable log. The state can be reconstructed by replaying these events in order. At the heart of full event sourcing architecture, the key principles are
Events drive state – The system’s state is not stored directly (e.g., in tables or models). Instead, the state is derived by processing or “replaying” the events.
Immutable event log – All events are stored in an append-only log or dedicated event store, providing a complete history of system.
Event replay – Historical states can be reconstructed or replayed to debug, audit, or rebuild current state after failures.
One might assume that storing the current state is equivalent to event sourcing, but that is not the case.
Event Sourcing vs. Storing Events in a Database
Event sourcing and storing events in a database may seem similar, but they serve fundamentally different purposes. Both of these can compliment EDA, in combination this can emerge as robust architecture.
Event Sourcing : Event sourcing , event log or a dedicated event store serves as primary source of truth for the system. The current state of the system is dynamically built by replaying all event. In addition, events are immutable and stored in dedicated event store, ensuring traceability and and the ability to recreate historical states. Eg : EventStoreDB
Storing Events in Database : The system’s current state is directly stored and maintained in a separate database or table, such as a relational or NoSQL database. This state represents the latest snapshot of the data and is updated as new events occur. The stored events are not used to reconstruct the state; they are kept only for auditing, logging, or analytics purposes. Eg : Any relational or NoSQL with well defined schema for a given event.
How Event Sourcing complements EDA
While Event-Driven Architecture focuses on decoupling communication between microservices through events, it relies heavily on message queues (like Kafka) to distribute events, which often have limited retention. These message queues are not designed for long-term storage or state recovery. Event Sourcing complements EDA by ensuring every event is persistently stored and can be replayed to rebuild the system’s state.
While full event sourcing introduces certain complexities (like computationally intensive state calculations), we can adopt a hybrid approach that selectively leverages event sourcing to enhance the robustness of Event-Driven Architecture (EDA). In this approach, only key events are stored in the database, serving as a reliable fallback for disaster recovery and fulfilling secondary needs such as audit trails or analytics.
Implement Event Storage Alongside EDA
Integrate with Message Queues: Use a database (e.g., EventStoreDB, Cassandra) to persist every event as an append-only record. When an event is published to a Kafka topic, also persist it in the event store.
Design for Querying and Replay: Replay requires a schema optimized for efficient access, which includes metadata such as event occurrence time and event type.
Use append-only table for storing events. Example schema.
| EventID | EventType | Payload (JSON) | Timestamp | Topic |
| 1 | Deposit | {"acc_num":123, "amt" : $1000} | 2024-12-20T12:00Z | Deposits |
| 2 | Withdrawal | {"acc_num":123, "amt" : $100} | 2024-12-24T12:00Z | Withdraws |
Reconstruction Logic: Implement logic to replay events in order to rebuilt the state of system when required.
Other storage best practices could be practically anything that we apply to traditional domain specific tables.
Partitioning and Indexing : Partition the table by event type or timestamp for better query performance. Index columns like EventType and Timestamp for fast lookups.
Retention Policies: Since the events stored are for secondary needs ( like auditing, debugging, analytics etc) , we can keep them indefinitely for disaster recovery, or implement archival solutions to move older events to cold storage (e.g., AWS Glacier).
Practical Use Case: Handling Consumer Lag with Event Sourcing
One challenge I repeatedly came across (until it is addressed )in our EDA projects was consumers struggling to process events quickly enough, resulting in data loss. This issue becomes particularly pronounced when the topic is managed by external systems.
To address this, a viable solution could be adopting an event-sourcing technique, where state changes in the system are stored as a series of events, where a service reads data from the topic and stores it in an event store in the order they appear in the queue. This approach not only ensures data is safeguarded for recovery or backfill but also allows the actual consumer to continue processing data from the topic without disruption.
Wrapping Up : EDA with Event Sourcing
In summary, combining Event-Driven Architecture (EDA) with Event Sourcing enhances system reliability and fault tolerance by capturing and storing every state change as an event. This approach allows for easy recovery and ensures data integrity. In the next post, we’ll explore how introducing Command Query Responsibility Segregation (CQRS) can further optimize the performance and scalability of EDA and Event Sourcing systems. Stay tuned!