Practical Event-Driven Microservices Architecture: Building Sustainable and Highly Scalable Event-Driven Microservices

Table of Contents....5

About the Author....12

About the Technical Reviewer....13

Acknowledgments....14

Introduction....15

Chapter 1: Embracing Event-Driven Architectures....17

1.1 The Truth About Monoliths....20

1.1.1 Anatomy of a Typical Monolith....20

Patchwork Monoliths....21

Modular Monoliths....23

1.1.2 What They Don’t Tell You About Monoliths; It’s Not All Bad....25

Business Flow Is Visible....25

No Network Overhead and Limited External Dependencies....26

Local Validation....26

Code Reuse....26

Monitoring and Troubleshooting....27

End-to-End Testing....27

Simpler Deployment Strategy....28

Data Is Centralized....28

Possible to Scale....28

Consistency....28

Concurrency....29

1.1.3 When Monoliths Become the Business Constrictor Knot....29

Coupling and Lack of Boundaries....29

Team’s Autonomy....30

Release Cycle....30

Scaling....30

Outdated Technology Stack....31

Reliability....32

1.1.4 Using Event-Driven Architectures to Move Away from a Monolith....32

1.2 What Are Microservices and How Do They Relate to Event-Driven....36

1.2.1 Deployment....37

1.2.2 Decoupled and Autonomously Developed....38

1.2.3 Data Ownership....38

1.3 SOA, Microservice, and Event-Driven Architectures....38

1.3.1 SOA....39

1.3.2 Microservice Architecture....40

1.3.3 Event-Driven Microservice Architecture....42

1.4 The Promise of Event-Driven Microservices....43

1.4.1 Evolving Architecture....44

1.4.2 Deployment....44

1.4.3 Team’s Autonomy....45

1.4.4 Flexible Technology Stack....46

1.4.5 Resilience and Availability....46

1.4.6 Tunable Scaling....48

1.4.7 The Past on Demand....48

1.5 When Should You Use Event-Driven Microservices?....49

1.6 Overview of the Challenges in Event-Driven Architectures....52

1.7 Summary....54

Chapter 2: Moving from a Monolith to an Event-Driven Architecture....56

2.1 Is Migrating to an Event-Driven Architecture Your Best Option?....58

2.2 Moving to an Event-Driven Architecture, How to Decide Where to Start....61

2.3 Using an Event-Driven Approach to Move Data from a Monolith....67

2.4 Using Change Data Capture (CDC) to Move Data from a Monolith....73

2.4.1 Event-Driven and Change Data Capture (CDC), a Real-World Example....77

CDC Example Using SQL Server and Kafka....79

2.5 Migrating Data from a Monolith: Event-Driven As a Source of Truth for Both Systems....83

2.6 Incremental Migration from a Monolith to an Event-Driven Architecture: Managing Dependencies....85

2.6.1 Managing a Dependency from a New Event-Driven Service to a Legacy Monolith....85

2.6.2 Managing a Dependency from a Legacy Application to a New Event-Driven Service....88

2.7 Gradually Moving Traffic to New Microservices....91

2.8 Migrating from a Monolith: Two-Way Synchronization and Living with Two Sources of Truth....94

2.9 Summary....97

Chapter 3: Defining an Event-Driven Microservice and Its Boundaries....99

3.1 Building Event-Driven Microservices....101

3.1.1 N-Tier Architectures....102

3.1.2 Clean Architecture....103

3.1.3 Event-Driven Microservices: Durable vs. Ephemeral Message Brokers and GDPR....108

The Right to Be Forgotten and Other Security Concerns....109

3.1.4 Event-Driven Message Types....112

3.1.5 Event-Driven Microservices: When to Use Documents over Events....116

3.1.6 Common Event-Driven Messaging Patterns....118

Send/Receive Pattern....119

Publish/Subscribe Pattern....119

Request/Response Pattern....120

3.1.7 Event-Driven Service Topologies....121

3.1.8 Common Event-Driven Pitfalls and Anti-patterns....125

Faking Synchronous Responses....125

Command Publishing....126

Passive-Aggressive Events....126

3.2 Organizing Event-Driven Microservice Boundaries....127

3.2.1 Organizational Composition....128

3.2.2 Likelihood of Changes....129

3.2.3 Type of Data....129

3.3 Brief and Practical Introduction to Domain-Driven Design and Bounded Contexts....131

3.3.1 How We Can Apply It in Practice....132

3.4 Event-Driven Microservices: The Impact of Aggregate Size and Common Pitfalls....136

3.5 Request-Driven vs. Event-Driven Services....138

3.6 Deciding When to Create a New Microservice or Add Functionality to an Existing One....141

3.7 Summary....144

Chapter 4: Structural Patterns and Chaining Processes....146

4.1 The Challenges of Transactional Consistency in Distributed Systems....148

4.1.1 Why Move from a Monolithic Database in the First Place?....150

4.1.2 The Limitations of Distributed Transactions....151

4.1.3 Managing Multi-step Processes with Sagas....156

4.2 Event-Driven Orchestration Pattern....159

4.3 Event-Driven Choreography Pattern....163

4.4 Event-Driven Microservices: Orchestration, Choreography, or Both?....167

4.5 Data Retrieval in Event-Driven Architectures and Associated Patterns....169

4.5.1 CQS, CQRS, and When to Use Them....173

When to Use CQRS?....176

4.5.2 The Different Flavors of CQRS....177

4.5.3 When and How to Use Event Sourcing....179

4.5.4 Concerns and When to Use Event Sourcing....184

4.5.5 Using Command Sourcing and Its Applicability....185

4.6 Building Multiple Read Models in Event-Driven Microservice Architectures....186

4.7 The Pitfall of Microservice Spaghetti Architectures and How to Avoid It....192

4.7.1 Domain Segregation and Clear Boundaries....193

4.7.2 Context Maps....195

4.7.3 Distributed Tracing....197

4.8 Summary....198

Chapter 5: How to Manage Eventual Consistency....200

5.1 The Impacts of Eventual Consistency and the Need for Alignment with the Business....203

5.1.1 Safety....206

5.1.2 Liveliness....206

5.1.3 The CAP Theorem in the Real World....208

5.2 Using Event Schema in Event-Driven Microservices to Leverage Eventual Consistency....212

5.3 Applying Microservice Domain Boundaries to Leverage Eventual Consistency....217

5.4 Handling Eventual Consistency Delays with Event Versioning....222

5.5 Saving State in Event-Driven Microservices to Avoid Eventual Consistency....224

5.5.1 Buffering State As an Alternative to Persistence....226

5.6 Tackling Eventual Consistency with the End-to-End Argument: A Real-World Use Case....228

5.7 For Most Use Cases, It’s Not Eventual If Nobody Notices....231

5.7.1 Event-Driven Autoscaling Use Case with Prometheus and Kafka....233

5.8 Discussing the Tradeoffs of Typical Eventual Consistency Handling Strategies....235

5.9 Summary....237

Chapter 6: Dealing with Concurrency and Out-of-Order Messages....239

6.1 Why Is Concurrency Different in a Monolith from an Event-Driven Architecture?....241

6.2 Pessimistic vs. Optimistic Concurrency, When and When Not to Use....248

6.2.1 Pessimistic vs. Optimistic Approaches....249

6.2.2 Solving Concurrency by Implementation and by Design....250

6.3 Using Optimistic Concurrency....252

6.4 Using Pessimistic Concurrency....256

6.4.1 Distributed Locks in Event-Driven Microservices....257

6.4.2 Database Transactions As a Concurrency Approach in Distributed Microservices....262

6.5 Dealing with Out-of-Order Events....263

6.5.1 How Can Events Lose Their Order?....264

6.5.2 Solving Out-of-Order Events with Versioning....266

6.6 Using End-to-End Message Partitioning to Handle Concurrency and Guarantee Message Ordering....268

6.6.1 Real-World Example of Event-Driven Message Routing Using Kafka....270

6.6.2 The Relevance of Message Routing and Partitioning in Event-Driven Microservices....275

6.6.3 Using End-to-End Partitioning to Handle Concurrency and Ordering....278

6.6.4 Limitations of End-to-End Partitioning in Event-Driven Microservices....281

Hotspotting....281

Momentary Hotspots....282

The Mix of Event-Driven and Synchronous APIs....283

6.7 Summary....284

Chapter 7: Achieving Resilience and Event Processing Reliability in Event-Driven Microservices....286

7.1 Common Failures in Microservice Architectures and How They Relate to Event-Driven Architectures....288

7.1.1 Cascading Failures and Event-Driven Services....290

7.1.2 Load Balancing and Rate Limiters in Event-Driven Services....295

7.2 Understanding Message Delivery Semantics....299

7.2.1 Exactly-Once Delivery Semantics in Kafka....304

7.3 Avoiding Inconsistencies When Saving State and Publishing Events in Event-Driven Microservices....304

7.3.1 Event Stream As the Only Source of Truth....306

7.3.2 Outbox Pattern in Event-Driven Microservices....309

7.3.3 Transactions and Compensating Actions to Avoid Inconsistencies in Event-Driven Microservices....311

7.4 Applying ACID 2.0 As a Resilience Strategy in Event-Driven Microservices....313

7.5 Avoiding Message Leak in Event-Driven Microservices....317

7.5.1 Poison Events....319

7.6 Applying Common Resilience Patterns in Event-Driven Microservices....319

7.6.1 Retries As a Resilience Approach in Event-Driven Microservices....320

7.6.2 Circuit Breakers in Event-Driven Microservices....322

7.7 Recovering Data and Repairing State in Event-Driven Microservices....325

7.8 Bulkhead Pattern in Event-Driven Microservices....327

7.8.1 Priority Queues....329

7.9 Conclusion....330

7.10 Summary....331

Chapter 8: Choosing the Correct Event Schema Design in Event-Driven Microservices....333

8.1 Event Storming and Event-Driven Microservices....334

8.1.1 What Are the Limitations of Event Storming?....339

8.2 Event Schema: Headers and Envelopes....339

8.2.1 Headers vs. Envelopes in Event Schema....340

8.2.2 Relevant Contextual Information in Events....342

8.3 Town Crier Events Pattern....344

8.4 Bee Events Pattern....347

8.5 The Event Schema Goldilocks Principle....350

8.6 Denormalized Event Schema....353

8.7 Schema Evolution in Event-Driven Microservices....355

8.7.1 Backward Compatibility....356

8.7.2 Forward Compatibility....357

8.7.3 Full Compatibility....357

8.7.4 No Compatibility....358

8.7.5 Managing Changes....358

8.7.6 Event Stream Versioning....361

8.7.7 Using a Downscaler/Upscaler When Evolving Event Schema....362

8.8 Summary....364

Chapter 9: How to Leverage the User Interface in Event-Driven Microservice Architectures....366

9.1 Using an Aggregating Layer to Build a UI in a Distributed Microservice Architecture....368

9.2 Backends for Frontends (BFFs)....371

9.3 UI Decomposition Pattern in Microservice Architectures....374

9.3.1 UI Application Decomposition Pattern....375

9.3.2 UI Page Decomposition Pattern....377

9.3.3 UI Section Decomposition Pattern....378

9.4 The Limitations of API Composition....380

9.5 Task-Based UIs....387

9.6 Event-Driven APIs....393

9.6.1 Event-Driven Combined with WebSockets....395

9.6.2 Event-Driven Combined with Server-Sent Events....396

9.6.3 Event-Driven Combined with WebHooks....398

9.7 Summary....400

Chapter 10: Overcoming the Challenges in Quality Assurance....402

10.1 Microservice Testing Approaches and How They Relate to Event-Driven....403

10.1.1 Unit Tests....404

10.1.2 Component Tests....406

10.1.3 Extended Component Tests....408

10.1.4 Integration Tests....411

10.1.5 Half-Breed Integration Tests....413

10.1.6 End-to-End Tests....417

Releases....419

Data and Environments....419

Governance....420

Approaches to Mitigate the End-to-End Tests Challenges....421

10.2 Applying Contract Tests and Consumer-Driven Contracts to Event-Driven....423

10.3 Test Categorization and Purpose....426

10.4 End-to-End Quality Without End-to-End Tests....428

10.5 Testing in Production....432

10.5.1 Shadowing....433

10.5.2 Canaries....438

10.5.3 Feature Flagging....440

10.5.4 Production Automated Testing....441

10.6 Summary....443

Index....445

In the simplest terms, event-driven architectures are like onions; they are manageable as a single layer (like a monolith) but when you get into them, they begin to cascade apart and you quickly realize that there are many complex layers (distributed microservices architecture). And that’s when the tears begin.

This prescriptive guide takes you through the steps of moving a platform with millions of users from a monolith to a microservices event-driven architecture. You will learn about the challenges and complexities that arise in high-throughput environments that often contain upwards of hundreds of microservices. This book is designed to be your single best resource for learning how to apply event-driven architectures in real-world scenarios and offers hundreds of patterns to overcome the common and not so common challenges.

While event-driven architectures have been the standard for decoupled, pluggable, evolutionary architectures for years, they have onlyrecently been adopted by enterprises for the purpose of distributed microservices and there is little information about adopting them. Using them at scale can save valuable resources, but requires different considerations, including the added complexity of supporting several moving parts and getting the event schema right from the start in order to avoid large restructuring later on.

Author Hugo Rocha understands that these kinds of challenges, as well as many others, need to be considered from the beginning, and helps teach you the mindset needed to create a deliberate strategy upfront. This book offers learning approaches and patterns to get you up to speed in order to sustainably build and manage event-driven architectures.

What You Will Learn

• Understand the real-world challenges of event-driven architectures and the patterns to deal with those challenges and the trade-offs of each solution
• Leverage the advantages of event-driven architectures to build scalable solutions and address legacy applications
• Plan successful future implementations to avoid common pitfalls and apply proven patterns to deal with challenges in a real-world platform with millions of users
• Decide whether event-driven solutions are the right choice for the requirements at hand
• Discuss and understand advanced concepts about event-driven architectures

Who Is This Book For

Software engineers and software architects. Anyone currently working with microservice architectures, primarily event-driven microservices, will greatly benefit from this book. Readers working with monoliths will benefit, as the book explores migration from a monolithic application to an event-driven microservice architecture.