Building Applications with AI Agents: Designing and Implementing Multiagent Systems

Copyright....6

Table of Contents....7

Preface....15

What This Book Is About....16

What This Book Is Not....17

Who This Book Is For....17

Navigating This Book....17

Conventions Used in This Book....19

Using Code Examples....19

O’Reilly Online Learning....20

How to Contact Us....20

Acknowledgments....21

Chapter 1. Introduction to Agents....23

Defining AI Agents....23

The Pretraining Revolution....24

Types of Agents....25

Model Selection....27

From Synchronous to Asynchronous Operations....28

Practical Applications and Use Cases....29

Workflows and Agents....30

Principles for Building Effective Agentic Systems....33

Organizing for Success in Building Agentic Systems....34

Agentic Frameworks....35

LangGraph....35

AutoGen....36

CrewAI....36

OpenAI Agents Software Development Kit (SDK)....36

Conclusion....37

Chapter 2. Designing Agent Systems....39

Our First Agent System....39

Core Components of Agent Systems....42

Model Selection....43

Tools....46

Designing Capabilities for Specific Tasks....46

Tool Integration and Modularity....47

Memory....47

Short-Term Memory....48

Long-Term Memory....48

Memory Management and Retrieval....48

Orchestration....49

Design Trade-Offs....49

Performance: Speed/Accuracy Trade-Offs....49

Scalability: Engineering Scalability for Agent Systems....50

Reliability: Ensuring Robust and Consistent Agent Behavior....51

Costs: Balancing Performance and Expense....52

Architecture Design Patterns....54

Single-Agent Architectures....54

Multiagent Architectures: Collaboration, Parallelism, and Coordination....54

Best Practices....56

Iterative Design....56

Evaluation Strategy....57

Real-World Testing....59

Conclusion....61

Chapter 3. User Experience Design for Agentic Systems....63

Interaction Modalities....64

Text-Based....65

Graphical Interfaces....68

Speech and Voice Interfaces....72

Video-Based Interfaces....75

Combining Modalities for Seamless Experiences....76

The Autonomy Slider....77

Synchronous Versus Asynchronous Agent Experiences....80

Design Principles for Synchronous Experiences....80

Design Principles for Asynchronous Experiences....81

Finding the Balance Between Proactive and Intrusive Agent Behavior....81

Context Retention and Continuity....82

Maintaining State Across Interactions....83

Personalization and Adaptability....84

Communicating Agent Capabilities....85

Communicating Confidence and Uncertainty....86

Asking for Guidance and Input from Users....87

Failing Gracefully....87

Trust in Interaction Design....88

Conclusion....90

Chapter 4. Tool Use....93

LangChain Fundamentals....94

Local Tools....95

API-Based Tools....97

Plug-In Tools....100

Model Context Protocol....103

Stateful Tools....106

Automated Tool Development....107

Foundation Models as Tool Makers....107

Real-Time Code Generation....108

Tool Use Configuration....109

Conclusion....110

Chapter 5. Orchestration....111

Agent Types....112

Reflex Agents....112

ReAct Agents....112

Planner-Executor Agents....113

Query-Decomposition Agents....113

Reflection Agents....113

Deep Research Agents....114

Tool Selection....115

Standard Tool Selection....116

Semantic Tool Selection....119

Hierarchical Tool Selection....123

Tool Execution....127

Tool Topologies....127

Single Tool Execution....128

Parallel Tool Execution....129

Chains....129

Graphs....131

Context Engineering....134

Conclusion....135

Chapter 6. Knowledge and Memory....137

Foundational Approaches to Memory....138

Managing Context Windows....138

Traditional Full-Text Search....139

Semantic Memory and Vector Stores....141

Introduction to Semantic Search....141

Implementing Semantic Memory with Vector Stores....141

Retrieval-Augmented Generation....143

Semantic Experience Memory....144

GraphRAG....145

Using Knowledge Graphs....145

Building Knowledge Graphs....146

Promise and Peril of Dynamic Knowledge Graphs....152

Note-Taking....155

Conclusion....156

Chapter 7. Learning in Agentic Systems....157

Nonparametric Learning....157

Nonparametric Exemplar Learning....157

Reflexion....159

Experiential Learning....163

Parametric Learning: Fine-Tuning....168

Fine-Tuning Large Foundation Models....168

The Promise of Small Models....173

Supervised Fine-Tuning....175

Direct Preference Optimization....180

Reinforcement Learning with Verifiable Rewards....183

Conclusion....184

Chapter 8. From One Agent to Many....185

How Many Agents Do I Need?....185

Single-Agent Scenarios....185

Multiagent Scenarios....192

Swarms....199

Principles for Adding Agents....200

Multiagent Coordination....202

Democratic Coordination....202

Manager Coordination....203

Hierarchical Coordination....204

Actor-Critic Approaches....204

Automated Design of Agent Systems....206

Communication Techniques....211

Local Versus Distributed Communication....211

Agent-to-Agent Protocol....211

Message Brokers and Event Buses....214

Actor Frameworks: Ray, Orleans, and Akka....217

Orchestration and Workflow Engines....221

Managing State and Persistence....223

Conclusion....224

Chapter 9. Validation and Measurement....227

Measuring Agentic Systems....227

Measurement Is the Keystone....228

Integrating Evaluation into the Development Lifecycle....229

Creating and Scaling Evaluation Sets....229

Component Evaluation....231

Evaluating Tools....231

Evaluating Planning....232

Evaluating Memory....234

Evaluating Learning....235

Holistic Evaluation....236

Performance in End-to-End Scenarios....236

Consistency....238

Coherence....239

Hallucination....240

Handling Unexpected Inputs....241

Preparing for Deployment....242

Conclusion....243

Chapter 10. Monitoring in Production....245

Monitoring Is How You Learn....246

Monitoring Stacks....248

Grafana with OpenTelemetry, Loki, and Tempo....249

ELK Stack (Elasticsearch, Logstash/Fluentd, Kibana)....249

Arize Phoenix....250

SigNoz....251

Langfuse....251

Choosing the Right Stack....252

OTel Instrumentation....252

Visualization and Alerting....254

Monitoring Patterns....257

Shadow Mode....257

Canary Deployments....257

Regression Trace Collection....258

Self-Healing Agents....258

User Feedback as an Observability Signal....258

Distribution Shifts....259

Metric Ownership and Cross-Functional Governance....261

Conclusion....263

Chapter 11. Improvement Loops....265

Feedback Pipelines....267

Automated Issue Detection and Root Cause Analysis....272

Human-in-the-Loop Review....273

Prompt and Tool Refinement....276

Aggregating and Prioritizing Improvements....281

Experimentation....282

Shadow Deployments....283

A/B Testing....284

Bayesian Bandits....285

Continuous Learning....287

In-Context Learning....287

Offline Retraining....289

Conclusion....290

Chapter 12. Protecting Agentic Systems....293

The Unique Risks of Agentic Systems....294

Emerging Threat Vectors....295

Securing Foundation Models....297

Defensive Techniques....298

Red Teaming....300

Threat Modeling with MAESTRO....303

Protecting Data in Agentic Systems....305

Data Privacy and Encryption....305

Data Provenance and Integrity....307

Handling Sensitive Data....308

Securing Agents....310

Safeguards....310

Protections from External Threats....312

Protections from Internal Failures....314

Conclusion....318

Chapter 13. Human-Agent Collaboration....319

Roles and Autonomy....319

The Changing Role of Humans in Agent Systems....320

Aligning Stakeholders and Driving Adoption....321

Scaling Collaboration....322

Agent Scope and Organizational Roles....324

Shared Memory and Context Boundaries....325

Trust, Governance, and Compliance....327

The Lifecycle of Trust....327

Accountability Frameworks....328

Escalation Design and Oversight....331

Privacy and Regulatory Compliance....332

Conclusion: The Future of Human-Agent Teams....334

Glossary....337

Index....341

About the Author....354

Colophon....354

Generative AI has revolutionized how organizations tackle problems, accelerating the journey from concept to prototype to solution. As the models become increasingly capable, we have witnessed a new design pattern emerge: AI agents. By combining tools, knowledge, memory, and learning with advanced foundation models, we can now sequence multiple model inferences together to solve ambiguous and difficult problems. From coding agents to research agents to analyst agents and more, we've already seen agents accelerate teams and organizations. While these agents enhance efficiency, they often require extensive planning, drafting, and revising to complete complex tasks, and deploying them remains a challenge for many organizations, especially as technology and research rapidly develops.

This book is your indispensable guide through this intricate and fast-moving landscape. Author Michael Albada provides a practical and research-based approach to designing and implementing single- and multiagent systems. It simplifies the complexities and equips you with the tools to move from concept to solution efficiently.

• Understand the distinct features of foundation model-enabled AI agents
• Discover the core components and design principles of AI agents
• Explore design trade-offs and implement effective multiagent systems
• Design and deploy tailored AI solutions, enhancing efficiency and innovation in your field