Modern Data Architecture on Azure: Design Data-centric Solutions on Microsoft Azure

Table of Contents....5

About the Author....10

About the Technical Reviewer....11

Acknowledgments....12

Introduction....13

Chapter 1: Introduction: Fundamentals of Data Management....14

Introduction to DAMA and DMBOK....15

Essential Data Concepts....16

Types of Data....16

Qualitative Data....17

Nominal Data....17

Ordinal Data....18

Quantitative Data....18

Discrete Data....19

Continuous Data....19

Data Management Principles....19

The Data Lifecycle....20

Consistency Models....21

Data Ingestion Patterns....21

Data Platform Paradigm....22

Data Management Principles and Challenges....25

Preparing a Data Strategy....25

Defining Roles and Responsibilities....26

Data Lifecycle Management....27

Data Quality Measurements....28

Metadata....28

Maximizing Data Value for Data-Driven Decisions....30

Dealing with Substantial Volumes of Data....30

Siloed and Varied Data Sources....31

Maintaining the Quality of the Data....31

Data Integration....31

Data Governance and Security....32

Data Automation....32

Data Management Frameworks....33

The Strategic Alignment Model....33

The Amsterdam Information Model....35

The DAMA DMBOK Framework....36

The DAMA Wheel....42

Data Governance....43

Data Architecture....44

Data Modeling and Design....45

Data Storage and Operations....46

Data Security....46

Data Integration and Interoperability....47

Document and Content Management....47

Reference and Master Data....47

Data Warehousing and Business Intelligence....48

Metadata....48

Data Quality....49

Understanding the Environmental Factors Hexagon....50

Understanding the Knowledge Area Context Diagram....51

Conclusion....52

Chapter 2: Build Relational and Non-Relational Data Solutions on Azure....53

Data Integration Using ETL....54

Data Extraction....55

Data Transformation....56

Data Loading....56

Designing ELT Pipelines Using the Azure Synapse Server....57

Online Analytical Processing for Complex Analyses....59

Semantic Data Modeling....63

Challenges of Using OLAP Solutions....65

Managing Transaction Data Using OLTP....66

Managing Non-Relational Data....72

Key-Value Pair Databases....73

Column Family Databases....74

Document Databases....74

Graph Databases....75

Handling Time-Series and Free-Form Search Data....77

Working with CSV and JSON Files for Data Solutions....84

Conclusion....87

Chapter 3: Building a Big Data Architecture....88

Core Components of a Big Data Architecture....89

Data Ingestion and Processing....90

Data Analysis....92

Data Visualization....93

Data Governance....94

Using Batch Processing....94

Azure Synapse Analytics....97

Azure Data Lake Analytics....97

Azure Databricks....98

Azure Data Explorer....99

Real-Time Processing....100

Real-Time Data Ingestion....103

The Lambda Architecture....106

The Kappa Architecture....110

Internet of Things (IoT)....116

Data Mesh Principles and the Logical Architecture....118

Conclusion....123

Chapter 4: Data Management Patterns and Technology Choices with Azure....124

Data Patterns and Trends in Depth....125

CQRS Pattern....125

Event Sourcing....128

Materialized Views....128

Index Table Pattern....129

Analytical Store for Big Data Analytics....131

Azure Synapse Analytics....131

Azure Databricks....133

Data Ingestion Process....134

Data Storage....135

Data Transformation and Model Training....135

Analytics....135

Azure Data Explorer....136

Building Enterprise Data Lakes and Data Lakehouses....137

Enterprise Data Lakes....138

Enterprise Data Lakehouses....142

Data Pipeline Orchestration....144

Real-Time Stream Processing in Azure....149

Conclusion....152

Chapter 5: Data Architecture Process....153

Guide to Data Modeling....153

Conceptual Data Model....155

Logical Data Model....156

Physical Data Model....157

Focus on Business Objectives and its Requirements....157

Data Lake for Ad Hoc Queries....160

Enterprise Data Governance: Data Scrambling, Obfuscation, and DataOps....165

Data Masking Techniques....168

Data Scrambling....170

Data Encryption....171

Data Ageing....171

Data Substitution....171

Data Shuffling....171

Pseudonymization....172

Master Data Management and Storage Optimization....173

Master Data Management....174

Data Encryption Patterns....180

Conclusion....184

Chapter 6: Data Architecture Framework Explained....185

Fundamentals of Data Modeling....185

The Network Data Model....187

The Hierarchical Data Model....188

The Relational Data Model....189

The Object-Oriented Data Model....190

The Dimensional Data Model....191

The Graph Data Model....192

The Entity Relationship Data Model....193

The Open Group Architecture Framework....194

Preliminary Phase....197

Defining the Architecture Vision....197

Business Architecture....198

Information System Architecture....198

Technology Architecture....198

Opportunities and Solutions....199

Migration Planning....199

Governance Implementation....200

Architecture Change Management....200

DAMA DMBOK....200

The Zachman Framework....205

Conclusion....208

Index....209

df-Capture.PNG....1

This book is an exhaustive guide to designing and implementing data solutions on Azure. It covers the process of managing data from end to end, starting from data collection all the way through transformation, distribution, and consumption.

Modern Data Architecture on Azure begins with an introduction to the fundaments of data management, followed by a demonstration of how to build relational and non-relational data solutions on Azure. Here, you will learn data processing for complex analysis and how to work with CSV and JSON files. Moving forward, you will learn the foundational concepts of big data architecture, along with data management patterns and technology options offered by Azure. From there, you’ll be walked through the data architecture process, including data consortium on Azure, enterprise data governance, and much more. The book culminates with a deep dive into data architecture frameworks with data modeling.

After reading this book, you will have a thorough understanding of data design and analytics using Azure, allowing you to collect and analyze massive amounts of data to optimize business performance, forecast future results, and more.

What Will You Learn

• Understand the fundamentals of data architecture including data management, data handling ethics, data governance, and metadata management
• Analyze and understand business needs to choose the right Azure services and make informed business decisions
• Understand Azure Cloud Data design patterns for relational and non-relational data, batch real-time processing, and ETL/ELT pipelines
• Modernize data architecture using Azure to leverage data and AI to enable digital transformation by securing and optimizing overall data lifecycle management

Who Is This Book For:

Data solution architects, data engineers, and IT consultants who want to gain a better understanding of modern data architecture design and implementation on Azure.