Ultimate Neural Network Programming with Python: Create Powerful Modern AI Systems by Harnessing Neural Networks with Python, Keras, and TensorFlow

Cover Page....2

Title Page....3

Copyright Page....4

Dedication Page....5

About the Author....6

About the Technical Reviewers....7

Welcome note....9

Acknowledgements....13

Preface....14

Errata....17

Table of Contents....20

1. Understanding AI History....28

Structure....28

Evolution of AI....28

The early history of AI....28

The most crucial development in the History of AI....31

AI started evolving into new fields....32

AI starts taking its modern form....32

Understanding Intelligent Behavior....33

AI beats humans at chess....33

AI learning reasoning and language....34

AI starts playing poker....35

Conquering GO and Dota 2....36

An experience with ChatGPT....37

Difference between Artificial Intelligence, Machine Learning, and Deep Learning....39

Formally defining AI terms....40

Learning representations from data....42

Sub-Fields of AI....44

Artificial Intelligence (AI)....44

Machine Learning (ML)....45

Deep Learning (DL)....45

Early Models of Neuron-Inspired Networks....46

Understanding biological neurons....46

McCulloch-Pitts model of a neuron....47

Multilayer Perceptron (MLP)....48

Conclusion....52

2. Setting up Python Workflow for AI Development....54

Structure....54

Setting up Python Environment....56

Installing Python....56

Getting Anaconda for Data Science Environment Setup....57

Setting up a Virtual Environment....57

Installing packages....58

Setting up VS Code....58

Installing Git....60

Setting up GitHub with VS Code....61

Concepts of OOPS....65

Encapsulation....67

Accessing Variables....69

Inheritance....71

Conclusion....76

3. Python Libraries for Data Scientists....77

Structure....77

Web Scraping....77

Regex....83

Multi-Threading and Multi-Processing....91

Multi-Threading....91

Multi-Processing....93

Pandas Basics....95

Conclusion....109

4. Foundational Concepts for Effective Neural Network Training....111

Structure....111

Activation Functions....111

RBF, Universal Approximators, and Curse of Dimensionality....115

Radial Bias Function....115

Neural Networks are universal approximators....117

The curse of dimensionality....118

Overfitting, Bias-Variance, and Generalization....120

Overfitting problem....120

Regularization and effective parameters....122

Dropout....124

Early stopping and validation set....124

Bias-Variance trade-off....125

Generalization....126

Conclusion....127

5. Dimensionality Reduction, Unsupervised Learning and Optimizations....129

Structure....129

Dimensionality reduction....129

Principal component analysis (PCA)....129

T-SNE....132

Non-linear PCA....137

Unsupervised learning....137

Clustering....138

Semi-supervised learning....144

Generalizing active learning to multi-class....147

Self-supervised learning....150

Version space....153

Understanding optimization through SVM....158

Conclusion....165

6. Building Deep Neural Networks from Scratch....167

Structure....167

Coding neurons....167

A single neuron....167

Layer of neurons....168

Understanding lists, arrays, tensors, and their operations....171

Dot product and vector addition....172

Cross-product, transpose, and order....173

Understanding neural networks through NumPy....174

Neural networks using NumPy....174

Processing batch of data....175

Creating a multi-layer network....176

Dense layers....177

Activation functions....181

Calculating loss through categorical cross-entropy loss....187

Calculating accuracy s....194

Conclusion....197

7. Derivatives, Backpropagation, and Optimizers....199

Structure....199

Weights Optimization....199

Derivatives....204

Partial Derivatives....208

Backpropagation....209

Optimizers: SGD, Adam, and so on....220

Gradient-based optimization....221

Momentum-based optimization....222

RMSProp....222

Adam....223

Conclusion....242

8. Understanding Convolution and CNN Architectures....243

Structure....243

Intricacies of CNN....243

Local Patterns and Global Patterns....244

Spatial Hierarchies and Abstraction....244

Convolution Operation and Feature Maps....246

Pooling....248

Padding....248

Stride....249

Introduction to CNN-based Networks....253

Understanding the Complete Flow of CNN-based Network....254

VGG16....257

Inception Module: Naïve and Improved Version....258

ResNet....261

Other Variants of ResNet....263

FractalNet and DenseNet....264

Scaling Conv Networks: Efficient Net Architecture....266

Different Types of Convolutions....268

Depth-Separable Convolution....268

Conclusion....270

9. Understanding Basics of TensorFlow and Keras....272

Structure....272

A Brief Look at Keras....272

Understanding TensorFlow Internals....278

Tensors....278

Computational Graphs....281

Operations (Ops)....282

Automatic Differentiation....282

Sessions....283

Variables....283

Eager Execution....284

Layers and Models (Keras)....285

TensorFlow vs. PyTorch vs. Theano....285

TensorFlow vs. PyTorch....285

TensorFlow vs. Theano....286

TensorFlow: Layers, Activations, and More....287

Types of Layers....287

Dense Layer (Fully Connected Layer)....287

Convolution Layer....289

Max Pooling Layer....289

Dropout Layer....289

Recurrent Layer (LSTM)....290

Embedding Layer....290

Flatten Layer....290

Batch Normalization Layer....290

Global Average Pooling Layer....291

Upsampling/Transposed Convolution Layer....291

Activation Functions....291

Optimizers....294

Weight Initialization....294

Loss Functions....296

Multi-Input Single-Output Network with Custom Callbacks....297

Conclusion....302

10. Building End-to-end Image Segmentation Pipeline....303

Structure....303

Fine-tuning and Interpretability....303

Power of Fine-Tuning in Deep Learning....303

SHAP - An Intuitive Way to Interpret Machine Learning Models....304

Structuring Deep Learning Code....307

Project Structure....307

Python modules and packages....308

Documentation....309

Unit testing....310

Debugging....311

Logging....313

Building End-to-end Segmentation Pipeline....315

UNet and Attention Gates....316

Config....319

Dataloader....319

Model building....323

Understanding Attention block....324

Executor....332

Utils....335

Evaluation....338

main....341

Conclusion....345

11. Latest Advancements in AI....346

Structure....346

Transformers: Improving NLP Using Attention....346

Recurrent Neural Network (RNN)....346

Long-Short Term Memory (LSTM)....347

Self-Attention....348

Example to understand the concept:....348

Understanding Key, Query, and Value....350

Example to understand the concept:....350

Transformer Architecture....354

ChatGPT/GPT Overview....356

Object Detection: Understanding YOLO....357

Object Detector Architecture Breakdown....360

Backbone, Neck, and Head....361

Bag of Freebies (BoF)....363

CmBN: Cross-mini-Batch Normalization....365

Bag of Specials (BoS)....366

Cross-Stage Partial (CSP) Connection....367

YOLO A rchitecture S election....368

Spatial Pyramid Pooling (SPP)....369

PAN Path — Aggregation Block....370

Spatial Attention Module (SAM)....371

Image Generation: GAN’s and Diffusion models....372

Generative Adversarial Networks....372

Generative Discriminative models....373

Variational Autoencoders....375

GANs....376

Diffusion Models....380

DALL-E 2 Architecture....381

The Encoder: Prior Diffusion Model....383

The Decoder: GLIDE....385

Conclusion....386

Index....387

This book is a practical guide to the world of Artificial Intelligence (AI), unraveling the math and principles behind applications like Google Maps and Amazon.The book starts with an introduction to Python and AI, demystifies complex AI math, teaches you to implement AI concepts, and explores high-level AI libraries.Throughout the chapters, readers are engaged with the book through practice exercises and supplementary learnings. The book then gradually moves to Neural Networks with Python before diving into constructing ANN models and real-world AI applications. It accommodates various learning styles, letting readers focus on hands-on implementation or mathematical understanding.This book isn't just about using AI tools; it's a compass in the world of AI resources, empowering readers to modify and create tools for complex AI systems. It ensures a journey of exploration, experimentation, and proficiency in AI, equipping readers with the skills needed to excel in the AI industry.