CPython: A Complete Guide to CPython’s Architecture and Performance

Table of Contents....4

About the Author....12

Chapter 1: Reading the CPython Source Code....13

What Is CPython?....13

Why Read the Source Code?....13

Where to Begin?....14

Obtaining the Source Code....14

Development Tools....15

Can I Understand This Without Knowing C?....16

Chapter 2: Overview of the CPython Project Structure....17

Project Structure....17

Building the Project....19

Greet CPython!....20

A (Very) Basic Module....23

Saying Goodbye on Exit....25

Chapter 3: Everything Is an Object: Part 1....27

What Is an “Object”?....27

The Previous and Next Object....29

Garbage Collection Mechanism....30

Immortal Objects....33

PyTypeObject....34

Summary....36

Chapter 4: How Objects Are Created in CPython....37

Running the Program!....37

Step 0: Code Analysis....37

Step 1: Transformation to AST....38

Step 2: Compilation to Bytecode....39

Step 3: Instantiating an Object....41

Summary....43

Chapter 5: Everything Is an Object: Part 2....45

PyTypeObject....45

Basic Members....46

Methods and Operators....46

Access Methods....47

Other....47

The List Type....48

Printing a List....49

Using the Bracket Operator....51

List Methods....53

List Addition....54

Number of Elements....56

List Comparison....57

Summary....59

Chapter 6: Defining a Custom Built-in Type....60

Creating a New Type....60

Defining Methods....61

Implementing the Type....62

Making It a Built-in Type....64

Building and Running....65

Parameterized Initialization....66

Chapter 7: What Happens During Module Import....72

Different Ways of Importing....72

The import Instruction....73

The from .. import .. Instruction....77

Crazy Side-Effects!....78

The Behind-the-Scenes Hero: meta_path....80

Summary....82

Chapter 8: The Internal Representation of Integers....83

How Are Numbers Created?....83

Integer Objects....85

Astronomical Numbers!....87

Small Integers....88

Chapter 9: Floating Point Numbers in CPython....91

What Is a “Floating Point Number”?....91

The Structure of Floating Point Numbers....92

About Floating Point Numbers....93

Floating Point Arithmetic....94

Infinity!....96

Not a Number!....97

Comparing Floating Point Numbers....98

Floating Point Performance....99

Chapter 10: Inside the String Object: Part 1....101

Creating a String....101

String Objects....102

The Fundamental String Structure....105

String Operations....106

Encoding Conversion....106

Strings Are Immutable....108

Chapter 11: Inside the String Object: Part 2....110

String Operations....110

Copying Strings....110

String Slicing....113

Performance....116

String Interning....116

Chapter 12: What Happens When Python Starts....119

Using a Debugger....119

Program Entry Point....120

Reading the Program File....123

Building the Abstract Syntax Tree....125

Creating the Code Object....126

Ready for Liftoff!....127

Chapter 13: From Source to Bytecode: How .py Becomes .pyc....129

Having a .pyc Is All You Need....129

“Maybe” a .pyc File?....131

Magic Number, Magic!....133

Unpacking a .pyc File....137

Chapter 14: The List Object and Its Internal Management....140

Internal Structure of Lists....140

Creating and Initializing a List....141

Memory Management....144

When Is More Memory Needed?....145

1. Adding Elements, But Within Current Capacity....146

2. Adding Elements, Now Exceeding Capacity....146

3. Removing Some Elements, But Still Above Half the Capacity....146

4. Sharply Decreasing Size....146

5. Clearing the List....146

The Over-allocation Formula....147

Common List Operations....148

Appending Elements....148

Inserting Elements....149

Removing Elements....150

Chapter 15: The Dictionary Object: Part 1....152

The Internal Structure of Dictionaries....152

Creating a Dictionary....154

Adding Elements....156

Handling Hash Collisions....161

Looking Up Elements....164

Chapter 16: The Dictionary Object: Part 2....166

Dictionary Memory Management Techniques....166

Adding More Elements....166

Should We Request More Capacity?....169

How Much Space to Allocate?....173

Returning Memory: Does It Happen?....174

Chapter 17: The Tuple Object and Its Immutability....176

Tuple Design....176

Creating a Tuple....177

Empty Tuples....179

Non-empty Tuples....180

Deallocation Mechanism....181

Common Tuple Operations....184

Modifying Tuples....184

Tuple Unpacking....185

Chapter 18: Inside the Python VM: Code Objects....188

Functions Are Also Objects....188

Preparing to Create a Function....191

Code Object....192

Chapter 19: Inside the Python VM: Function Objects....198

Creating Function Objects....198

What Do the Parameters Look Like?....200

Accessing Function Attributes....201

Calling a Function....202

What Is “Vectorcall”?....203

Chapter 20: Inside the Python VM: Frame Objects....206

Frame Object....206

The Life Cycle of a Frame Object....208

Chapter 21: Inside the Python VM: Namespaces and Scopes....212

Variable Scope....212

Local Variables (L)....213

Global and Built-in Variables (G, B)....214

Enclosing Variables (E)....218

Chapter 22: Inside the Python VM: Cells and Closures....221

Creating a Cell Object....221

Closures....223

Free Variables....225

From the Python Perspective....226

Chapter 23: Classes and Where They Come From....229

Creating a Class....229

The Mastermind Behind the Scenes....232

Selecting the Metaclass....233

Preparing the Namespace....234

The Birth of a Class!....235

Chicken or the Egg?....237

Chapter 24: Class Inheritance in CPython....239

Classes and Inheritance....239

Creating a Class....239

How Does Inheritance Work?....241

Method Lookup....242

Chapter 25: Method Resolution Order and C3 Linearization....248

The C3 Linearization Algorithm....248

Whose Method Gets Called?....249

MRO Calculation....250

Single Inheritance....250

Multiple Inheritance....254

A More Complex Inheritance Example....255

What If It Can’t Be Calculated?....258

Chapter 26: The Role of super() in Multiple Inheritance....261

Algorithm Implementation....261

Preparation Before Merging....261

Merging....264

Family Feuds....266

Super!....267

Whose Child Is It?....269

Solving Family Feuds....273

Specifying the Superclass....274

Quick Quiz: Who Am I?....275

Chapter 27: The Generator Object and the Yield Statement....276

The Generator Class....277

yield, Please!....281

Next, Please!....282

Chapter 28: How Iterators Work Internally....286

The Iterator Protocol....286

Halt! Password, Please!....288

Different Types of Iterators?....294

Chapter 29: Understanding Descriptors in Python....297

When Calling Methods....297

Attribute Lookup Process....298

Process Summary....302

Method Descriptors....303

Chapter 30: Exception Handling Internals in CPython....306

Exception Handling....306

Stacking Up....307

Exception Table....308

Entry Portals....309

Exception Type Matching....309

Handling the Exception....311

Finally!....312

Index....314

About this book

Deep Dive CPython explores the internal mechanics of CPython, the widely used Python interpreter written in C. Starting with a practical guide on downloading and compiling the CPython source, this book is perfect for developers eager to understand Python’s behaviour at a fundamental level.

The book takes readers from basic concepts to complex details with a systematic breakdown of core components. It covers everything from CPython’s data structures like PyObject and PyTypeObject to object lifecycle management, giving insight into memory allocation and object reference counting. Each chapter illustrates CPython's architecture, such as Python's "everything is an object" philosophy, list handling, string manipulation, and dictionary operations. Readers will explore Python’s REPL modifications, string internals, and custom type creation with practical examples, like crafting a "backflipping" PyKitty_Type. Detailed sections on Python’s virtual machine operations, bytecode generation, and exception handling enrich readers’ understanding of how Python code is parsed, compiled, and executed.

This book is a thorough guide for readers who want to go beyond basic Python use and understand how it works internally. Covering complex concepts like generators, iterators, descriptors, and metaclasses, this book equips readers with a thorough grasp of Python's performance optimization and design complexities.

What you will learn:

• How to download, compile, and modify CPython's source code
• Gain insight into fundamental structures like PyObject and PyTypeObject,
• Understand Python's detailed handling of lists, strings, dictionaries, and the REPL environment.
• What are bytecode generation, custom types, and the inner workings of Python’s virtual machine.

Who this book is for:

Python programmers aiming to gain a deeper understanding of Python’s internals and move beyond standard usage, as well as software professionals interested in CPython’s C-based implementation and core architecture.