Python in a Nutshell: A Desktop Quick Reference. 4 Ed

Preface....5

How To Use This Book....5

Part I, Getting Started with Python....6

Part II, Core Python Language and Built-ins....7

Part III, Python Library and Extension Modules....9

Part IV, Network and Web Programming....11

Part V, Extending, Distributing, and Version Upgrade and Migration....12

Conventions Used in This Book....12

Reference Conventions....12

Version Conventions....13

Typographic Conventions....13

Using Code Examples....14

OReilly Online Learning....15

How to Contact Us....15

Acknowledgments....16

1. Introduction to Python....18

The Python Language....18

The Python Standard Library and Extension Modules....19

Python Implementations....20

CPython....20

PyPy....21

Choosing Between CPython, PyPy, and Other Implementations....21

Other Developments, Implementations, and Distributions....22

Jython and IronPython....22

Numba....23

Pyjion....23

IPython....23

MicroPython....24

Anaconda and Miniconda....25

pyenv: Simple support for multiple versions....26

Transcrypt: Convert your Python to JavaScript....26

Licensing and Price Issues....27

Python Development and Versions....28

Python Resources....29

Documentation....29

Python documentation for nonprogrammers....29

Extension modules and Python sources....30

Books....30

Community....31

Python Software Foundation....31

Workgroups....31

Python conferences....32

User groups and organizations....32

Mailing lists....32

Social media....32

Installation....33

Installing Python from Binaries....33

Installing Python from Source Code....35

Microsoft Windows....35

Uncompressing and unpacking the Python source code....35

Building the Python source code....36

Unix-Like Platforms....36

Uncompressing and unpacking the Python source code....36

Configuring, building, and testing....36

Installing after the build....37

2. The Python Interpreter....39

The python Program....39

Environment Variables....39

Command-Line Syntax and Options....40

The Windows py Launcher....42

The PyPy Interpreter....43

Interactive Sessions....44

Python Development Environments....45

IDLE....46

Other Python IDEs....46

Free Text Editors with Python Support....47

Tools for Checking Python Programs....47

Running Python Programs....48

Running Python in the Browser....49

PyScript....50

Jupyter....50

3. The Python Language....52

Lexical Structure....52

Lines and Indentation....52

Character Sets....53

Tokens....54

Identifiers....54

Keywords....57

Operators....57

Delimiters....58

Literals....58

Statements....59

Simple statements....59

Compound statements....60

Data Types....60

Numbers....61

Integer numbers....61

Floating-point numbers....62

Complex numbers....63

Underscores in numeric literals....64

Sequences....64

Iterables....64

Strings....64

bytes objects....68

bytearray objects....68

Tuples....68

Lists....69

Sets....70

Dictionaries....71

None....73

Ellipsis (...)....73

Callables....73

Boolean Values....74

Variables and Other References....75

Variables....75

Object attributes and items....75

Accessing nonexistent references....76

Assignment Statements....76

Plain assignment....77

Augmented assignment....79

del Statements....80

Expressions and Operators....81

Comparison Chaining....82

Short-Circuiting Operators....82

The conditional operator....83

Assignment Expressions....83

: in an ifelif statement....83

: in a while statement....84

: in a list comprehension filter....84

Numeric Operations....85

Numeric Conversions....85

Arithmetic Operations....86

Division....86

Exponentiation....87

Comparisons....87

Bitwise Operations on Integers....87

Sequence Operations....87

Sequences in General....88

Sequence conversions....88

Concatenation and repetition....88

Membership testing....88

Indexing a sequence....89

Slicing a sequence....89

Strings....91

Tuples....91

Lists....91

Modifying a list....91

In-place operations on a list....92

List methods....92

Sorting a list....93

Set Operations....94

Set Membership....95

Set Methods....95

Dictionary Operations....97

Dictionary Membership....97

Indexing a Dictionary....97

Dictionary Methods....98

Control Flow Statements....100

The if Statement....100

The match Statement....102

Building patterns....103

Literal patterns....104

The wildcard pattern....104

Capture patterns....105

Value patterns....105

OR patterns....106

Group patterns....107

Sequence patterns....107

as patterns....108

Mapping patterns....109

Class patterns....110

Guards....111

Configuring classes for positional matching....112

The while Statement....113

The for Statement....113

Iterators....116

Iterables versus iterators....117

range....118

List comprehensions....118

Set comprehensions....121

Dictionary comprehensions....121

The break Statement....121

The continue Statement....122

The else Clause on Loop Statements....123

The pass Statement....123

The try and raise Statements....124

The with Statement....124

Functions....124

Defining Functions: The def Statement....125

Parameters....127

Positional parameters....127

Named parameters....127

Positional-only marker....128

Positional argument collector....128

Named argument collector....129

Parameter sequence....129

Mutable default parameter values....129

Argument collector parameters....130

Attributes of Function Objects....131

Docstrings....131

Other attributes of function objects....133

Function Annotations....133

The return Statement....134

Calling Functions....134

Positional and named arguments....135

Keyword-only parameters....137

Matching arguments to parameters....138

The semantics of argument passing....140

Namespaces....140

The global statement....141

Nested functions and nested scopes....142

lambda Expressions....144

Generators....145

yield from....147

Generators as near-coroutines....147

Generator expressions....147

Recursion....149

4. Object-Oriented Python....153

Classes and Instances....153

Python Classes....153

The class Statement....154

The Class Body....155

Attributes of class objects....155

Function definitions in a class body....157

Class-private variables....157

Class documentation strings....158

Descriptors....158

Overriding and nonoverriding descriptors....159

Instances....159

__init__....159

Attributes of instance objects....160

The factory function idiom....162

__new__....162

Attribute Reference Basics....163

Getting an attribute from a class....164

Getting an attribute from an instance....165

Setting an attribute....166

Bound and Unbound Methods....166

Inheritance....170

Method resolution order....170

Overriding attributes....171

Delegating to superclass methods....171

Cooperative superclass method calling....173

Dynamic class definition using the type built-in function....174

Deleting class attributes....175

The Built-in object Type....176

Class-Level Methods....176

Static methods....177

Class methods....177

Properties....178

Why properties are important....180

Properties and inheritance....181

__slots__....182

__getattribute__....183

Per Instance Methods....183

Inheritance from Built-in Types....184

Special Methods....184

General-Purpose Special Methods....185

Special Methods for Containers....189

Sequences....189

Mappings....190

Sets....190

Container slicing....191

Container methods....191

Abstract Base Classes....192

The abc module....193

ABCs in the collections module....193

ABCs in the numbers module....195

Special Methods for Numeric Objects....196

Decorators....198

Metaclasses....199

Alternatives to Custom Metaclasses for Simple Class Customization....199

How Python Determines a Classs Metaclass....201

How a Metaclass Creates a Class....201

Defining and using your own metaclasses....202

A substantial custom metaclass example....203

Data Classes....205

Enumerated Types (Enums)....208

5. Type Annotations....213

History....213

Type-Checking Utilities....214

mypy....215

Other Type Checkers....215

Type Annotation Syntax....216

The typing Module....219

Types....219

Type Expression Parameters....221

Abstract Base Classes....221

Protocols....222

Utilities and Decorators....224

Defining Custom Types....226

Generics and TypeVars....226

Restricting TypeVar to specific types....229

NamedTuple....230

TypedDict....230

TypeAlias....233

NewType....234

Using Type Annotations at Runtime....235

How to Add Type Annotations to Your Code....237

Adding Type Annotations to New Code....237

Adding Type Annotations to Existing Code (Gradual Typing)....238

Using .pyi Stub Files....239

Summary....240

6. Exceptions....241

The try Statement....241

tryexcept....242

tryfinally....244

tryexceptfinally....246

The raise Statement....246

The with Statement and Context Managers....248

Generators and Exceptions....251

Exception Propagation....251

Exception Objects....253

The Hierarchy of Standard Exceptions....253

Standard Exception Classes....254

OSError subclasses....255

Exceptions wrapping other exceptions or tracebacks....256

Custom Exception Classes....258

Custom Exceptions and Multiple Inheritance....260

Other Exceptions Used in the Standard Library....260

ExceptionGroup and except....261

Error-Checking Strategies....262

LBYL Versus EAFP....262

Handling Errors in Large Programs....264

Logging Errors....266

The logging module....266

Configuring logging....267

The assert Statement....269

7. Modules and Packages....271

Module Objects....271

The import Statement....272

The module body....272

Attributes of module objects....273

Python built-ins....275

Module documentation strings....276

Module-private variables....276

The from Statement....276

from...import ....277

from versus import....278

Handling import failures....278

Module Loading....279

Built-in Modules....279

Searching the Filesystem for a Module....279

The Main Program....281

Reloading Modules....282

Circular Imports....283

Custom Importers....284

Rebinding __import__....284

Import hooks....284

Packages....286

Special Attributes of Package Objects....287

Absolute Versus Relative Imports....287

Distribution Utilities (distutils) and setuptools....288

Python Environments....290

Enter the Virtual Environment....291

What Is a Virtual Environment?....292

Creating and Deleting Virtual Environments....292

Working with Virtual Environments....294

Managing Dependency Requirements....296

Other Environment Management Solutions....297

Best Practices with Virtualenvs....298

8. Core Built-ins and Standard Library Modules....301

Built-in Types....301

Built-in Functions....304

The sys Module....311

The copy Module....314

The collections Module....315

ChainMap....316

Counter....316

OrderedDict....318

defaultdict....318

deque....320

The functools Module....321

The heapq Module....323

The Decorate–Sort–Undecorate Idiom....324

The argparse Module....326

The itertools Module....327

9. Strings and Things....332

Methods of String Objects....332

The string Module....336

String Formatting....337

Formatted String Literals (F-Strings)....338

Debug printing with f-strings....339

Formatting Using format Calls....340

Value Conversion....342

Value Formatting: The Format Specifier....343

Fill and alignment....343

Sign indication....344

Zero normalization (z)....345

Radix indicator ()....346

Leading zero indicator (0)....346

Field width....346

Grouping option....347

Precision specification....348

Format type....348

Nested Format Specifications....350

Formatting of User-Coded Classes....351

Legacy String Formatting with %....352

Format Specifier Syntax....354

Text Wrapping and Filling....355

The pprint Module....356

The reprlib Module....356

Unicode....357

The codecs Module....357

The unicodedata Module....358

10. Regular Expressions....360

Regular Expressions and the re Module....360

REs and bytes Versus str....360

Pattern String Syntax....361

Common Regular Expression Idioms....363

Sets of Characters....365

Alternatives....366

Groups....367

Optional Flags....367

Match Versus Search....370

Anchoring at String Start and End....370

Regular Expression Objects....371

Match Objects....374

Functions of the re Module....375

REs and the : Operator....376

The Third-Party regex Module....377

11. File and Text Operations....378

The io Module....379

Creating a File Object with open....380

mode....381

Binary and text modes....381

Buffering....381

Sequential and nonsequential (random) access....382

Attributes and Methods of File Objects....382

Iteration on File Objects....384

File-Like Objects and Polymorphism....384

The tempfile Module....385

Auxiliary Modules for File IO....387

The fileinput Module....387

The struct Module....389

In-Memory Files: io.StringIO and io.BytesIO....391

Archived and Compressed Files....392

The tarfile Module....392

The TarFile class....394

The TarInfo class....395

The zipfile Module....396

The ZipFile class....397

The ZipInfo class....399

The os Module....400

Filesystem Operations....400

Path-string attributes of the os module....400

Permissions....401

File and directory functions of the os module....402

File descriptor operations....406

The os.path Module....408

OSError Exceptions....410

The errno Module....411

The pathlib Module....411

The stat Module....417

The filecmp Module....418

The fnmatch Module....420

The glob Module....421

The shutil Module....421

Text Input and Output....423

Standard Output and Standard Error....423

The print Function....423

Standard Input....424

The getpass Module....425

Richer-Text IO....426

The readline Module....426

Console IO....427

curses....428

The msvcrt module....428

Internationalization....428

The locale Module....429

The gettext Module....434

Using gettext for localization....434

Essential gettext functions....436

More Internationalization Resources....436

12. Persistence and Databases....438

Serialization....439

The csv Module....439

csv functions and classes....439

A csv example....440

The json Module....441

json functions....441

A json example....443

The pickle Module....443

pickle functions and classes....445

A pickling example....446

Pickling instances....447

Pickling customization with the copyreg module....448

The shelve Module....449

A shelving example....450

DBM Modules....451

The dbm Package....452

Examples of DBM-Like File Use....453

The Python Database API (DBAPI)....454

Exception Classes....454

Thread Safety....455

Parameter Style....455

Factory Functions....457

Type Description Attributes....457

The connect Function....458

Connection Objects....458

Cursor Objects....459

DBAPI-Compliant Modules....460

SQLite....461

The sqlite3.Connection class....463

The sqlite3.Row class....464

A sqlite3 example....464

13. Time Operations....466

The time Module....466

The datetime Module....469

The date Class....470

The time Class....471

The datetime Class....472

The timedelta Class....475

The tzinfo Abstract Class....476

The timezone Class....476

The zoneinfo Module....476

The dateutil Module....479

The sched Module....480

The calendar Module....481

14. Customizing Execution....484

Per-Site Customization....484

Termination Functions....485

Dynamic Execution and exec....485

Avoiding exec....486

Expressions....487

compile and Code Objects....487

Never exec or eval Untrusted Code....489

Internal Types....490

Type Objects....490

The Code Object Type....491

The Frame Type....491

Garbage Collection....491

The gc Module....492

Instrumenting garbage collection....495

The weakref Module....495

15. Concurrency: Threads and Processes....498

Threads in Python....500

The threading Module....501

Thread Objects....502

Thread Synchronization Objects....503

Timeout parameters....503

Lock and RLock objects....504

Condition objects....506

Event objects....507

Semaphore and BoundedSemaphore objects....508

Timer objects....509

Barrier objects....509

Thread Local Storage....511

The queue Module....512

The multiprocessing Module....514

Differences Between multiprocessing and threading....515

Structural differences....515

The Process class....515

Differences in queues....516

Sharing State: Classes Value, Array, and Manager....516

The Value class....517

The Array class....517

The Manager class....518

Process Pools....520

The Pool class....520

The AsyncResult class....522

The ThreadPool class....522

The concurrent.futures Module....523

Threaded Program Architecture....525

Process Environment....530

Running Other Programs....531

Using the Subprocess Module....531

What to run, and how....532

Subprocess files....532

Other, advanced arguments....532

Attributes of subprocess.Popen instances....533

Methods of subprocess.Popen instances....533

Running Other Programs with the os Module....534

The mmap Module....536

Methods of mmap Objects....537

Using mmap Objects for IPC....539

16. Numeric Processing....540

Floating-Point Values....540

The math and cmath Modules....544

The statistics Module....548

The operator Module....549

Random and Pseudorandom Numbers....551

The random Module....551

Crypto-Quality Random Numbers: The secrets Module....553

The fractions Module....554

The decimal Module....556

Array Processing....559

The array Module....559

Extensions for Numeric Array Computation....561

NumPy....561

Creating a NumPy array....562

Shape, indexing, and slicing....565

Matrix operations in NumPy....568

SciPy....570

Additional numeric packages....570

17. Testing, Debugging, and Optimizing....572

Testing....573

Unit Testing and System Testing....573

The doctest Module....576

The unittest Module....579

The TestCase class....582

Unit tests dealing with large amounts of data....584

Testing with nose2....585

Testing with pytest....586

Debugging....587

Before You Debug....588

The inspect Module....588

Introspecting callables....590

An example of using inspect....590

The traceback Module....592

The pdb Module....592

Other Debugging Modules....595

ipdb....596

pudb....596

The warnings Module....597

Classes....597

Objects....598

Filters....598

Functions....600

Optimization....601

Developing a Fast-Enough Python Application....602

Benchmarking....603

Large-Scale Optimization....604

List operations....607

String operations....607

Dictionary operations....607

Set operations....608

Summary of big-O times for operations on Python built-in types....608

Profiling....609

The profile module....609

Calibration....609

The pstats module....610

Small-Scale Optimization....611

The timeit module....612

Memoizing....613

Precomputing a lookup table....615

Building up a string from pieces....616

Searching and sorting....617

Avoid exec and from ... import ....618

Short-circuiting of Boolean expressions....618

Short-circuiting of iterators....619

Optimizing loops....620

Using multiprocessing for heavy CPU work....623

Optimizing IO....623

18. Networking Basics....626

The Berkeley Socket Interface....626

Socket Addresses....628

ClientServer Computing....628

Connectionless client and server structures....629

Connection-oriented client and server structures....630

The socket Module....632

Socket Objects....634

A Connectionless Socket Client....638

A Connectionless Socket Server....639

A Connection-Oriented Socket Client....640

A Connection-Oriented Socket Server....641

Transport Layer Security....642

SSLContext....643

19. Client-Side Network Protocol Modules....646

Email Protocols....646

The poplib Module....647

The smtplib Module....648

HTTP and URL Clients....649

URL Access....649

The urllib Package....650

The urllib.parse module....650

The urllib.request module....652

The Third-Party requests Package....653

Sending requests....653

requests optional named parameters....654

The files argument (and other ways to specify the requests body)....654

How to interpret requests examples....655

The Response class....656

Other Network Protocols....657

20. Serving HTTP....660

http.server....660

WSGI....661

WSGI Servers....662

ASGI....662

Python Web Frameworks....662

Full-Stack Versus Lightweight Frameworks....663

A Few Popular Full-Stack Frameworks....663

Considerations When Using Lightweight Frameworks....663

A Few Popular Lightweight Frameworks....664

Flask....664

Flask request objects....666

Flask response objects....666

FastAPI....668

21. Email, MIME, and Other Network Encodings....673

MIME and Email Format Handling....673

Functions in the email Package....673

The email.message Module....674

The email.Generator Module....677

Creating Messages....677

The email.encoders Module....678

The email.utils Module....678

Example Uses of the email Package....679

Encoding Binary Data as ASCII Text....680

The base64 Module....681

The quopri Module....682

The uu Module....683

22. Structured Text: HTML....684

The html.entities Module....685

The BeautifulSoup Third-Party Package....685

The BeautifulSoup Class....686

Which parser BeautifulSoup uses....686

BeautifulSoup, Unicode, and encoding....687

The Navigable Classes of bs4....688

Indexing instances of Tag....688

Getting an actual string....689

Attribute references on instances of BeautifulSoup and Tag....691

contents, children, and descendants....692

parent and parents....692

next_sibling, previous_sibling, next_siblings, and previous_siblings....693

next_element, previous_element, next_elements, and previous_elements....693

bs4 find Methods (aka Search Methods)....694

Arguments of search methods....695

Filters....695

name....696

string....697

attrs....697

Other named arguments....698

bs4 CSS Selectors....698

An HTML Parsing Example with BeautifulSoup....698

Generating HTML....699

Editing and Creating HTML with bs4....699

Building and adding new nodes....700

Replacing and removing nodes....701

Building HTML with bs4....702

Templating....703

The jinja2 Package....704

The jinja2.Environment class....704

The jinja2.Template class....705

Building HTML with jinja2....705

23. Structured Text: XML....708

ElementTree....708

The Element Class....709

The ElementTree Class....712

Functions in the ElementTree Module....713

Parsing XML with ElementTree.parse....714

Selecting Elements from an ElementTree....715

Editing an ElementTree....716

Building an ElementTree from Scratch....717

Parsing XML Iteratively....717

24. Packaging Programs and Extensions....720

What We Dont Cover in This Chapter....721

A Brief History of Python Packaging....722

Online Material....723

25. Extending and Embedding Classic Python....725

Online Material....726

26. v3.7 to v3.n Migration....728

Significant Changes in Python Through 3.11....728

Planning a Python Version Upgrade....731

Choosing a Target Version....732

Scoping the Work....732

Applying the Code Changes....733

Upgrade Automation Using pyupgrade....734

Multiversion Testing....735

Use a Controlled Deployment Process....735

How Often Should You Upgrade?....735

Summary....735

A. New Features and Changes in Python 3.7 Through 3.11....737

Python 3.7....737

Python 3.8....740

Python 3.9....743

Python 3.10....745

Python 3.11....748

Index....751

Python was recently ranked as today's most popular programming language on the TIOBE index, thanks to its broad applicability to design and prototyping to testing, deployment, and maintenance. With this updated fourth edition, you'll learn how to get the most out of Python, whether you're a professional programmer or someone who needs this language to solve problems in a particular field.

Carefully curated by recognized experts in Python, this new edition focuses on version 3.10, bringing this seminal work on the Python language fully up to date on five version releases, including preview coverage of upcoming 3.11 features.

This handy guide will help you:

• Learn how Python represents data and program as objects
• Understand the value and uses of type annotations
• Examine which language features appeared in which recent versions
• Discover how to use modern Python idiomatically
• Learn ways to structure Python projects appropriately
• Understand how to debug Python code