Expert Python Programming: Master Python by learning the best coding practices and advanced programming concepts. 4 Ed

Cover....1

Copyright....3

Contributors....4

Table of Contents....6

Preface....14

Chapter 1: Current Status of Python....20

Where are we now and where are we going?....21

What to do with Python 2....22

Keeping up to date....24

PEP documents....25

Active communities....27

Other resources....30

Summary....31

Chapter 2: Modern Python Development Environments....34

Technical requirements....35

Python's packaging ecosystem....36

Installing Python packages using pip....36

Isolating the runtime environment....38

Application-level isolation versus system-level isolation....42

Application-level environment isolation....43

Poetry as a dependency management system....46

System-level environment isolation....51

Containerization versus virtualization....53

Virtual environments using Docker....55

Writing your first Dockerfile....56

Running containers....60

Setting up complex environments....62

Useful Docker and Docker Compose recipes for Python....65

Virtual development environments using Vagrant....75

Popular productivity tools....78

Custom Python shells....78

Using IPython....80

Incorporating shells in your own scripts and programs....84

Interactive debuggers....85

Other productivity tools....87

Summary....89

Chapter 3: New Things in Python....90

Technical requirements....91

Recent language additions....91

Dictionary merge and update operators....92

Alternative – Dictionary unpacking....95

Alternative – ChainMap from the collections module....95

Assignment expressions....98

Type-hinting generics....102

Positional-only parameters....103

zoneinfo module....106

graphlib module....107

Not that new, but still shiny....112

breakpoint() function....112

Development mode....113

Module-level __getattr__() and __dir__() functions....116

Formatting strings with f-strings....117

Underscores in numeric literals....119

secrets module....119

What may come in the future?....120

Union types with the | operator....121

Structural pattern matching....122

Summary....127

Chapter 4: Python in Comparison with Other Languages....128

Technical requirements....129

Class model and object-oriented programming ....129

Accessing super-classes....131

Multiple inheritance and Method Resolution Order....133

Class instance initialization....139

Attribute access patterns....143

Descriptors....144

Real-life example – lazily evaluated attributes....147

Properties....151

Dynamic polymorphism....157

Operator overloading....159

Dunder methods (language protocols)....160

Comparison to C++....164

Function and method overloading....166

Single-dispatch functions....168

Data classes....170

Functional programming....174

Lambda functions....176

The map(), filter(), and reduce() functions....178

Partial objects and partial functions....181

Generators....182

Generator expressions....184

Decorators....185

Enumerations....187

Summary....190

Chapter 5: Interfaces, Patterns, and Modularity....192

Technical requirements....193

Interfaces....194

A bit of history: zope.interface....196

Using function annotations and abstract base classes....205

Using collections.abc....210

Interfaces through type annotations....211

Inversion of control and dependency injection....214

Inversion of control in applications....216

Using dependency injection frameworks....225

Summary....231

Chapter 6: Concurrency....232

Technical requirements....233

What is concurrency?....233

Multithreading....235

What is multithreading?....236

How Python deals with threads....240

When should we use multithreading?....242

Application responsiveness....242

Multiuser applications....243

Work delegation and background processing....244

An example of a multithreaded application....245

Using one thread per item....248

Using a thread pool....250

Using two-way queues....255

Dealing with errors in threads....257

Throttling....260

Multiprocessing....264

The built-in multiprocessing module....266

Using process pools....270

Using multiprocessing.dummy as the multithreading interface....273

Asynchronous programming....274

Cooperative multitasking and asynchronous I/O....275

Python async and await keywords....276

A practical example of asynchronous programming....281

Integrating non-asynchronous code with async using futures....284

Executors and futures....286

Using executors in an event loop....287

Summary....288

Chapter 7: Event-Driven Programming....290

Technical requirements....291

What exactly is event-driven programming?....291

Event-driven != asynchronous....292

Event-driven programming in GUIs....293

Event-driven communication....296

Various styles of event-driven programming....298

Callback-based style....299

Subject-based style....300

Topic-based style....305

Event-driven architectures....307

Event and message queues....309

Summary....312

Chapter 8: Elements of Metaprogramming....314

Technical requirements....315

What is metaprogramming?....315

Using decorators to modify function behavior before use ....316

One step deeper: class decorators....318

Intercepting the class instance creation process....323

Metaclasses....326

The general syntax....328

Metaclass usage....331

Metaclass pitfalls....334

Using the __init__subclass__() method as an alternative to metaclasses....336

Code generation....338

exec, eval, and compile....338

The abstract syntax tree ....340

Import hooks....342

Notable examples of code generation in Python....342

Falcon's compiled router....343

Hy....344

Summary....345

Chapter 9: Bridging Python with C and C++....346

Technical requirements....348

C and C++ as the core of Python extensibility....348

Compiling and loading Python C extensions....349

The need to use extensions....351

Improving performance in critical code sections....352

Integrating existing code written in different languages....353

Integrating third-party dynamic libraries....354

Creating efficient custom datatypes....354

Writing extensions....355

Pure C extensions....356

A closer look at the Python/C API....360

Calling and binding conventions....364

Exception handling....368

Releasing GIL....370

Reference counting....372

Writing extensions with Cython....375

Cython as a source-to-source compiler....375

Cython as a language....379

Downsides of using extensions....381

Additional complexity....382

Harder debugging....383

Interfacing with dynamic libraries without extensions....384

The ctypes module....384

Loading libraries....384

Calling C functions using ctypes....386

Passing Python functions as C callbacks....388

CFFI....391

Summary....393

Chapter 10: Testing and Quality Automation....396

Technical requirements....397

The principles of test-driven development....398

Writing tests with pytest....400

Test parameterization....408

pytest's fixtures....411

Using fakes....421

Mocks and the unittest.mock module....424

Quality automation....429

Test coverage....430

Style fixers and code linters....434

Static type analysis....438

Mutation testing....439

Useful testing utilities....446

Faking realistic data values....446

Faking time values....448

Summary....449

Chapter 11: Packaging and Distributing Python Code....452

Technical requirements....453

Packaging and distributing libraries....453

The anatomy of a Python package....454

setup.py....457

setup.cfg....459

MANIFEST.in....459

Essential package metadata....461

Trove classifiers....462

Types of package distributions....464

sdist distributions....464

bdist and wheel distributions....466

Registering and publishing packages....469

Package versioning and dependency management....472

The SemVer standard for semantic versioning....474

CalVer for calendar versioning....475

Installing your own packages....476

Installing packages directly from sources....476

Installing packages in editable mode....477

Namespace packages....478

Package scripts and entry points....480

Packaging applications and services for the web....484

The Twelve-Factor App manifesto....485

Leveraging Docker....486

Handling environment variables....489

The role of environment variables in application frameworks....494

Creating standalone executables....499

When standalone executables are useful....500

Popular tools....500

PyInstaller....501

cx_Freeze....505

py2exe and py2app....507

Security of Python code in executable packages....509

Summary....510

Chapter 12: Observing Application Behavior and Performance....512

Technical requirements....513

Capturing errors and logs....513

Python logging essentials....514

Logging system components....516

Logging configuration....524

Good logging practices....528

Distributed logging....530

Capturing errors for later review....533

Instrumenting code with custom metrics....537

Using Prometheus....539

Distributed application tracing....549

Distributed tracing with Jaeger....553

Summary....559

Chapter 13: Code Optimization....560

Technical requirements....561

Common culprits for bad performance....561

Code complexity....562

Cyclomatic complexity....563

The big O notation....564

Excessive resource allocation and leaks....567

Excessive I/O and blocking operations....568

Code profiling....568

Profiling CPU usage....570

Macro-profiling....570

Micro-profiling....576

Profiling memory usage....579

Using the objgraph module....581

C code memory leaks....589

Reducing complexity by choosing appropriate data structures....590

Searching in a list....590

Using sets....592

Using the collections module....593

deque....593

defaultdict....595

namedtuple....597

Leveraging architectural trade-offs....599

Using heuristics and approximation algorithms....599

Using task queues and delayed processing....600

Using probabilistic data structures....604

Caching....605

Deterministic caching....606

Non-deterministic caching....609

Summary....614

Why subscribe?....616

Packt Page....616

Other Books You May Enjoy....618

Index....620

Attain a deep understanding of building, maintaining, packaging, and shipping robust Python applications

Key Features:

• Discover the new features of Python, such as dictionary merge, the zoneinfo module, and structural pattern matching
• Create manageable code to run in various environments with different sets of dependencies
• Implement effective Python data structures and algorithms to write, test, and optimize code

Book Description:

Python is used in a wide range of domains owing to its simple yet powerful nature. Although writing Python code is easy, making it readable, reusable, and easy to maintain can be challenging. Complete with best practices, useful tools, and standards implemented by professional Python developers, this fourth edition will help you in not only overcoming such challenges but also learning Python's latest features and advanced concepts. The book begins with a warm-up, where you will catch-up with the latest Python improvements, syntax elements, and interesting tools to boost your development efficiency. Further, the initial few chapters should allow experienced programmers coming from different languages to safely land in the Python ecosystem. As you progress, you will explore common software design patterns and various programming methodologies, such as event-driven programming, concurrency, and metaprogramming. You will also go through complex code examples and try to solve meaningful problems by bridging Python with C and C++, writing extensions that benefit from the strengths of multiple languages. Finally, you will understand the complete lifetime of any application after it goes live. By the end of the book, you should be proficient in writing efficient and maintainable Python code.

What You Will Learn:

• Explore modern ways of setting up repeatable and consistent Python development environments
• Effectively package Python code for community and production use
• Learn modern syntax elements of Python programming, such as f-strings, enums, and lambda functions
• Demystify metaprogramming in Python with metaclasses
• Write concurrent code in Python
• Extend and integrate Python with code written in different languages

Who this book is for:

 The Python programming book is intended for expert programmers who want to learn Python's advanced-level concepts and latest features.

Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to Python 3.9 for those who are still a bit behind and continue to use other older versions.