Deep Dive Python: Techniques and Best Practices for Developers

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Lists
Initializing Empty Lists
Indexing
Slicing
Dynamic Slices
Example from scikit-learn
Snippet from IPython
Full Slice
Example from Pandas
List Comprehensions
With Conditional Filtering
With Tuple Unpacking
Nested Comprehensions
Nested Loops
Multidimensional Lists
Copying Lists
Shallow Copy
Deep Copy
Sorting Lists
Built-In Key Functions
The attrgetter Function
The itemgetter Function
The methodcaller Function
The cmp_to_key Function
Stability of sort
Custom List Classes
Inheriting from the list Class
SQLAlchemy
Inheriting from UserList
Androguard
OpenMMLab mmrazor
MutableSequence from collections.abc
Choosing Between list, UserList, and MutableSequence
Conclusion
Chapter 2: Tuples
Initializing Tuples
Immutability
Lists vs. Tuples
Tuple Packing and Unpacking
Tuple Packing
Tuple Unpacking
Variable-Length Assignment
Unpacking in Loops
Ignoring Values
Comparisons and Sorting
Version Checking
As Sort Keys
Sorting Tuples
Tuples as Dict Keys
namedtuple (collections.namedtuple)
typing.NamedTuple
Conclusion
Chapter 3: Set and Dictionary
Set in Python
Dict in Python
Comprehensions
Dictionary Comprehensions
Set Comprehensions
Hashability
Hashability in Sets
Hashability in Dictionaries
Hash Collisions
Custom Hashable Objects
More Set Operations
Symmetric Difference
Subset and Superset Checks
Disjoint Set Check
Clearing a Set
The pop Method
Update Methods
The update Method
Intersection Update
Difference Update
Symmetric Difference Update
More Dictionary Operations
The setdefault() Method
The popitem() Method
The update() Method
The fromkeys() Method
Dictionary View Objects
Dictionary Packing and Unpacking
Packing **kwargs into a Dictionary
Dictionary Unpacking
Dictionary Merging and Unpacking
Set Variant: frozenset
Dictionary: Variants
collections.defaultdict
collections.OrderedDict
collections.Counter
collections.ChainMap
Custom Dictionary Classes
From the Built-In dict
From MutableMapping
From UserDict
Custom Set Class: MutableSet
Typing Sets and Dicts
Typing Dictionaries
Typing Sets
TypedDict
Conclusion
Chapter 4: Logging
Configuration
Using logging.basicConfig
DictConfig
FileConfig
Log Levels
Using logging.error vs. logging.exception
Loggers
Formatters
Handlers
SMTPHandler
StreamHandler
Log Rotation and Timed File Handlers
RotatingFileHandler
TimedRotatingFileHandler
Custom Handlers
LogRecord
Filters
LoggerAdapter
Log Propagation
The Logging Flow
Logger Flow
Handler Flow
Logger Propagation
Logging in Libraries
Conclusion
Chapter 5: Exceptions
Exceptions in Python
Built-In Exceptions
Tracebacks
Exception Base Classes and Exception Hierarchy
The BaseException and Exception Classes
BaseException
Exception
Built-In Exception Hierarchy
Handling Exceptions: The try Block
Exception Chaining
The __cause__, __context__, and __suppress_context__ Attributes
__cause__
__context__
__suppress_context__
Raise from Exception
Raise from None
Exceptions in Threads and Processes
Exceptions in Threads
Handling Exceptions in ThreadPoolExecutor
Exceptions in Multiprocessing and ProcessPoolExecutor
Custom Exception Classes
The traceback Module
traceback.print_stack
traceback.print_exc
traceback.format_exc
traceback.format_exception_only
sys module: excepthook and exc_info
sys.exc_info
sys.excepthook
Zero-Cost Exceptions
Conclusion
Chapter 6: Functions and Functools
Positional and Keyword Arguments
Positional-Only Arguments
Keyword-Only Arguments
Argument-Passing Model in Python
Passing Immutable Objects
Passing Mutable Objects
Argument Unpacking and Packing
Argument Unpacking
Argument Packing
Recursion
Default Arguments
Nested Functions/Closures
The nonlocal Keyword
Higher-Order Functions
map
filter
Function Introspection
Accessing Function Signature
Accessing Annotations
Function Attributes and Docstring
The functools Module
partial
partialmethod
lru_cache
cached_property
total_ordering
reduce
singledispatch
singledispatchmethod
Conclusion
Chapter 7: Classes and Object- Oriented Programming
Class Methods
Static Method
Namespace Dictionary
Property
Private Attributes and Name Mangling
Name Mangling
Slotted Classes
Attribute Access Using __getattr__ and  __getattribute__
Dataclass
Why a Dataclass
Frozen Dataclass
Dataclass Field
Post Init
Inheritance
Inheritance in Python
Method Resolution Order
The super Built-In
The object Built-In
Alternatives to Inheritance
Mixins
When Not to Use Classes
Conclusion
Chapter 8: Dunder Methods
__str__ and __repr__ Dunder Methods
__add__ and __sub__: Addition and Subtraction Operations
__eq__ and __lt__: Equality Check and Comparison
__int__ and __float__: Conversion to Numbers
__len__: Length Calculation
__call__: Callable Objects
__hash__: Hashability and Hash Calculation
__contains__, __getitem__, and __setitem__: Membership Checks and Item Access
__getattr__ and __getattribute__: Attribute Access
__setattr__: Setting Attributes
__missing__: Handle Missing Keys
__copy__ and __deepcopy__: Copying Objects
__instancecheck__ and __subclasscheck__: Instance and Subclass Checks
Conclusion
Chapter 9: Decorators
Closures
Decorators: Syntactic Sugar
Usage of functools.wraps
Practical Decorators
Nesting Decorators
Decorators with Arguments
Decorators with Optional Arguments Only
Decorating Classes
Using Classes as Decorators
Decorators with Asyncio Support
Context Management Decorators
Decorator Use Cases
Caching
Timing
Registering
Validation
Conditional Execution
Logging
Exception Handling
Conclusion
Chapter 10: Metaclasses
Class Creation Process
type in Python
Calling with a Single Argument
Calling with Three Arguments
Defining Metaclasses
Metamethods
Defining a Custom Metaclass
Method Propagation
Inheritance Behavior
Metaclass Determination Order
Metaclass Usage Patterns
Registration
Validation or Enforcement
Modifying Instance Creation
Modifying Instance Features
Usage in Enum
Metaclass Alternatives
PEP 487
Class Decorators
Conclusion
Chapter 11: Typing
Type Checking
Annotating Variables
Protocols
TypeVar
Static Type Checking
urllib3
Runtime Type Checking
Nebuly
Libraries Leveraging Type Hints
Pydantic
SQLModel
Type Stubs
Adopting Type Annotations in Your Project
Conclusion
Chapter 12: Generators and Iterators
Iterators
The Iterator Protocol
next() and __iter__ Methods
The iter() Function and __iter__() Dunder Method
The next() Function and __next__() Dunder Method
Converting Functions to Iterators Using iter()
Async Iterators
The Async Iterator Protocol
Generators
Generator vs. Iterators
Yield and Yield From
Yield
Yield From
Async Generators
Generator Methods: send(), throw(), and close()
send()
throw()
close()
Generator Comprehension
Conclusion
Chapter 13: Itertools
count
cycle
repeat
product
permutations
combinations
accumulate
batched
chain
compress
dropwhile
filterfalse
groupby
islice
pairwise
starmap
takewhile
tee
zip_longest
Conclusion
Chapter 14: Multithreading
Thread Class for Creating Threads
Implications of the GIL on Threading
Daemon Threads
Thread-Local Data
threading.Timer
ThreadPoolExecutor
Handling Exceptions in Threads
Custom Thread Classes
Thread Safety
Race Conditions
Locks: Mutual Exclusion
Lock
Deadlock
RLock
Semaphores: Limiting Access
Events: Signaling
Conditions: Conditional Waiting
Conclusion
Chapter 15: Multiprocessing
Processes vs. Threads
Bypassing the Global Interpreter Lock (GIL)
When to Use Multiprocessing
Creating and Managing Processes
The multiprocessing.Process Class
Starting Processes with start()
Waiting for Processes to Finish with join()
Custom Processes with a Class
Daemon Processes
Checking Process Information
Inter-process Communication (IPC) and Data Sharing
Why Processes Don’t Share Memory
Shared Memory with multiprocessing.Value and  multiprocessing.Array
Example: Using Value to Share a Counter
Example: Using Array to Share a List of Numbers
Message Passing with Pipes (multiprocessing.Pipe)
Example: Sending a Message from Child to Parent
Message Passing with Queues (multiprocessing.Queue and multiprocessing.SimpleQueue)
Example: Multiple Processes Sending Results to a Shared Queue
Managers for Shared Objects
Example: Using a Manager to Share a Dictionary
Process Pools for Task Distribution
Introduction to multiprocessing.Pool
Example: Using pool.map() to Compute Squares in Parallel
Submitting Tasks to the Pool: apply_async() and map_async()
apply_async(func, args=(), kwds={}, callback=None, error_callback=None)
map_async(func, iterable, chunksize=None, callback=None, error_callback=None)
Example: Using apply_async() and map_async()
concurrent.futures.ProcessPoolExecutor: A Modern Alternative
Example: Using ProcessPoolExecutor to Compute Squares
Handling Results and Exceptions in Pools
Example: Handling Exceptions from Worker Functions
Comparing Pool and Executor with Threads
Process Start Methods: Choosing the Right Approach
Explanation of Start Methods: spawn, fork, forkserver
spawn
fork
forkserver
When to Use Each Start Method and Platform Dependencies
When to Choose spawn
When to Choose fork
When to Choose forkserver
Setting the Start Method in Code
Synchronization Between Processes
Need for Synchronization Primitives in Multiprocessing
Overview and Usage of Synchronization Primitives
Locks and RLocks (multiprocessing.Lock, multiprocessing.RLock)
Example: Protecting a Shared Counter
Semaphores (multiprocessing.Semaphore, multiprocessing.BoundedSemaphore)
Events (multiprocessing.Event)
Example: Starting All Workers at the Same Time
Conditions (multiprocessing.Condition)
Example: Signaling a Consumer When a Queue Is Non-empty
Barrier (multiprocessing.Barrier)
Example: Synchronizing Three Processes
Real-World Use: Shared Locks with Managers
Example: Locking Access to a Shared Dictionary
Using Synchronization Primitives with Context Managers
Shared State vs. Message Passing
Conclusion
Chapter 16: Asyncio
What Is Asynchronous Programming?
Event Loop and Cooperative Multitasking
async and await Keywords
When to Use Asyncio (I/O Bound vs. CPU Bound)
Benefits of Asyncio
Context Switching in Asyncio
Creating and Running Asyncio Programs
async def for Coroutines
asyncio.run() to Start the Event Loop
Basic Example: Concurrent Network Requests
Working with Tasks
Creating Tasks with asyncio.create_task()
Task Cancellation
Synchronization Primitives
Asyncio Lock
Asyncio Semaphore
Asyncio Event
Asyncio Condition
Context Variables
The Need for Context Variables in Asyncio
Creating and Using Context Variables
Context Variables and Task Hierarchies
Default Values and Token Management
Concurrency Patterns in Asyncio
Fan-Out/Fan-In
Producer–Consumer
Worker Pool
Error Handling in Asyncio
Try–Except Blocks in Coroutines
Exceptions in Tasks
Handling asyncio.CancelledError
Best Practices
Multithreading vs. Multiprocessing vs. Asyncio
Multithreading
Multiprocessing
Asyncio
Summary Table
Conclusion
Chapter 17: Data Serialization and Persistence
JSON
json.dumps()
json.loads()
json.dump() and json.load()
JSONEncoder Class
JSONDecoder Class
Pickle
Pickle Module Functions
pickle.dump
pickle.load
pickle.loads() and pickle.dumps()
Pickler and Unpickler Classes
Picklability
Pickling Limitations
Pickling Class Restrictions
Class Constructor Behavior
Pickling Function Restrictions
Pickling Stateful Objects
The __getstate__() Method
The __setstate__() Method
Pickle vs. JSON
Human Readability
Interoperability
Type Support
Security
Performance
Shelve
Key Methods
Shelf Object Methods
Shelve vs. Pickle
Differences Between shelve and pickle
Marshal
CSV
The reader() and writer() Functions
The DictReader() and DictWriter() Classes
CSV Dialects and Sniffer
XML
xml.etree.ElementTree (etree)
xml.dom
xml.sax
The etree Module
Conclusion
Chapter 18: Context Managers and Contextlib
Context Managers in the Standard Library
File Handling
Locks for Thread Safety
Managing Thread Pools
SQLite Database Connection Management
Defining Custom Context Managers Using Dunders
Context Manager Base Classes
AbstractAsyncContextManager
Decorators for Defining a Context Manager
@contextmanager
@asynccontextmanager
Context Manager as a Decorator
contextlib.ExitStack()
AsyncExitStack
Built-In Context Managers from contextlib.suppress
closing
redirect_stdout
Use Cases
Timer
Patching
Conclusion
Chapter 19: Abstract Base Classes
abc.ABC and abc.ABCMeta
The abc.ABC Class
The abc.ABCMeta Metaclass
ABC Method Decorators
@abstractmethod Decorator
Decorating Class Methods
Decorating Static Methods
Decorating Property Methods
Subclassing ABCs
Use of isinstance() and issubclass() with ABCs
Multiple Inheritance with ABCs
The register() Method: Registering Virtual Subclasses
The __subclasshook__() Method
The collections.abc Module
collections.abc.MutableSequence
collections.abc.Iterable
collections.abc.Iterator
collections.abc.Coroutine
Conclusion
When to Use Abstract Base Classes
When to Avoid Abstract Base Classes
ABCs vs. Protocols
Chapter 20: Packaging
PyPI
History
Egg Files
Distutils
Wheel
pyproject.toml
Tools
Setuptools
Flit
Poetry
Hatch
Twine
Best Practices
__init__.py Files in Python Packages
Example: __init__.py in Action
Best Practices for __init__.py
Code Organization: Structuring Packages for Clarity and Maintainability
Managing Private Python Packages for Internal Use
Hosting Options for Private Packages
pyproject.toml and Private Packages
Classifiers and the Special “Private:: Do Not Upload” Classifier
What Classifiers Do
Optional Dependencies with “Extras”
Defining Extras in pyproject.toml
Installing Packages with Extras
Automating Package Deployments with CI/CD
Example: GitHub Action Workflow for PyPI Release
Version Management
Semantic Versioning
Alternative Versioning Schemes
Tools for Automated Versioning
Testing Packages
Types of Testing for Python Packages
Environment Testing with tox
Security Considerations
Auditing Dependencies with Safety
Best Practices for Secure Packaging
Handling Security Vulnerabilities
Conclusion
Chapter 21: Dependency Management
Virtual Environments
Managing Python Versions
Deterministic Builds
Dependency Specifiers
Locking Dependencies
Tools
pip
pip-tools
pipenv
pdm
uv
Dependency Conflicts
Security in Dependency Management
Typosquatting Attacks
Supply Chain Attacks
Auditing Dependencies
Pip-audit
Safety
Hash-Checking Mode to Secure Dependencies
Conclusion
Index

Take your programming skills to the next level by expanding real-world Python applications in open-source code. This book is an essential guide for experienced developers looking to enhance their Python proficiency.

Using code examples from popular open-source projects you’ll master Python concepts and see how they are applied in production-level code. By understanding code from established projects, you’ll develop a deeper appreciation for Python’s capabilities and learn best practices that have been battle-tested in the real world. You’ll gain the confidence to tackle complex projects, optimize your code, and implement advanced Python features effectively.

You’ll also examine typing, generators, iterators, the itertools module, multithreading, multiprocessing, and asyncio. This will equip you to build production-ready Python programs and libraries. Whether you’re building large-scale applications, contributing to open-source projects, or solving intricate programming challenges, Deep Dive Python will provide you with the knowledge and practical experience to excel in your Python development journey.

What You Will Learn

• Master Python's core data structures and how to use them effectively.

• Implement decorators, descriptors, metaclasses, and abstract base classes in your code

• Enhance your code with functional programming techniques using the functools module

• Improve iteration with generators, iterators, and the itertools module

• Select the best Python concurrent programming models to use for optimal performance

• Apply best practices for code organization, packaging, and creating maintainable Python projects

Who This Book Is For

Experienced Python developers who want to expand their programming skils