Python Polars: The Definitive Guide: Transforming, Analyzing, and Visualizing Data with a Fast and Expressive DataFrame API

Foreword xvii
Preface xxi
Part I. Begin
1. Introducing Polars....3
What Is This Thing Called Polars?....4
Key Features....4
Key Concepts....4
Advantages....5
Why You Should Use Polars....5
Performance....6
Usability....6
Popularity....7
Sustainability....8
Polars Compared to Other Data Processing Packages....8
Why We Focus on Python Polars....10
How This Book Is Organized....10
An ETL Showcase....11
Extract....12
Bonus: Visualizing Neighborhoods and Stations....17
Transform....21
Bonus: Visualizing Daily Trips per Borough....26
Load....28
Bonus: Becoming Faster by Being Lazy....29
Takeaways....32
2. Getting Started....33
Setting Up Your Environment....33
Downloading the Project....34
Installing uv....35
Installing the Project....35
Working with the Virtual Environment....35
Verifying Your Installation....36
Crash Course in JupyterLab....37
Keyboard Shortcuts....38
Installing Polars on Other Projects....39
All Optional Dependencies....40
Optional Dependencies for Interoperability....40
Optional Dependencies for Working with Spreadsheets....40
Optional Dependencies for Working with Databases....41
Optional Dependencies for Working with Remote Filesystems....41
Optional Dependencies for Other I/O Formats....41
Optional Dependencies for Extra Functionality....42
Installing Optional Dependencies....42
Configuring Polars....42
Temporary Configuration Using a Context Manager....43
Local Configuration Using a Decorator....46
Compiling Polars from Scratch....46
Edge Case: Very Large Datasets....47
Edge Case: Processors Lacking AVX Support....48
Takeaways....48
3. Moving from pandas to Polars....49
Animals....50
Similarities to Recognize....50
Appearances to Appreciate....51
Differences in Code....51
Differences in Display....52
Concepts to Unlearn....57
Index....57
Axes....58
Indexing and Slicing....59
Eagerness....61
Relaxedness....63
Syntax to Forget....64
Common Operations Side By Side....65
To and From pandas....69
Takeaways....70
Part II. Form
4. Data Structures and Data Types....73
Series, DataFrames, and LazyFrames....73
Data Types....75
Nested Data Types....77
Missing Values....79
Data Type Conversion....84
Takeaways....86
5. Eager and Lazy APIs....87
Eager API: DataFrame....87
Lazy API: LazyFrame....89
Performance Differences....90
Functionality Differences....91
Attributes....92
Aggregation Methods....92
Computation Methods....93
Descriptive Methods....93
GroupBy Methods....94
Exporting Methods....94
Manipulation and Selection Methods....95
Miscellaneous Methods....97
Tips and Tricks....98
Going from LazyFrame to DataFrame and Vice Versa....98
Joining a DataFrame with a LazyFrame....99
Caching Intermittent Results....100
Takeaways....101
6. Reading and Writing Data....103
Format Overview....104
Reading CSV Files....105
Parsing Missing Values Correctly....107
Reading Files with Encodings Other Than UTF-8....108
Reading Excel Spreadsheets....110
Working with Multiple Files....111
Reading Parquet....114
Reading JSON and NDJSON....115
JSON....115
NDJSON....118
Other File Formats....120
Querying Databases....121
Writing Data....123
CSV Format....123
Excel Format....124
Parquet Format....124
Other Considerations....125
Takeaways....125
Part III. Express
7. Beginning Expressions....129
Methods and Namespaces....131
Expressions by Example....131
Selecting Columns with Expressions....132
Creating New Columns with Expressions....133
Filtering Rows with Expressions....135
Aggregating with Expressions....135
Sorting Rows with Expressions....136
The Definition of an Expression....137
Properties of Expressions....139
Creating Expressions....141
From Existing Columns....142
From Literal Values....143
From Ranges....145
Other Functions to Create Expressions....146
Renaming Expressions....147
Expressions Are Idiomatic....149
Takeaways....151
8. Continuing Expressions....153
Types of Operations....154
Example A: Element-Wise Operations....155
Example B: Operations That Summarize to One....155
Example C: Operations That Summarize to One or More....156
Example D: Operations That Extend....156
Element-Wise Operations....157
Operations That Perform Mathematical Transformations....157
Operations Related to Trigonometry....159
Operations That Round and Categorize....160
Operations for Missing or Infinite Values....161
Other Operations....163
Nonreducing Series-Wise Operations....164
Operations That Accumulate....164
Operations That Fill and Shift....166
Operations Related to Duplicate Values....167
Operations That Compute Rolling Statistics....168
Operations That Sort....170
Other Operations....171
Series-Wise Operations That Summarize to One....172
Operations That Are Quantifiers....173
Operations That Compute Statistics....174
Operations That Count....176
Other Operations....178
Series-Wise Operations That Summarize to One or More....179
Operations Related to Unique Values....179
Operations That Select....180
Operations That Drop Missing Values....181
Other Operations....182
Series-Wise Operations That Extend....185
Takeaways....185
9. Combining Expressions....187
Inline Operators Versus Methods....188
Arithmetic Operations....190
Comparison Operations....191
Boolean Algebra Operations....195
Bitwise Operations....197
Using Functions....199
When, Then, Otherwise....202
Takeaways....204
Part IV. Transform
10. Selecting and Creating Columns....209
Selecting Columns....211
Introducing Selectors....212
Selecting Based on Name....213
Selecting Based on Data Type....214
Selecting Based on Position....216
Combining Selectors....218
Creating Columns....220
Related Column Operations....225
Dropping....225
Renaming....225
Stacking....226
Adding Row Indices....227
Takeaways....227
11. Filtering and Sorting Rows....229
Filtering Rows....230
Filtering Based on Expressions....230
Filtering Based on Column Names....231
Filtering Based on Constraints....232
Sorting Rows....233
Sorting Based on a Single Column....234
Sorting in Reverse....235
Sorting Based on Multiple Columns....235
Sorting Based on Expressions....236
Sorting Nested Data Types....237
Related Row Operations....239
Filtering Missing Values....239
Slicing....240
Top and Bottom....241
Sampling....241
Semi-Joins....241
Takeaways....242
12. Working with Textual, Temporal, and Nested Data Types....245
String....246
String Methods....246
String Examples....248
Categorical....252
Categorical Methods....253
Categorical Examples....253
Enum....256
Temporal....257
Temporal Methods....257
Temporal Examples....259
List....263
List Methods....263
List Examples....265
Array....267
Array Methods....267
Array Examples....268
Struct....270
Struct Methods....270
Struct Examples....271
Takeaways....274
13. Summarizing and Aggregating....275
Split, Apply, and Combine....276
GroupBy Context....276
The Descriptives....279
Advanced Methods....284
Row-Wise Aggregations....289
Window Functions in Selection Context....291
Dynamic Grouping....293
Rolling Aggregations....294
Upsampling....297
Takeaways....299
14. Joining and Concatenating....301
Joining....301
Join Strategies....302
Joining on Multiple Columns....306
Validation....306
Inexact Joining....308
Inexact Join Strategies....310
Additional Fine-Tuning....312
Use Case: Marketing Campaign Attribution....312
Vertical and Horizontal Concatenation....316
Vertical....317
Horizontal....318
Diagonal....318
Align....319
Relaxed....322
Stacking....323
Appending....324
Extending....324
Takeaways....325
15. Reshaping....327
Wide Versus Long DataFrames....327
Pivot to a Wider DataFrame....330
Unpivot to a Longer DataFrame....335
Transposing....337
Exploding....339
Partition into Multiple DataFrames....342
Takeaways....345
Part V. Advance
16. Visualizing Data....349
NYC Bike Trips....351
Built-In Plotting with Altair....353
Introducing Altair....353
Methods in the Plot Namespaces....354
Plotting DataFrames....355
Too Large to Handle....357
Plotting Series....359
pandas-Like Plotting with hvPlot....363
Introducing hvPlot....363
A First Plot....364
Methods in the hvPlot Namespace....365
pandas as Backup....366
Manual Transformations....367
Changing the Plotting Backend....368
Plotting Points on a Map....369
Composing Plots....369
Adding Interactive Widgets....371
Publication-Quality Graphics with plotnine....372
Introducing plotnine....373
Plots for Exploration....373
Plots for Communication....377
Styling DataFrames With Great Tables....381
Takeaways....386
17. Extending Polars....387
User-Defined Functions in Python....387
Applying a Function to Elements....388
Applying a Function to a Series....390
Applying a Function to Groups....391
Applying a Function to an Expression....394
Applying a Function to a DataFrame or LazyFrame....395
Registering Your Own Namespace....396
Polars Plugins in Rust....397
Prerequisites....398
The Anatomy of a Plugin Project....398
The Plugin....398
Compiling the Plugin....401
Performance Benchmark....401
Register Arguments....402
Using a Rust Crate....405
Use Case: geo....405
Takeaways....416
18. Polars Internals....417
Polars’ Architecture....417
Arrow....419
Multithreaded Computations and SIMD Operations....421
The String Data Type in Memory....422
ChunkedArrays in Series....423
Query Optimization....424
LazyFrame Scan-Level Optimizations....425
Other Optimizations....427
Checking Your Expressions....429
meta Namespace Overview....429
meta Namespace Examples....430
Profiling Polars....432
Tests in Polars....434
Comparing DataFrames and Series....435
Common Antipatterns....437
Using Brackets for Column Selection....437
Misusing Collect....437
Using Python Code in your Polars Queries....438
Takeaways....439
Appendix: Accelerating Polars with the GPU....441
Index....461

Unlock the power of Polars, a Python package for transforming, analyzing, and visualizing data. In this hands-on guide, Jeroen Janssens and Thijs Nieuwdorp walk you through every feature of Polars, showing you how to use it for real-world tasks like data wrangling, exploratory data analysis, building pipelines, and more.

Whether you're a seasoned data professional or new to data science, you'll quickly master Polars' expressive API and its underlying concepts. You don't need to have experience with pandas, but if you do, this book will help you make a seamless transition. The many practical examples and real-world datasets are available on GitHub, so you can easily follow along.

• Process data from CSV, Parquet, spreadsheets, databases, and the cloud

• Get a solid understanding of Expressions, the building blocks of every query

• Handle complex data types, including text, time, and nested structures

• Use both eager and lazy APIs, and know when to use each

• Visualize your data with Altair, hvPlot, plotnine, and Great Tables

• Extend Polars with your own Python functions and Rust plugins

• Leverage GPU acceleration to boost performance even further