Micha Gorelick, Ian Ozsvald.
High Performance Python: practical performant programming for humans: 2nd edn.
O'Reilly. 2020
rating : 2.5 : great stuff
review : 10 January 2026
How do you take advantage of multicore architectures or clusters? Or build a system that scales up and down without losing reliability? Experienced Python programmers will learn concrete solutions to many issues, along with war stories from companies that use high-performance Python for social media analytics, productionized machine learning, and more.
• Get a better grasp of NumPy, Cython, and profilers
• Learn how Python abstracts the underlying computer architecture
• Use profiling to find bottlenecks in CPU time and memory usage
• Write efficient programs by choosing appropriate data structures
• Speed up matrix and vector computations
• Use tools to compile Python down to machine code
• Manage multiple I/O and computational operations concurrently
• Convert multiprocessing code to run on local or remote clusters
• Deploy code faster using tools like Docker
This is an excellent introduction to many ways to improve the efficiency (speed, memory usage, etc) of Python code. It assume you can already program in Python, but need more advanced techniques to get better performance.
It starts with a chapter on profiling, and the stern warning to profile before optimising, as your intuition of where the problems are is most likely to be wrong. Then there are some fairly gentle chapters on core Python: tradeoffs between lists and tuples, using sets and dictionaries (sets are just dictionaries with keys but no values), and list versus generator comprehensions. Then we learn more about numpy and pandas, and why numpy arrays are more efficient than plain lists.
Then we get into the rather more advanced topics: compiling, multicores, GPUs, clusters, and more. Some of the more advanced techniques, such as using an in-house cluster, are not single-person tasks. These are mostly about increased speed; there is also a chapter on reducing RAM use, including some interesting algorithms I have heard of but have not used in anger (such as tries) and some I haven’t come across before, such as probabilistic data structures that trade off space for accuracy, and how to count (probabilistically!) to 1076 using one byte!
There is a lot of great material in here, from simple changes that will have big effects, to highly sophisticated business-level advice. Some I will implement immediately (after profiling, of course), and some I will come back to later if I need it.