Performance Optimization

Python is not the fastest language, but most programs are never actually bottlenecked by raw CPU speed — they're bottlenecked by network calls, database queries, disk I/O, or a handful of specific hot spots in the code. The single most important performance skill isn't knowing tricks; it's knowing how to find out where time is actually being spent before you touch anything.

Warning

Premature optimization is a real trap. Rewriting readable code into a clever, "faster" version that you merely suspect is faster — without measuring — usually wastes time, adds bugs, and hurts readability for no measurable benefit. The rule of thumb: measure, don't guess.

Micro-benchmarks with timeit

The timeit module runs a snippet of code many times and reports how long it took, avoiding the usual pitfalls of hand-rolled timing (startup cost, other processes stealing CPU time, garbage collection noise).

Python

import timeit

# Compare building a list with a loop vs. a list comprehension
loop_time = timeit.timeit(
    'result = []\nfor i in range(1000):\n    result.append(i * i)',
    number=10000,
)

comp_time = timeit.timeit(
    'result = [i * i for i in range(1000)]',
    number=10000,
)

print(f"loop:            {loop_time:.4f}s")
print(f"comprehension:   {comp_time:.4f}s")

From the command line, timeit is even more convenient for quick comparisons:

Bash

python -m timeit "'-'.join(str(i) for i in range(1000))"
python -m timeit "s = ''
for i in range(1000): s += str(i)"

Note

`timeit` is for small, isolated snippets — comparing two ways of writing the same few lines of code. For understanding where time goes across an entire program, reach for `cProfile` instead.

Full profiling with cProfile

cProfile instruments an entire run of your program (or a specific function) and reports how many times each function was called and how much time was spent in it, which is exactly what you need to find real bottlenecks instead of guessing.

Python

import cProfile
import pstats


def slow_function():
    total = 0
    for i in range(1_000_000):
        total += i ** 2
    return total


def main():
    slow_function()
    slow_function()


cProfile.run('main()', 'profile_output')

stats = pstats.Stats('profile_output')
stats.sort_stats('cumulative').print_stats(5)

You can also run it directly from the command line against a script, without editing any code:

Bash

python -m cProfile -s cumulative my_script.py

         6 function calls in 0.312 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.312    0.156    0.312    0.156 script.py:5(slow_function)
        1    0.000    0.000    0.312    0.312 script.py:11(main)
        ...

Read the report top-down: tottime is time spent in that function alone, cumtime includes time spent in everything it calls. The functions with the highest cumtime are where optimization effort will actually pay off.

Common, high-value performance wins

Once profiling points you at an actual hot spot, these are the fixes that tend to matter most in ordinary Python code — well before reaching for anything exotic.

Situation	Slower approach	Faster approach
Building a transformed list	Manual `for` loop with `.append()`	List/dict/set comprehension or generator expression
Concatenating many strings	Repeated `s += chunk` in a loop	Collect pieces in a list, then `''.join(pieces)`
Checking "is X in this collection?"	Membership test on a `list` (`O(n)`)	Membership test on a `set`/`dict` (`O(1)`)
Calling an expensive pure function repeatedly with the same input	Recomputing every time	`functools.lru_cache` to memoize results
Using built-in operations	Hand-written loops for sum/min/max/sort	`sum()`, `min()`, `max()`, `sorted()` — implemented in optimized C

Python

# Slow: quadratic-ish string building
def build_report_slow(rows):
    text = ""
    for row in rows:
        text += str(row) + "\n"
    return text


# Fast: linear, using join()
def build_report_fast(rows):
    return "\n".join(str(row) for row in rows)

Python

# Slow: O(n) membership check against a list, repeated for every lookup
blocked_users = ["alice", "bob", "carol", "dave"]  # imagine thousands of entries
if username in blocked_users:
    deny_access()

# Fast: O(1) membership check against a set
blocked_users = {"alice", "bob", "carol", "dave"}
if username in blocked_users:
    deny_access()

Python

from functools import lru_cache


@lru_cache(maxsize=None)
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)


# Without caching, fibonacci(35) makes millions of redundant calls.
# With lru_cache, each unique input is computed exactly once.
print(fibonacci(35))

Tip

`lru_cache` only helps for pure functions — same input always produces the same output, with no side effects. Caching a function that depends on mutable state or the current time will produce stale or wrong results.

When built-in tricks aren't enough

For a small number of genuinely CPU-bound bottlenecks — tight numerical loops that profiling shows dominate total runtime — pure Python optimizations eventually hit a wall. At that point:

Reach for optimized libraries first — NumPy, pandas, or existing C-backed packages often already solve the problem without you writing any C.
Cython lets you compile Python-like code (with optional static types) down to C for a hot function, keeping most of your codebase in plain Python.
C extensions (or tools like cffi/ctypes) let you write the hottest inner loop in C and call it from Python.
PyPy is an alternative Python implementation with a JIT compiler that can dramatically speed up long-running, CPU-heavy pure-Python code with no source changes — worth trying if your dependencies are compatible with it.

Note

These are last resorts, not starting points. The vast majority of real-world slowness in Python applications is fixed by better algorithms, better data structures, or reducing I/O — not by dropping into C.

Optimize in this order: measure with timeit/cProfile to find the actual bottleneck, apply the standard Python-level fixes (comprehensions, join(), sets/dicts, lru_cache), and only reach for Cython, C extensions, or PyPy when profiling proves that pure Python genuinely can't go fast enough.