James Coates

Computer Science student. Sydney, Australia.

← Back to blog

Compiling Python 3.11 from Source on Raspberry Pi: Thermal Limits, OOM Crashes, and What Actually Works

· Technology

Compiling Python 3.11 from Source on Raspberry Pi: Thermal Limits, OOM Crashes, and What Actually Works

The Raspberry Pi is capable hardware for embedded projects, but it is constrained hardware for compilation. When I needed Python 3.11 on the Raspberry Pi running my DoorPi access control system — a Debian Bullseye installation on an armv7l kernel — the package repositories weren't offering it. That meant compiling from source.

What followed was an instructive exercise in understanding why compiler flags exist, how thermal management affects long-running processes, and why make altinstall is not optional when you care about system stability.

Why Compile from Source?

Debian Bullseye's package repositories include Python 3.9. For the DoorPi project's speech recognition libraries and some of their dependencies, 3.11 was required — specifically for performance improvements in the asyncio implementation and compatibility with updated library versions.

The options were:

  1. Use an older library version compatible with Python 3.9
  2. Add a third-party repository (deadsnakes PPA for Debian, with associated trust implications)
  3. Compile Python 3.11 directly from the CPython source

Option 3 is the most controlled. You know exactly what you're installing, you choose your compilation flags, and you maintain full ownership of the resulting installation. The cost is time and, as it turned out, careful management of the compilation process itself.

The Hardware Reality

The Raspberry Pi running DoorPi is an older model with armv7l architecture — 32-bit ARM, limited RAM (1 GB), and a thermal ceiling that becomes relevant during sustained CPU-intensive operations like compilation.

Memory: Python compilation is memory-hungry. The C compiler (GCC), linker, and various build tools all run simultaneously during a parallel build. 1 GB of RAM is workable but tight.

Thermal: The Pi has no active cooling in its deployment environment (mounted at a door, running headless). During compilation, CPU temperatures climbed to 77 °C+ — above the thermal throttling threshold, which caused compilation processes to slow and in some cases produce compiler errors from incomplete compilation units.

Compilation time: Even with parallel jobs, expect several hours for a complete Python 3.11 build on this hardware. Plan accordingly.

Getting the Source

wget https://www.python.org/ftp/python/3.11.x/Python-3.11.x.tgz
tar -xzf Python-3.11.x.tgz
cd Python-3.11.x

Substitute the specific 3.11 point release version as needed.

Dependencies

Before configuring, install build dependencies:

sudo apt update
sudo apt install -y build-essential libssl-dev libffi-dev zlib1g-dev \
    libbz2-dev libreadline-dev libsqlite3-dev libncurses5-dev \
    libgdbm-dev libnss3-dev liblzma-dev uuid-dev tk-dev

Missing dependencies won't necessarily fail the build outright — Python will compile without the modules that depend on them, and you'll discover the gaps only when you try to import them. Installing all dependencies upfront avoids this.

The Configuration Step

This is where the critical decisions happen. The standard configuration command:

./configure --enable-optimisations

The --enable-optimisations flag enables Profile Guided Optimisation (PGO) and Link-Time Optimisation (LTO). On modern desktop hardware with cooling, this produces a noticeably faster Python binary. On a thermally constrained, memory-limited Pi, it causes the build to fail.

Why PGO fails here:

PGO requires the compiler to first build an instrumented version of Python, run a training workload to collect profiling data (using gcov under GCC), and then rebuild using that data. This doubles the compilation time, significantly increases peak memory usage during the rebuild phase, and generates large .gcda profiling data files on disk.

The combination of extended compilation time (more heat accumulation), doubled memory pressure, and the overhead of GCC's gcov instrumentation pushes the Pi beyond its limits. The result is either OOM kills that terminate GCC mid-compilation, or thermal throttling so severe that the build effectively stalls.

The correct configuration for constrained hardware:

./configure --prefix=/usr/local --without-ensurepip

Stripping --enable-optimisations removes PGO/LTO entirely. The resulting Python binary is somewhat slower than an optimised build — benchmarks typically show 10–20% slower execution — but it is a binary that actually compiles and runs correctly on this hardware.

If you want some optimisation without the full PGO overhead:

./configure --prefix=/usr/local --without-ensurepip CFLAGS="-O2"

-O2 enables standard compiler optimisations without profiling instrumentation.

Controlling Parallel Jobs

The default make command uses all available cores. On the Pi, this maximises thermal load and memory pressure simultaneously:

# Do not do this on constrained hardware:
make -j$(nproc)

# Use a limited job count instead:
make -j2

Two parallel jobs is a reasonable balance — faster than single-threaded, but with enough headroom to avoid thermal and memory spikes. On the older Pi models, I found -j2 reliably completed builds where -j4 would fail mid-way.

Thermal Management During Compilation

Monitor temperatures during the build:

watch -n 5 cat /sys/class/thermal/thermal_zone0/temp

The output is in millidegrees Celsius — divide by 1000. If you see values above 80000 (80 °C), the CPU is throttling and the build is at risk. Options:

For long unattended builds, a simple Bash script that monitors temperature and pauses make if it exceeds a threshold is worth the setup time.

Installing Without Overwriting the System Python

This is the most important step. Debian Bullseye's system tools depend on the system Python 3.9. Replacing it will break apt, system scripts, and various other utilities that the OS depends on.

make install installs to the prefix and creates symlinks at the standard python3 and pip3 locations — potentially overwriting system Python symlinks.

make altinstall installs the binaries as python3.11 and pip3.11 without touching python3 or pip3 symlinks:

sudo make altinstall

This is not optional. A system with a broken Python is very difficult to repair without a fresh install, and on embedded hardware that means re-imaging the SD card and rebuilding your entire configuration from scratch.

Verify the installation:

python3.11 --version
# Python 3.11.x
which python3.11
# /usr/local/bin/python3.11
which python3
# /usr/bin/python3  ← still pointing to system Python 3.9

Virtual Environments for Project Dependencies

With Python 3.11 installed alongside the system Python, use virtual environments for all project-specific packages:

python3.11 -m venv /home/pi/doorpi-venv
source /home/pi/doorpi-venv/bin/activate
pip install SpeechRecognition RPi.GPIO pigpio

This isolates DoorPi's dependencies from both the system Python and any other projects. Dependency conflicts between projects become impossible when each has its own environment.

The virtual environment activation can be added to the tmux session startup or the DoorPi service script, ensuring the correct Python and packages are used regardless of how the application is started.

Persistent Operation with tmux

DoorPi runs headlessly. The speech recognition and GPIO control scripts need to persist after SSH sessions disconnect. tmux provides this:

tmux new-session -d -s doorpi 'source /home/pi/doorpi-venv/bin/activate && python3.11 /home/pi/doorpi/main.py'

This creates a detached tmux session named doorpi that survives SSH disconnection. The session can be reattached for debugging:

tmux attach -t doorpi

For automatic startup on boot, a cron @reboot job handles it:

crontab -e
# Add:
@reboot sleep 10 && tmux new-session -d -s doorpi 'source /home/pi/doorpi-venv/bin/activate && python3.11 /home/pi/doorpi/main.py'

The sleep 10 gives the system time to complete boot before the session starts — without it, the tmux session may start before the GPIO interface is ready.

What I'd Do Differently

Use a Pi with more RAM for compilation. A Pi 4 with 2 GB or 4 GB would handle this build substantially more easily. The 1 GB constraint was the primary source of difficulty.

Build on a more capable machine and cross-compile. Cross-compilation for armv7l on a modern x86 machine would take minutes rather than hours, and produce an identical binary. The resulting compiled Python can then be transferred to the Pi and installed directly.

Plan for a full day. The build process, including false starts and temperature management, took the better part of a day the first time. With the correct flags documented, subsequent builds are faster — but still not fast.

The Result

A working Python 3.11 installation on Debian Bullseye, coexisting safely with the system Python, with DoorPi's dependencies isolated in a virtual environment and the application running persistently under tmux.

The compilation challenges were genuinely instructive. Understanding why PGO fails on constrained hardware — the doubled compilation pass, the gcov instrumentation overhead, the memory and thermal implications — is the kind of knowledge that only comes from having the build fail and working out exactly why.

The Python documentation and GCC manual are thorough. The specific combination of constraints on embedded hardware is less documented. Hopefully this fills part of that gap.

← Back to blog