Summary
Working with Python does not usually involve debugging memory problems: the interpreter takes care of allocating and releasing system memory and you get to enjoy working on real world issues. But what if you encounter such problems? What if your program never releases memory? How do you debug it?
This talk describes some of the lesser known properties of cPython memory allocator and some ways to debug memory-related problems, all this based on real events.
Description
Working with Python does not usually involve debugging memory problems: the interpreter takes care of allocating and releasing system memory and you get to enjoy working on real problems. But what if you encounter such problems? What if your program never releases memory? How do you debug it?
I will tell a story of one programmer discovering such problems. The talk will take listeners on a journey of issues they can encounter, tools they can use to debug the problems and possible solutions to seek out. There will also be a brief mention of general memory management principles.
cPython uses a combination of its own allocator, malloc, and mmap pools to manage memory of Python programs. It usually is smart enough, but there are some darker corners that are not well known by an average Joe Programmer (read: me).
There are tools that can help debug memory problems, but those are also relatively unknown, and tend to have documentation that one might find lacking. I will describe one such tool, called guppy, which I have found particulary helpful.