Description
You've heard it many times: the GIL is a problem for using all your CPU cores in one program. Among the generally accepted solutions there's multiprocessing, a way to orchestrate a group of worker processes to spread CPU load over many cores. This solves the problem for many use cases but if you have a lot of data to pass around there and back again, it's much less efficient.
In this short talk we'll go through two examples of data processing with Python 3.11 and how asyncio with shared memory helps speed things up. To cover all bases, one example will run on macOS, the other on Windows Subsystem for Linux. You'll see how the built-in building blocks of Python allow to compose scalable systems. Our focus is on the base programming language. We won't be reimplementing data pipelines or covering any MLops best practices.