Description
Let’s take a closer look at generators and why you may not be getting your expected improvements. Sometimes the memory benefit is negated by necessity; sometimes it's by mistake. "Meta generators" can solve this problem, allowing a much wider range of safe, fast operations on very large generators.
Through this talk, I want to highlight the dangers of trying to write very large or theoretically infinite Python generators to lists (obvious), and the ease with which it can be done accidentally (less obvious).
The talk will start with an example of when one might want to use some very large generators to solve a problem. I will discuss some common operations that we use to combine generators (chain, product, zip, etc.), and then go into detail about why some of these are "safe" to use on very large generators (chain, zip), while others are very much not (product).
The second part of the talk, which will explain our solution to this problem – "meta-generators" – also doubles as a recommendation for a future language feature.