Source code processing with Python

Summary

[EuroPython 2011] Kay Schluehr - 24 June 2011 in "Track Lasagne"

"/>

Description

Folklore says that having a problem and trying to solve it with regular expressions gives you two problems. However not applying regular expressions to advanced textual search'n replace doesn't solve your problem either. One step above you have large portions of recursively structured text aka "source code" and using context free grammars and tools supporting them gives you two problems but not using them also doesn't solve your original problem. Maybe you get uneasy at that point because what I say implies parsers and computing science and what not and you still wake up in the night believing that you have to learn automata theory but you are lucky it was just a nightmare. Otherwise you are laughing about the little diatribe against regexps and use them without much deliberation, verifying your SQL input, mining source code and do all the other things they are not made for.

In my talk I'm addressing daily use of grammars outside of the scope of compiler implementation or natural language processing. My talk covers:

  • Search & Replace using grammars

  • CodeTemplates for source code transformation

  • Generative grammars for expression generation

I'm touching this from the lightweight, "pythonic" angle and you might wonder why not everyone uses those techniques already for decades in their daily work. I can't answer this, I wonder about this too.