Contribute Media
A thank you to everyone who has made this possible: Read More

Python (and Fortran and C) as used in large-scale number crunching and scientific programming


Python is an incredibly powerful and versatile language, but it is not practical for lots of deeply nested loops because it's not compiled, and a lot of the big scientific libraries only have C interfaces. But, if you bring Fortran (via f2py) and C (Cython) into the mix then you can use the best features of all three languages and shrink your development/maintenance time and resulting code size by almost an order of magnitude.

My satellite has a lot of legacy code in a Perl/C/Fortran mixture that is incredibly difficult to read and maintain, but our new software is in a Python/C/Fortran mixture (with Python being the bulk of the code) that is a fraction of the size of the old code, much much easier to develop and maintain, yet it has the speed and stability to be used in our production system.

I use Python for the upper and middle layers, Fortran for the few places where I need to do lots of looping that cannot be transformed into whole-array operations, and then C to interface with the various 3rd party libraries. I can wrap Fortran and C using f2py and Cython respectively, call them directly from the Python, and pass numpy arrays back and forth seamlessly.

I also have layers of scripts: a Python code to run a single orbit's worth of data through a single stage of the processing pipeline, a second Python script to run multiple orbits through a single stage, and then a third script that runs the entire processing chain through many orbits, thereby using Python's numerous scripting capabilities.

By using all three languages as described above, I truly get the best of all worlds: dramatically reduced development time, a code base that is a fraction of the size of the old legacy system, yet all the speed and stability I need to process huge amounts of data.


Improve this page