Contribute Media
A thank you to everyone who has made this possible: Read More

Python (and Fortran and C) as used in large-scale number crunching and scientific programming

Description

Python is an incredibly powerful and versatile language, but it is not practical for lots of deeply nested loops because it's not compiled, and a lot of the big scientific libraries only have C interfaces. But, if you bring Fortran (via f2py) and C (Cython) into the mix then you can use the best features of all three languages and shrink your development/maintenance time and resulting code size by almost an order of magnitude.

My satellite has a lot of legacy code in a Perl/C/Fortran mixture that is incredibly difficult to read and maintain, but our new software is in a Python/C/Fortran mixture (with Python being the bulk of the code) that is a fraction of the size of the old code, much much easier to develop and maintain, yet it has the speed and stability to be used in our production system.

I use Python for the upper and middle layers, Fortran for the few places where I need to do lots of looping that cannot be transformed into whole-array operations, and then C to interface with the various 3rd party libraries. I can wrap Fortran and C using f2py and Cython respectively, call them directly from the Python, and pass numpy arrays back and forth seamlessly.

I also have layers of scripts: a Python code to run a single orbit's worth of data through a single stage of the processing pipeline, a second Python script to run multiple orbits through a single stage, and then a third script that runs the entire processing chain through many orbits, thereby using Python's numerous scripting capabilities.

By using all three languages as described above, I truly get the best of all worlds: dramatically reduced development time, a code base that is a fraction of the size of the old legacy system, yet all the speed and stability I need to process huge amounts of data.

Details

Improve this page