Description
Ingest and analyze streams of data generated by sensors, social media an other sources.
Sensors, social media, news feeds, webcams and other sources generate streams of data which are analyzed to control actuators, generate alerts, and feed displays. These applications process streams on onboard computers, such as the Raspberry Pi, connected directly to sensors, and send summarized information to the cloud for further processing. These applications have two characteristics: (1) Concurrency: The applications are concurrent using multiple threads to connect to sensors and actuators, shared memory across multiple processes on multicore machines and message passing for distributed systems spanning multiple computers. (2) Data Analysis: The applications use programs from a variety of libraries including those for signal processing, machine learning and natural language processing.
Developers of streaming applications can use open-source software to deal with both characteristics. Concurrency: multiprocessing.Array can be used to construct shared-memory multiprocessing Python programs in multicore computers, and frameworks such as APMQ and Kafka can be used to build distributed applications. Data Analysis: A vast collection of open-source Python libraries can be used to analyze data in streams. Developers of streaming applications encounter an impedance mismatch between the software libraries that address these two characteristics. The next paragraph describes the mismatch and how IoTPy addresses it.
Programs in most software libraries apply a function to data, get results, and terminate execution. By contrast, streaming applications are perpetual processes that analyze endless streams of data. IoTPy helps developers: (1) build non-terminating streaming applications by harnessing conventional terminating programs from Python’s huge base of libraries and (2) create multithreaded, multicore and distributed Python applications by simply connecting streams to each other.