Description
In the past 8 years we've been collecting, scraping and processing government data - overcoming technical, legal and other hurdles - to create Israel's richest database of fiscal data (and growing!). In this talk I will talk about how we're doing this (we - at the Public Knowledge Workshop), and present the unique software frameworks and tools we've built to accomplish this goal. In particular, I will talk about the data-package pipelines framework, which combines a suite of standards and tools for 'packaging' data as well as a versatile engine for processing data streams.