Description
Testing ML Systems: More than just accuracy - PyCon Italia 2022
Zillow lost millions because they didn’t have proper MLOps or testing processes. They relied on the training metrics, like accuracy, but Software Quality goes beyond that. I will cover different testing types, techniques, and concepts that you can apply to make high-quality ML Systems. As ML/AI systems are becoming more prevalent, the need for setting quality standards and testing practices has become crucial. Testing these models goes beyond validation metrics like accuracy, precision, and recall.
Instead, quality attributes like model Behaviors, Usability, and Fairness need to be tested and measured using exploratory and automated strategies.
In this talk, we’ll cover some of the risks and biases that can happen throughout the MLOps pipeline, demonstrate a few techniques to test a model’s behaviors and fairness, and apply them against some real-world scenarios and state-of-the-art models.
By the end, you will have new ideas and techniques that you can use to test your own ML/AI systems and approach these quality attributes from a customer’s perspective.
Speaker: Carlos Kidman