DataHour: Predictive Analytics - Performance Estimation without the Target Data

This presentation will explore advancements in Performance Estimation for classification models when ground truth data is not readily available. Once your model is deployed to production, ensuring its optimal performance becomes crucial. However, this task often presents challenges, particularly when predicting events far into the future or automating certain processes.

We will introduce three key algorithms: Confidence-Based Performance Estimation (CBPE), Importance Weighting (IW), and Multi Calibrated Confidence-Based Performance Estimation (M-CBPE). Our analysis demonstrates that all three algorithms offer improved performance estimation compared to traditional test set evaluations.

During the session, we will delve into the operational mechanisms of these algorithms, elucidate their underlying intuition, and discuss their respective strengths and limitations. Additionally, we will illustrate typical scenarios where Performance Estimation plays a pivotal role and elucidate its application in ML Monitoring and Root Cause Analysis.

Wojtek Kuberski is an AI professional and entrepreneur with a master's in AI from KU Leuven. He founded Prophecy Labs, a consultancy specializing in machine learning, before assuming his current role as a co-founder and CTO of NannyML. NannyML is an OSS for ML monitoring and silent ML failure detection. At NannyML, he leads the research and product teams, contributing to novel algorithms in model monitoring.