ML monitoring with Evidently. A tutorial from CS 329S: Machine Learning Systems Design.

00:00 Introduction to the Tutorial
00:30 Speaker Introduction
01:05 What ML monitoring setup depends on
02:40 How to design ML monitoring
03:58 Toy example: bike demand monitoring
05:08 Code example starts
06:02 Dataset preparation
06:29 Model training
07:45 Model validation. Regression performance dashboard.
13:18 Production model training. Shorter version of the report.
14:55 Week 1 of production use.
17:30 Week 2 of production use. Choice of widgets.
19:08 Week 3 of production use. Model quality drop.
20:10 Quality drop debugging. Data drift dashboard.
24:43 Dashboard customization. Statistical tests, bins, tabs.
31:33 How to automate batch monitoring. MLflow example.
36:43 Q: How can I share reports with my coworker?
37:43 Q: What other features are most requested?
38:44 Q: Are standard deviations useful only for normal distributions?

Code example: https://github.com/evidentlyai/evidently/blob/main/examples/data_stories/bicycle_demand_monitoring_setup.ipynb

Information about the course: https://stanford-cs329s.github.io/syllabus.html

monitoringEvidently.tutorial