At my previous job, we used joblib + a handbuilt file format to persist ML model blobs across versions.
The file format was a zipfile with:
- a joblib dump of the python class wrapping the model and its data processing pipeline code
- a small csv (~100 rows) of test X data, test Y data and the expected outputs of the model on that data
- a little json with important library versions (pandas, numpy, etc.) so that the process loading the code could check it ran the same as the one that produced the model blob
Then the loader would try to load the file and verify it, and if failed would keep the existing model in memory and raise an alert to slack, so engineers could fix it and reupload a new model without downtime.
The whole thing was inspired by the talk "Alex Gaynor: Pickles are for Delis, not Software - PyCon 2014" [1]
At my current job we don't need such infrastructure because lambda/kubeflow gateways take care of it for us.
The file format was a zipfile with:
- a joblib dump of the python class wrapping the model and its data processing pipeline code
- a small csv (~100 rows) of test X data, test Y data and the expected outputs of the model on that data
- a little json with important library versions (pandas, numpy, etc.) so that the process loading the code could check it ran the same as the one that produced the model blob
Then the loader would try to load the file and verify it, and if failed would keep the existing model in memory and raise an alert to slack, so engineers could fix it and reupload a new model without downtime.
The whole thing was inspired by the talk "Alex Gaynor: Pickles are for Delis, not Software - PyCon 2014" [1]
At my current job we don't need such infrastructure because lambda/kubeflow gateways take care of it for us.
[1] https://www.youtube.com/watch?v=7KnfGDajDQw