{"id":557,"date":"2020-08-21T13:40:39","date_gmt":"2020-08-21T13:40:39","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/08\/21\/data-science-meets-devops-mlops-with-jupyter-git-kubernetes\/"},"modified":"2020-08-21T13:40:39","modified_gmt":"2020-08-21T13:40:39","slug":"data-science-meets-devops-mlops-with-jupyter-git-kubernetes","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/08\/21\/data-science-meets-devops-mlops-with-jupyter-git-kubernetes\/","title":{"rendered":"Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes"},"content":{"rendered":"
\n

By Jeremy Lewi<\/a>, Software Engineer at Google & Hamel Husain<\/a>, <\/b><\/p>\n

\u00a0<\/p>\n

The Problem<\/h3>\n

\u00a0
Kubeflow<\/a>\u00a0is a fast-growing open source project that makes it easy to deploy and manage machine learning on Kubernetes.<\/p>\n

Due to Kubeflow\u2019s explosive popularity, we receive a large influx of GitHub issues that must be triaged and routed to the appropriate subject matter expert. The below chart illustrates the number of new issues opened for the past year:<\/p>\n

\n\"Figure\"
<\/p>\n

Figure 1:<\/strong>\u00a0Number of Kubeflow Issues<\/p>\n

<\/span>\n<\/div>\n

\u00a0<\/p>\n

To keep up with this influx, we started investing in a Github App called\u00a0Issue Label Bot<\/a>\u00a0that used machine learning to auto label issues. Our\u00a0first model<\/a>\u00a0was trained using a collection of popular public repositories on GitHub and only predicted generic labels. Subsequently, we started using\u00a0Google AutoML<\/a>\u00a0to train a Kubeflow specific model. The new model was able to predict Kubeflow specific labels with average precision of 72% and average recall of 50%. This significantly reduced the toil associated with issue management for Kubeflow maintainers. The table below contains evaluation metrics for Kubeflow specific labels on a holdout set. The\u00a0precision and recall<\/a>\u00a0below coincide with prediction thresholds that we calibrated to suit our needs.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Label<\/th>\nPrecision<\/th>\nRecall<\/th>\n<\/tr>\n<\/thead>\n
area-backend<\/td>\n0.6<\/td>\n0.4<\/td>\n<\/tr>\n
area-bootstrap<\/td>\n0.3<\/td>\n0.1<\/td>\n<\/tr>\n
area-centraldashboard<\/td>\n0.6<\/td>\n0.6<\/td>\n<\/tr>\n
area-components<\/td>\n0.5<\/td>\n0.3<\/td>\n<\/tr>\n
area-docs<\/td>\n0.8<\/td>\n0.7<\/td>\n<\/tr>\n
area-engprod<\/td>\n0.8<\/td>\n0.5<\/td>\n<\/tr>\n
area-front-end<\/td>\n0.7<\/td>\n0.5<\/td>\n<\/tr>\n
area-frontend<\/td>\n0.7<\/td>\n0.4<\/td>\n<\/tr>\n
area-inference<\/td>\n0.9<\/td>\n0.5<\/td>\n<\/tr>\n
area-jupyter<\/td>\n0.9<\/td>\n0.7<\/td>\n<\/tr>\n
area-katib<\/td>\n0.8<\/td>\n1.0<\/td>\n<\/tr>\n
area-kfctl<\/td>\n0.8<\/td>\n0.7<\/td>\n<\/tr>\n
area-kustomize<\/td>\n0.3<\/td>\n0.1<\/td>\n<\/tr>\n
area-operator<\/td>\n0.8<\/td>\n0.7<\/td>\n<\/tr>\n
area-pipelines<\/td>\n0.7<\/td>\n0.4<\/td>\n<\/tr>\n
area-samples<\/td>\n0.5<\/td>\n0.5<\/td>\n<\/tr>\n
area-sdk<\/td>\n0.7<\/td>\n0.4<\/td>\n<\/tr>\n
area-sdk-dsl<\/td>\n0.6<\/td>\n0.4<\/td>\n<\/tr>\n
area-sdk-dsl-compiler<\/td>\n0.6<\/td>\n0.4<\/td>\n<\/tr>\n
area-testing<\/td>\n0.7<\/td>\n0.7<\/td>\n<\/tr>\n
area-tfjob<\/td>\n0.4<\/td>\n0.4<\/td>\n<\/tr>\n
platform-aws<\/td>\n0.8<\/td>\n0.5<\/td>\n<\/tr>\n
platform-gcp<\/td>\n0.8<\/td>\n0.6<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 1:<\/span><\/strong>\u00a0Evaluation metrics for various Kubeflow labels.<\/p>\n

\n\u00a0<\/p>\n

Given the rate at which new issues are arriving, retraining our model periodically became a priority. We believe continuously retraining and deploying our model to leverage this new data is critical to maintaining the efficacy of our models.<\/p>\n

\u00a0<\/p>\n

Our Solution<\/h3>\n

\u00a0
Our CI\/CD solution is illustrated in\u00a0Figure 2<\/a>. We don\u2019t explicitly create a directed acyclic graph (DAG) to connect the steps in an ML workflow (e.g. preprocessing, training, validation, deployment, etc\u2026). Rather, we use a set of independent controllers. Each controller declaratively describes the desired state of the world and takes actions necessary to make the actual state of the world match. This independence makes it easy for us to use whatever tools make the most sense for each step. More specifically we use<\/p>\n