{"id":1577,"date":"2020-09-15T14:37:26","date_gmt":"2020-09-15T14:37:26","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/09\/15\/heres-what-you-need-to-look-for-in-a-model-server-to-build-ml-powered-services\/"},"modified":"2020-09-15T14:37:26","modified_gmt":"2020-09-15T14:37:26","slug":"heres-what-you-need-to-look-for-in-a-model-server-to-build-ml-powered-services","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/09\/15\/heres-what-you-need-to-look-for-in-a-model-server-to-build-ml-powered-services\/","title":{"rendered":"Here\u2019s what you need to look for in a model server to build ML-powered services"},"content":{"rendered":"
\n

By Ben Lorica<\/a> (helping organize #Raysummit) and Ion Stoica<\/a> (Berkeley, Anyscale)<\/b><\/p>\n

<\/p>\n

Machine learning is being embedded in applications that involve many data types and data sources. This means that software developers from different backgrounds need to work on projects that involve ML. In our\u00a0previous post<\/a>, we listed key features that machine learning platforms need to have in order to meet current and future workloads. We also described MLOps, a set of practices focused on productionizing the machine learning lifecycle.<\/p>\n

<\/p>\n

In this post, we focus on model servers, software at the heart of machine learning services that operate in real-time or offline. There are two common approaches used for serving machine learning models. The first approach embeds model evaluation in a web server (e.g., Flask) as an API service endpoint dedicated to a prediction service.<\/p>\n

The second approach offloads model evaluation to a separate service. This is an active area for startups, and there are a growing number of options that fall into this category. Offerings include services from\u00a0cloud providers<\/em>\u00a0(SageMaker<\/a>,\u00a0Azure<\/a>,\u00a0Google Cloud<\/a>),\u00a0open source projects for model serving<\/em>\u00a0(Ray Serve<\/a>, Seldon, TorchServe, TensorFlow Serving, etc.),\u00a0proprietary software<\/em>\u00a0(SAS, Datatron, ModelOp, etc.), and bespoke solutions usually written in some generic framework.<\/p>\n

While machine learning can be used for one-off projects, most developers seek to embed machine learning across their products and services. Model servers are important components of software infrastructure for productionizing machine learning, and as such, companies need to carefully evaluate their options. This post focuses on key features companies should look for in a model server.<\/p>\n

\u00a0<\/p>\n

Support for popular toolkits<\/h3>\n

\u00a0<\/p>\n

Your model server is probably separate from your model training system. Choose a model server that is able to use a trained model artifact produced using a number of popular tools. Developers and machine learning engineers build models using many different libraries, including ones for deep learning (PyTorch, TensorFlow), machine learning, and statistics (scikit-learn, XGBoost, SAS,\u00a0statsmodel<\/a>). Model builders also continue to use a variety of programming languages. While Python has emerged as the dominant language for machine learning, other languages like R, Java, Scala, Julia, SAS, still have many users as well. More recently, many companies have implemented data science workbenches like Databricks, Cloudera, Dataiku, Domino Data Lab, and others.<\/p>\n

\u00a0<\/p>\n

A GUI for model deployment and more<\/h3>\n

\u00a0<\/p>\n

Developers may use a command line interface, but enterprise users will want a graphical user interface that guides them through the process of deploying models and highlights the different stages of the machine learning lifecycle. As the deployment processes mature, they might migrate more to scripting and automation. Model servers with user interfaces include\u00a0Seldon Deploy<\/a>,\u00a0SAS Model Manager<\/a>,\u00a0Datatron<\/a>, and others that target enterprise users.<\/p>\n

\u00a0<\/p>\n

Easy to operate and deploy, but with high-performance and scalability<\/h3>\n

\u00a0<\/p>\n

As machine learning gets embedded in critical applications, companies will need low-latency model servers that can power large-scale prediction services. Companies like Facebook and Google have machine learning services that provide real-time responses\u00a0billions of times each day<\/a>. While these might be extreme cases, many companies also deploy applications like\u00a0recommendation and personalization systems<\/a>\u00a0that interact with many users on a daily basis. With the availability of open source software like Ray Serve, companies now have access to low-latency model servers that can scale to many machines.<\/p>\n

Most model servers use a microservice architecture and are accessible through a REST or gRPC API.\u00a0 This makes it easier to integrate machine learning (\u201crecommender\u201d) with other services (\u201cshopping cart\u201d). Depending on your setup, you may want a model server that lets you deploy models on the cloud, on-premise, or both. Your model server has to participate in infrastructure features like auto-scaling, resource management, and hardware provisioning.<\/p>\n

Some model servers added recent innovations that reduce complexity, boost performance, and provide flexible options for integrating with other services. With the introduction of a new Tensor data type, RedisAI supports\u00a0data locality<\/em>\u00a0\u2013 a feature that enables users to get and set Tensors from their favorite client and \u201crun their AI model where their data lives.\u201d Ray Serve\u00a0brings model evaluation logic closer to business logic\u00a0<\/a>by giving developers end-to-end control from the API endpoint to model evaluation, and back to the API endpoint. In addition, Ray Serve is easy to operate and is as easy to deploy as a simple web server.<\/p>\n

\u00a0<\/p>\n

Includes tools for testing, deployment, and rollouts<\/h3>\n

\u00a0<\/p>\n

Once a model is trained, it has to be reviewed and tested before it gets deployed. Seldon Deploy, Datatron, and other model servers have some interesting capabilities that let you test models with a single prediction or using a load test. To facilitate error identification and testing, these model servers also let you upload test data and visualize test predictions.<\/p>\n

Once your model has been reviewed and tested, your model server should give you the ability to safely promote and demote models. Other popular rollout patterns include:<\/p>\n