Machine Learning Lifecycle: Deployment

 

Introduction:

This week, I signed up for a course called Machine Learning Engineering for Production (MLOps) by DeepLearning.ai. The first part of the course teaches about the different stages of a machine learning project and how to handle the challenges that come up along the way.

The course explains that MLOps combines machine learning with software development. It encourages students to explore what happens after they've trained a model in a Jupyter Notebook.

In this article and future ones, I plan to share the lessons I've learned. Feel free to join the discussion and leave your comments and suggestions after you've finished reading.

The Machine Learning Project Lifecycle:

The machine learning project goes through different stages, kind of like a circle, instead of just going in a straight line. It's not just about doing one thing after another – we often have to go back and make things better at different points in the process, just like when we train and test an AI model.

We can define the different stages of an AI project as follows:

• Scoping: During scoping, you're figuring out what you want to achieve with the project. You decide what factors (X) you'll look at and what you're trying to predict (y).

• Data: Before starting your project, you need to figure out what you want to achieve. Then, you have to gather the data you need to train and test your model. You can find datasets that are already labeled and organized, either for free or for a fee. Or, you might have to collect and organize your own data.

• Modeling: In this stage, you train and adjust your model's settings, while also checking if your data is suitable for your project. Andrew Ng, who started DeepLearning.ai, talks about how academia and product teams approach modeling differently. Researchers often stick to one dataset and tweak various models and settings. Meanwhile, product teams might stick to one model but focus on adjusting settings and improving the data. There's no universal approach. After training, you assess how well your model performs by analyzing errors. Sometimes, you need to go back to your data to see if it's good enough for your project.

• Deployment: After you've finished training, testing, and evaluating your model, it's time to put it into action. Even though this step comes last in the process, it might feel like you're only halfway there. Deploying the model means making sure it works smoothly in real-life situations. As a Machine Learning Engineer, you'll need to keep an eye on it to ensure everything runs as planned. If any issues crop up or updates are needed, you might have to go back to tweaking the model or updating the data.

We can use speech recognition as an example to show the different phases of its development process.

First, we outline the project, which in this example is a mobile app for voice search. During this step, we also determine the important measures of success and figure out how much time and resources we'll need to create the app.

When working with data, different projects may need different measures to check how good the data is for building models. For example, if you're working with speech data, you might need to decide how much quiet time should come before and after someone speaks. You also need to check if you should make the volume consistent across different audio files and if the files are of good quality. For speech recognition, many datasets come with both speech and text, so you should check if the labeling styles are the same in all the text.

During modeling, you'll use algorithms or neural networks to see how well they perform. You'll also adjust settings to make the model predict better. At this stage, you might need to check and update the data too.

You'll set up the model on a mobile app. This app will listen to what users say through their phone's microphone. Then, it will send their words to a prediction server using a speech API. The server will analyze what the users said and send back both the text of their speech and the search results.

Once the model is up and running, your main job is to keep an eye on how well it's doing. Sometimes, problems pop up, like if someone speaks with an accent the model didn't learn about before. Then, it might have trouble understanding them. Your job is to go back to the database and make sure the model learns about that accent, so it can handle it better next time.

Deployment:

Key Challenges:

While deployment is usually the final step in the process, Andrew Ng advises starting by identifying any potential problems or challenges you might encounter when deploying the model. These could include issues with data and problems related to software engineering.

After you've put your system into action, it's important to keep an eye on how the data evolves over time. Changes can happen slowly, like when new words and phrases become popular in speech, or they can happen suddenly, like when the COVID-19 pandemic caused people to change their shopping habits. For example, some people who never shopped online before started doing so, which could trigger fraud detection systems to flag these transactions as suspicious.

Overall, we can define the data issues into two categories:

• Data Drift:Data Drift is when the type of data the model gets changes over time, but the model doesn't know about it. For example, if there's suddenly a lot more searches for a celebrity or politician.

• Concept Drift: Concept Drift happens when the characteristics of what we're studying change over time. For instance, someone who never shopped online before might start doing so after the COVID-19 pandemic.

Software engineering problems involve determining the capabilities and requirements of the system. First, you need to decide if you want a system that gives quick responses in real-time, like within milliseconds or a few seconds, or if you prefer a batch system, which processes a lot of data at once, typically overnight or during off-peak times. Each type of system has its own needs and requirements for working effectively.

You need to choose where the data processing and decision-making happen. Cloud systems run on faraway servers in big buildings called data centers. They have lots of power and tools for processing data. Edge/Browser systems work on your own device or in your web browser. This might help keep your information private, and it can still work even if your internet connection isn't great.

It's important to consider your computer's power — like GPU, CPU, and memory. Basically, make sure you can handle what you're working on. Also, keep the user's data safe and private.

Deployment Patterns:

When putting a model into use, it's smart to do it step by step. First, we give the model a little bit of work and keep an eye on how well it does. If there's a problem, we can easily go back to using the old model without causing too much trouble.

For instance, consider a system used in a factory to find cracks on cellphone screens. Here are the various ways and steps to put it into action:

• Shadow Mode Deployment: In this scenario, we have an AI system that works alongside humans to detect cracks in screens. However, the AI's findings are not yet used to make decisions. Instead, the goal is to test the new model using real-world data without impacting the production system.

• Canary Deployment: We start by sending a small amount of traffic to the new model and keep an eye on its results. This way, we lower the risk by slowly introducing the model and switching back if there are any problems. At this point, the model acts more like a helper to people.

Blue/Green Deployment: This method involves having an old model (blue) and a new model (green). We switch the traffic from the old model to the new one once it's tested and good to go. If there are any problems with the new model, we can easily switch back to the old one.

Degrees of Automation:

When thinking about different projects and their goals, it's important to think about how much of the work can be done automatically by Machine Learning systems. This depends on the task and the data involved.

In systems where only humans are involved, there's no AI input or output. Humans handle the entire task process. In shadow mode systems, an AI model copies the human task, but its outputs don't affect decision-making. This setup helps spot problems early.

In systems with AI assistance and partial automation, humans and AI interact more. AI assistants help humans by pointing out important areas, like cracks on cellphone screens. With partial automation, the model helps make decisions, and humans step in only when the model isn't sure. Full automation means AI makes all decisions without humans.

These deployment apps vary widely, from those where people make all the decisions to those where the AI model does everything. Different apps need different levels of automation. For a big mobile search app that understands speech, it's best to let the AI do everything because there's so much happening. But in healthcare, it's important to have humans involved, too. That's because if the AI makes a mistake, it could seriously affect someone's life.

Deployment is Iterative:

As I mentioned before, deploying the model isn't the final step. After it's deployed, you need to keep an eye on the system for any software or data problems.

When dealing with software, you keep an eye on things like memory usage, how long it takes to compute, delays, and how much strain it puts on servers. For data, you watch both what goes in and what comes out. Take speech recognition, for example. You need to keep track of things like how long people talk on average and how loud they speak. If you notice any problems, like people having to redo searches a lot, it could mean the model isn't working as well as it should.

To keep your system running smoothly, it's important to establish specific benchmarks and measurements. This helps determine when adjustments are needed or when the model should be retrained. Knowing how quickly your data changes based on the task at hand is crucial. For instance, think about face recognition: faces generally stay the same over time, so data changes slowly. However, business data, like information about factory products, may change much more rapidly.

Conclusions:

In this article, we've explore one important part of the Machine Learning project journey: deployment. But deploying the model isn't the end of the road for an ML project. It brings its own set of challenges, from software issues to dealing with different kinds of data.

It's important to recognize how changes in culture and the environment can affect the model's performance over time. Being prepared to make quick adjustments is key to preventing any negative impacts on the final product.

Another thing to consider is how much of the project can be automated. While AI systems can handle some tasks on their own, human input remains vital in many industries and projects.