Skip to main content
Hello and welcome to our comprehensive lesson on model development and training. In this guide, we explore how data scientists transform ideas into robust, testable machine learning models and how MLOps engineers facilitate this process by simplifying model training and experimentation.

​
Use Case: Pharmacy Inventory Prediction

Imagine managing a network of pharmacies where predicting inventory needs is critical. By training a machine learning model with historical data from these pharmacies, you can forecast future inventory requirements accurately. This use case highlights several key considerations that data scientists must address before they begin model development:
  • Where and how should model development start?
  • Which development environment and tools are optimal?
  • How can the model be effectively trained on large datasets?
  • What are the best practices for model versioning and evaluation?
  • How should experiments be tracked to maintain a data-driven research process?
The diagram below visually summarizes these five essential questions:
The image is about model development and training, listing five questions related to model development, training, evaluation, version tracking, and experimentation. It includes an illustration of a data scientist.
Before initiating model development, it is crucial to outline the challenges and questions mentioned above to ensure a smooth building and training process.

​
Model Development Process

The initial phase of model development involves selecting the computing strategy best suited for your task. Once you have made this decision, the focus shifts to training the model. Data scientists commonly use several flexible and powerful development platforms such as Jupyter Notebook, Google Colab Notebook, AWS SageMaker, and GCP Vertex AI. Each of these environments offers unique benefits for building, training, and experimenting with machine learning models. The general steps in the model development workflow include:
  1. Installing the required packages in the working environment.
  2. Connecting to the relevant data sources.
  3. Building and training the machine learning model.
The illustration below represents this process, showing how a data scientist initiates model computing to create an ML model either locally or on popular cloud platforms:
The image illustrates the process of model development, showing a data scientist working on model computing, leading to an ML model that can be built locally or in cloud IDEs like Jupyter, Colab, Amazon SageMaker, and Vertex.ai.

​
The Role of MLOps Engineers

MLOps engineers are key players in the model development lifecycle. They ensure that data scientists have access to scalable, efficient environments and tools. Their responsibilities include:
  • Setting up and managing cloud environments like AWS SageMaker or GCP Vertex AI.
  • Facilitating ad hoc deployment of Jupyter Notebooks on Kubernetes.
  • Automating infrastructure so that data scientists can concentrate on model experimentation without technical interruptions.
For instance, a data scientist might start with data loading and preprocessing within a Jupyter Notebook. After experimenting with various algorithms, they may switch to Google Colab Notebook or AWS SageMaker to leverage GPU acceleration. Finally, the trained model versions are stored and tracked on GCP Vertex AI for future experiments and deployment.
The collaboration between data scientists and MLOps engineers is vital for maintaining an efficient, scalable model development and training pipeline.

​
Summary

In summary, model development and training encompass the following core aspects:
  • Choosing the appropriate machine learning algorithms.
  • Leveraging cloud-based tools and environments for efficient model building.
  • Implementing robust processes for experiment tracking and model versioning.
MLOps engineers support data scientists by streamlining infrastructure deployment and enabling a smooth transition from model experimentation to production. This coordinated approach ensures that models are not only developed efficiently but are also scalable and easy to maintain. Thank you for joining us in this lesson on model development and training. We look forward to exploring more advanced topics with you in future sessions.
For further reading and resources, consider checking out the following links:

Watch Video