Skip to content

Chapter 5: Step-by-Step Model Development for Stage 2 MLOps

MLOps Stage 2 (see Chapter 2 on what is "Stage 2") represents a significant shift toward scalability and infrastructure maturity. Agencies begin to see ML as a full-fledged “data product” and invest in building out specialized teams and robust pipelines. This stage can be challenging, as it involves integrating new components like data pipelines, CI/CD for ML, and advanced monitoring.

Operating Model at Stage 2

  • People: A broader team structure emerges, including data scientists, ML engineers, DevOps, data engineers, and security/infrastructure specialists. Each role addresses different aspects of the ML life cycle.
  • Process: Automated data pipelines, model monitoring, experiment tracking, and systematic retraining are introduced. MLOps best practices become embedded in day-to-day workflows.
  • Technology: End-to-end CI/CD pipelines support continuous integration, advanced resource provisioning, and a robust environment for production ML applications.

5.1. Objective

The primary goal of this guide is to help readers who are familiar with Stage 1 fundamentals progress to Stage 2. Specifically, we aim to:

  • Demonstrate how to scale your existing ML operations into a fully automated MLOps pipeline.
  • Explore team composition, tools, and processes required to sustain advanced levels of ML deployment.
  • Provide end-to-end technical guide that illustrate how each MLOps component fits into a larger production ecosystem.

5.2. End State: Stage 2 Maturity

Upon reaching Stage 2, agencies benefit from:

  • Stable, scalable pipelines that handle data ingestion, transformation, model training, and automated deployment.
  • Comprehensive monitoring of both model performance and system health, enabling timely detection of data drift or model deterioration.
  • Cross-functional collaboration, where data scientists, ML engineers, DevOps, and other stakeholders each have distinct yet integrated roles, ensuring continuous improvement and reliable delivery of AI-driven solutions.

By the end of Chapter 5F, you will have the knowledge and reference materials to drive your agency or project from Stage 1 to Stage 2 MLOps maturity, backed by proven practices and real-world insights from the AI Practice team.

5.3. Overview of Contents

We have organized this chapter into six parts, each summarized below:

Chapter 5A: Setup
Chapter 5A focuses on the steps for forking the case study project and setting up a local development environment on your SEED device. The chapter covers generating personal access tokens and GPG signing for verified commits. This comprehensive guide ensures developers can quickly establish a secure, collaborative foundation for model development work.

For non-SEED users, you may want to reach out to your IT team or AI Practice to explore development setup solutions.

Chapter 5B: Data Processing
Chapter 5B focuses on core data processing steps for model development, including data imputation, feature scaling, one-hot-encoding, feature engineering, and splitting the data into train-validation-test sets. The chapter provides practical implementation guidance for creating a robust preprocessing pipeline that prepares data for effective model training while maintaining data integrity.

Chapter 5C: Hyperparameter Tuning
Chapter 5C explores effective hyperparameter tuning strategies for optimizing model performance, comparing methods like Grid Search, Random Search, and Bayesian Optimization based on their efficiency and computational requirements. The chapter provides practical implementation, with examples demonstrating how to integrate these approaches into the existing pipeline code to improve model performance.

Chapter 5D: Model Evaluation
Chapter 5D reviews evaluation metrics for regression, classification, anomaly detection, and clustering problems, including mathematical formulations, pros and cons, and key assumptions. It highlights the importance of fairness and bias detection, and walks through the code updates to include these metrics in the model evaluation script as part of the ML pipeline.

Chapter 5E: Experiment Tracking
This section provides a high-level introduction to MLFlow and its features. It compares managed and manual integration with AWS Sagemaker, noting the pros and cons of each integration approach. You will be guided through the recommended managed-service approach, along with any code modifications needed to log experiments in MLFlow.

Chapter 5F: Build & Deploy Chapter 5F guides readers through the final stages of the model development process: building reproducible artifacts and deploying models to production environments.