Skip to content

mlops-playbook_banner.png

Purpose of This Playbook

Machine Learning (ML) and Artificial Intelligence (AI) are among the most exciting and promising technologies for both companies and consumers. Tasks like reviewing credit limits, evaluating job applications, and handling customer queries are increasingly automated using discriminative and generative AI.

In the public sector, ML and AI can support policymaking through better sensemaking, streamline operations by automating tedious tasks, and enhance service delivery with AI-powered tools like job coaches and chatbots. Given their vast potential, it is no surprise that the demand for ML and AI is rapidly growing across public agencies.

However, a gap still remains. Even in agencies that have successfully developed ML or AI models, many stay stuck at the prototyping stage. Data science teams frequently express frustration over the challenges of moving models into production, citing issues such as a lack of deployment tools and IT processes that aren’t suited for ML production. This issue isn’t unique to the public sector — a McKinsey study found that 36% of companies struggle to deploy ML models beyond the pilot phase.

mlops-playbook_home_mckinsey_chart.png

This doesn’t have to be the case. With significant investment and focus, DBS reduced the time from ML development to deployment from 18 months to just 5 months. Our goal is to make ML production both effective and efficient, enabling agencies to unlock the full value of their models in weeks, instead of months or years.

Target Audience

This playbook serves as a practical guide to MLOps, offering an approach to ML production that improves deployment quality and reduces turnaround time. Our focus is on leveraging the platforms already available within the public sector.

To determine whether MLOps is right for you, consider the following common questions. If you answer "yes" to any of them, this playbook will be highly valuable for your work:

  1. Model Complexity: Are your ML models becoming more complex, requiring structured tracking of parameters and version control?

  2. Model Update Frequency: Do you need to update or retrain models frequently due to changing data or business requirements?

  3. Impact of Model Failure: Would a failure in your model significantly impact business operations? MLOps enables robust monitoring and tools to quickly detect and resolve issues.

  4. Prediction Request Volume: Does your model need to scale to handle more prediction requests or provide instant predictions with low latency?

  5. Cost of ML Maintenance: Are manual maintenance processes becoming costly? MLOps can reduce long-term costs by automating pipelines and integrating DevOps best practices?

  6. Underutilization of Data Lake / Data Warehouse: Are you struggling to develop new products despite having a strong data engineering foundation? The right MLOps or LLMOps setup can enhance your team's ability to build and deploy models more effectively—whether for short-term experiments or long-term solutions.

This Playbook tries to cover everything, from zero to hero. This is what you can expect from reading the whole PlayBook.

  1. Accelerate MLOps Adoption: Offer a structured approach for agencies looking to implement best practices in ML lifecycle management, as they progress from Stage 1 to Stage 2 maturity.
  2. Hands-On Guidance: Provide concrete, practical instructions by applying MLOps concepts to a case study on HDB resale price prediction.
  3. Align With Existing Architectures: Demonstrate how this example can be adapted from existing infrastructures adopted by MAESTRO, supplemented by additional customisations based on your agency requirements.

By following this playbook, you will gain insights into the design and maintenance of robust and scalable machine learning pipelines, including data management, CI/CD setup, model deployment, monitoring, and updates.

Chapter Overview

Using this playbook, agencies and Data Science teams can streamline the development, deployment, and maintenance of ML models, while progressively enhancing their MLOps maturity. We trust you will find this playbook beneficial and look forward to guiding you through your MLOps journey. Below is an outline of each chapter, which we can help guide where you want to read, feel free to skip chapters as required.

Chapter 1: The Case for MLOps

This chapter establishes the distinction between traditional data science work and the challenges of operationalizing machine learning in production environments. It explores how notebook-based approach of data scientists often conflicts with the stability and governance requirements of IT operations.

Chapter 2: Stages of MLOps

This chapter provides a structured framework for understanding the progressive maturity levels of MLOps implementation, from manual processes to fully automated systems. It outlines four key maturity stages. For each stage, the chapter details the essential components and capabilities across data engineering, model development, continuous integration/delivery, and monitoring systems. Organizations typically evolve through these stages incrementally, with each level building on previous capabilities. This helps agencies assess their current state and develop a strategic roadmap for advancing their MLOps maturity.

Chapter 3: Deploying You First ML Model

This chapter provides a practical walkthrough of deploying a machine learning model as a prediction service. It covers the essential steps of transforming a trained model into an API endpoint that can serve real-time predictions.

Chapter 4: Reference Guide for Stage 2 MLOps Setup

This chapter provides a MLOps framework that outlines the essential components for establishing a robust MLOps environment, including infrastructure setup with security considerations like IAM roles, S3 encryption, and VPC configurations. The chapter details CI/CD pipeline implementation, covering both data engineering workflows and model development processes with security best practices embedded throughout.

Chapter 5: Model Development

This chapter provides a guided walk-through of the entire model development process, starting with an introduction to the HDB resale price dataset, its storage practices, and methods for bringing data into your workflow. It covers everything from data preparation to model build and deploy.

To demonstrate these cocepts in practice, we also provide a detailed implementation guide using SageMaker.

Chapter 6: Model Monitoring & System Observability

Effective model monitoring and system observability are critical components of a robust MLOps infrastructure, enabling organizations to ensure their machine learning models perform reliably and consistently in production environments. This chapter explores the essential frameworks and maturity levels for implementing comprehensive monitoring solutions, from basic performance tracking to advanced observability with automated remediation capabilities.

To demonstrate these concepts in practice, we also provide a detailed implementation guide using Grafana and Prometheus.

Chapter 7: Ten Category Framework

This Chapter presents a comprehensive Ten Category Framework for assessing and advancing MLOps maturity across an organization. The framework breaks down MLOps capabilities into ten essential dimensions: Environment & CICD Setup, Networking & Security, Data Storage, Data Pipeline, Orchestration & ML Pipeline Deployment, Data Testing & Unit Testing, Model Monitoring & System Observability, ML/Infra Optimization, Feedback Loop & Model Re-Training, and Experimentation & Model Evaluation. For each category, the chapter defines three maturity levels (Basic, Intermediate, and Advanced) with specific characteristics and capabilities, providing organizations with a structured approach to evaluate their current state and plan strategic improvements. This holistic framework enables teams to identify gaps, prioritize investments, and develop a roadmap for MLOps evolution that aligns with their specific business needs and technical environment.

Chapter 8: Other Considerations

This chapter serves as a catch-all section for important MLOps topics that fall outside the core baseline practices. It includes concepts and tools that are that valuable to ensure a successful MLOps implementation.

About the AI Practice

The AI Practice group is a team in GovTech which aims to deepen data science capabilities within and across the government. In particular, we are focusing on MLOps as we see it as a critical enabler for agencies in realising the full value of data science and AI.

We started this workstream in late 2023 in response to ground feedback on how difficult it was to deploy models, and how these deployments were not very efficient nor reliable. We observed common challenges with ML deployment across the government, and sought to examine how we could best support and uplift agencies in this area.

For enquiries, please contact Victor Ong (victor_ong@tech.gov.sg), and we will direct you to the appropriate experts.