What you will learn
- What belongs in experiment tracking versus the model registry
- How deployment patterns differ for batch, online, and edge models
- The build order from tracking to deployment to monitoring
- Why monitoring, not training, is where most production models fail
What this project is
In a notebook, a model is an experiment. In production, it is a service with uptime, cost, latency, and a habit of degrading quietly as the world changes. MLOps is the discipline of treating that service properly: every model version traceable to its data, code, and parameters, and every deployment reversible.
The platform's core promise is reproducibility. When a production model misbehaves, the team must be able to answer which data trained it, which code ran it, and which version ran yesterday. Without that trail, debugging becomes guesswork and rollbacks become risky.
Core features
- Experiment tracking that records parameters, metrics, data versions, and code commits for every run.
- A model registry with stages for staging, production, and archived, plus approval gates.
- Pipeline definitions for training and batch scoring, scheduled and triggered, with retry and alerting.
- Deployment targets for online endpoints, batch jobs, and edge exports from the same registry.
- Drift and performance monitoring with alerts tied to model metrics, not just server health.
- Rollback in one action, restoring the previous model version and its config.
How it is built
- Tracking first. Experiment records and a registry give immediate value even before automated deployment exists.
- Training pipelines second. Repeatable, scheduled training jobs that write to the registry.
- Deployment third. Online endpoints and batch scoring, with staging gates and rollback.
- Monitoring fourth. Drift, accuracy, latency, and cost dashboards with alerting.
- Governance last. Approvals, audit trails, and access controls for regulated teams.
Technology considerations
The registry is the centre of the system, so its data model deserves care: a model version links to a training run, a dataset version, and a deployment record. Compute should be separate from orchestration, so training can run on GPUs while the control plane stays light. Monitoring needs both prediction logs and ground truth where available; without ground truth, drift metrics are the best available signal. Costs vary with deployment scale; confirm with a quote.
- A registry service with versioned models, stages, and approvals.
- An orchestrator for training and scoring pipelines with scheduling and retries.
- Separate compute pools for training and inference to control cost.
- A metrics store for predictions, drift, and latency over time.
Challenges to plan for
- Reproducibility gaps. A run that cannot be repeated because data or library versions changed is a debugging dead end.
- Drift without ground truth. Many teams detect input drift but never learn whether predictions actually worsened.
- Deployment fear. Teams that deploy by hand avoid shipping, so automation must make rollback visibly safe before it earns trust.
- Cost creep. Retraining on a schedule, rather than on drift signals, burns GPU budget for little gain.
A worked example
An online retailer runs a recommendation model that quietly degraded after a catalogue change introduced new product categories the model had never seen. Before the MLOps platform, the team would have discovered this from falling sales weeks later. With the platform, drift monitoring flags the shift in input categories within a day, and the registry shows exactly which model version is live, which data trained it, and which version ran last week.
The team retrains on the updated catalogue, registers the new version as staging, and runs it in shadow mode beside the live model, comparing recommendations on identical traffic. When the shadow results beat the live model on click-through and revenue per session, the team promotes the new version through the approval gate and rolls it out with a canary release. The old version stays one rollback action away for the first week. Total time from drift alert to improved recommendations is under ten days, and every step is traceable if the new model ever needs explaining.
Who it is for, and its limits
This fits teams running more than a handful of production models, or any team where a bad model version has real business cost, such as pricing, fraud, or recommendations.
It does not fit a team with one model updated twice a year, where manual deployment plus good version control is enough. It also does not fix model quality; it ships and watches models, it does not make them accurate.
Frequently asked questions
What is the difference between experiment tracking and a model registry?
Tracking records every experiment run. The registry holds the few models chosen for deployment, with stages and approvals. Tracking is a lab notebook; the registry is a release system.
How do you monitor a model without ground truth?
With input drift, prediction distribution shifts, and latency and error rates. These flag change, though only ground truth confirms accuracy loss.
Should retraining be scheduled or drift-triggered?
Drift-triggered where ground truth or strong drift signals exist, scheduled where they do not. Scheduled retraining is simpler but costs more.
What deployment pattern suits most teams?
A shadow or canary release first, comparing the new model against the old on live traffic before full rollout.
Does MLOps replace data engineering?
No. It depends on versioned, reliable data pipelines, and weak data work shows up as model problems.
What happens when a model degrades in production?
Drift and performance monitoring alert the owning team, the registry identifies the live version and its training data, and the team retrains, tests in shadow or canary, and rolls back instantly if the replacement misbehaves.