What you will learn
- What training-serving skew is and why it breaks models
- How online and offline stores divide the work
- The build order from shared definitions to a full store
- Why feature discovery matters as much as feature serving
What this project is
A model learns from features computed one way in training and served another way in production: a slightly different window, a stale value, a rounding difference. The result is a model that scores well offline and misbehaves live, and the cause hides in plumbing nobody owns. A feature store fixes this by defining each feature once and serving it from the same definition in both places.
The store has two faces. The offline store serves historical features for training, correct as of each training example's time. The online store serves the latest values fast for live prediction. Keeping those two consistent is the platform's central job.
Core features
- Feature definitions in code, versioned and reviewed like any other source.
- An offline store with point-in-time correct historical values for training.
- An online store with low-latency reads for live prediction.
- Feature discovery with descriptions, owners, and usage examples.
- Backfill and materialisation jobs that compute features on schedule and on change.
- Monitoring for staleness, missing values, and distribution shifts per feature.
How it is built
- Shared definitions first. Features defined once, used by one training pipeline and one serving path.
- Offline store second. Point-in-time correct history for training, proven against existing pipelines.
- Online store third. Low-latency serving for live models once offline correctness holds.
- Discovery fourth. Search, ownership, and documentation as feature counts grow.
- Monitoring last. Freshness, drift, and usage views for the features teams depend on.
Technology considerations
Point-in-time correctness is the hardest technical requirement: a training row must see feature values as they were at that row's event time, never later values that leak the future. Materialisation jobs must therefore record effective times, and the offline store must query by them. The online store needs a fast key-value layer, while the offline store suits a columnar warehouse; one store rarely serves both well. Scope varies with feature counts; confirm with a quote.
- Versioned feature definitions stored as code.
- A columnar offline store with point-in-time queries.
- A low-latency online store for live feature reads.
- Materialisation jobs with freshness and drift monitoring.
Challenges to plan for
- Leakage through time. A feature computed with future information inflates offline scores and fails live.
- Stale online features. A pipeline that stops updating leaves live models predicting on old values.
- Definition sprawl. Hundreds of similar features with unclear owners slow every new model.
- Backfill cost. Recomputing history for a new feature over years of data is expensive and slow.
A worked example
A bank's credit-risk model scored well in testing but underperformed live, and the investigation found the cause: training computed a customer's average balance over ninety days, while serving computed it over thirty, because two teams had implemented the same feature differently. After adopting a feature store, the balance feature is defined once in code, versioned, and served from the same definition to training and to live prediction.
Training pulls point-in-time correct values from the offline store, so no training row ever sees a balance from after its decision date, eliminating a subtle leakage source the team had not detected. Live prediction reads the latest value from the online store in milliseconds. When the risk team adds a new feature for recent transaction velocity, it is backfilled historically, tested offline with correct timestamps, and promoted to serving without a second implementation. The next model audit takes hours instead of weeks, because every feature has an owner, a definition, and a history.
Who it is for, and its limits
This fits teams with many models sharing features, or any team bitten by training-serving skew in production.
It does not fit a team with one model and a handful of features, where a shared library and careful pipeline code are enough. It also does not create good features; it serves the features teams define.
Frequently asked questions
What is training-serving skew?
When training and live prediction compute features differently, so offline performance does not match production behaviour.
What is point-in-time correctness?
Serving each training example the feature values that existed at that example's time, never values from later.
Why two stores, online and offline?
Training needs historical correctness over large data; serving needs the latest value in milliseconds. Different stores suit each job.
Do feature stores replace data pipelines?
No. They sit on top of pipelines, adding shared definitions, versioning, and consistent serving.
When is a feature store overkill?
With one model and few features, a shared feature library plus disciplined pipelines usually suffices.
What is training-serving skew?
When training and live prediction compute features differently, so a model that scores well offline behaves differently in production. A feature store prevents it by defining each feature once.