What you will learn
- How embedding search differs from keyword search
- Which index types suit which scale and recall targets
- The build order from single collection to multi-tenant service
- Why retrieval quality depends on embeddings and chunking, not only the index
What this project is
Embeddings turn text, images, or audio into lists of numbers where similar items sit close together. A vector database indexes those numbers so a query can retrieve the closest matches without scanning every record. That powers retrieval-augmented generation, recommendation, duplicate detection, and semantic search.
The hard part is not the maths. It is the service around it: keeping indexes fresh as data changes, filtering by tenant or permission before ranking, and returning results fast enough for a chat response. A research demo that searches ten thousand vectors says little about a service holding a hundred million.
Core features
- Approximate nearest-neighbour indexes with tunable recall and latency trade-offs.
- Metadata filtering combined with vector ranking, so results respect tenant and permission rules.
- Collection management with per-tenant isolation, quotas, and usage metering.
- Hybrid search blending vector scores with keyword scores for queries where exact terms matter.
- Incremental indexing so new and changed vectors become searchable without full rebuilds.
- Backup, restore, and replication for collections customers depend on.
How it is built
- Single-collection search first. Ingest, index, and query for one tenant, with measured recall and latency.
- Filtering and hybrid search second, because real products filter by customer, document type, or date.
- Multi-tenancy third. Isolation, quotas, and metering for a hosted service.
- Scale fourth. Sharding, replication, and index tuning for large collections.
- Operations last. Backups, upgrades without downtime, and per-collection health views.
Technology considerations
Index choice is the central technical decision: graph indexes give strong recall at moderate scale, while quantised indexes trade some accuracy for memory savings at very large scale. Vectors are large, so memory and storage costs dominate the bill, and dimension reduction or quantisation is often the cheapest performance win. The query path should filter before ranking where possible, so tenants never see each other's data through a ranking leak. Confirm hosting costs with a quote for your expected collection sizes.
- Graph or quantised indexes chosen per collection size and recall target.
- A metadata store kept in step with the vector index on every write.
- Sharding and replication for large or high-traffic collections.
- A query service that enforces filters before returning ranked results.
Challenges to plan for
- Recall versus latency. Faster indexes miss neighbours; the platform must let customers choose the trade-off per collection.
- Stale indexes. Deletes and updates that lag behind the source data return results that no longer exist.
- Embedding drift. When the embedding model changes, old vectors need re-embedding, which is a large backfill job.
- Filter leakage. Ranking before filtering can surface another tenant's items in scores or counts, so filter-first design matters.
A worked example
A support software company holds eight million help-centre articles, tickets, and community answers, and wants its chatbot to retrieve the right article before answering. The platform ingests embeddings for every document, with metadata filters for product, version, and language, so a query from a German user on version 3 of the product never retrieves an English article for version 1. Hybrid search blends vector similarity with keyword matching, catching article titles that contain exact error codes the embedding model would blur.
When the company switches embedding models, the platform re-embeds the collection as a background backfill into a versioned collection, while the old collection keeps serving traffic. Cutover happens only after recall measurements on a held-out query set match or beat the old collection. Query latency stays under a few hundred milliseconds at peak, because the index is sharded and hot collections are replicated. The chatbot's answer quality improves measurably, but the platform team measures retrieval recall weekly, since a retrieval regression would otherwise surface as vague chatbot complaints rather than a clear bug.
Who it is for, and its limits
This fits teams building retrieval-augmented generation, semantic search, or recommendation features, and vendors offering search as a service.
It does not fit small collections, where a simple library search inside the application is faster to build and cheaper to run. It also does not fix poor retrieval: if chunks are badly split or embeddings are weak, a faster index returns poor results faster.
Frequently asked questions
How many vectors justify a dedicated vector database?
Roughly when collections pass the low millions, need multi-tenant filtering, or must stay fast under concurrent queries. Below that, an in-process library often suffices.
What is recall in vector search?
The share of true nearest neighbours the approximate index actually returns. Higher recall costs more memory and latency.
Do I need hybrid search?
When queries contain exact names, codes, or numbers, keyword matching catches what embeddings miss. Many production systems blend both.
What happens when the embedding model changes?
Old vectors become incompatible, so the platform must re-embed stored content, ideally as a background backfill with versioned collections.
Is a vector database a replacement for a normal database?
No. It pairs with one: the normal database holds source records and metadata, the vector store holds embeddings for search.
How do you switch embedding models without downtime?
By backfilling a new versioned collection in the background while the old one serves traffic, then cutting over after recall measurements on held-out queries confirm the new collection performs at least as well.