What is a Vector Database?

Why does your search box return nothing for the query “car” when the document you’re seeking has the word “sedan” in the title? You didn’t misspell anything. The words simply didn’t match. And to a keyword index, that’s all that matters. But a vector database can close the gap between what someone typed and what they intended. Close it once, in a layer every feature can call, and the assistant you ship next quarter starts from something that works.

What Does a Vector Database Store?

A vector database stores embeddings. An embedding, or vector, is a list of numbers a model produces to stand in for the content’s meaning. Feed it a paragraph, a product photo or a PDF, and anything long gets cut into chunks first, with each chunk becoming a few hundred to a few thousand floating-point numbers. For text, the word embeddings idea is the same one scaled up: passages that mean similar things, land on similar numbers.

You skip the synonym table entirely. Because close numbers mean close meaning, “invoice” and “bill” land next to each other without anyone maintaining one. The model already learned most of the ways people phrase a request, which is exactly the maintenance work keyword systems bury you in.

How Does a Vector Database Search Based on Meaning Instead of Keywords?

Storing meaning as numbers is only half of it. The search is where it pays off: you ask for a car and it finds the sedan. That is semantic search. A query runs through the same embedding model and becomes its own vector, and the database returns the stored items whose vectors sit closest. That is why “how do I expense a flight” can surface a page titled “travel reimbursement policy” though the two share almost no words.

Doing that honestly means measuring the distance from your query to every vector in the store, which stops being feasible when they number in the millions. So vector databases lean on approximate nearest-neighbor search: they trade a sliver of exactness for a search that returns in milliseconds. That tradeoff has a sharp edge. Approximate search can miss the true closest match, so an exact hit on a specific order ID still belongs in a key lookup. A vector database complements your exact-match queries; it doesn’t retire them.

Why Do Your AI Features Depend on a Vector Database?

That retrieval step is not just for search boxes. The moment you put an AI assistant in front of your content, the vector database becomes the part that decides what the model gets to see. The pattern has a name: retrieval-augmented generation, or RAG. Before the model answers, retrieval pulls the most relevant passages from your content into the prompt, so the model writes from your material rather than from training.

That grounding is the whole point. Feeding the model real passages curbs its habit of inventing confident answers, and the team behind the original RAG paper found that models produce more factual, more specific language when they retrieve before they generate. It doesn’t make a model incapable of being wrong, though, and treating retrieval as a hallucination cure is how you ship a confidently wrong answer to a customer. What grounding does is anchor the answer to something you can point at, which separates a demo from a feature you’re willing to put your name on.

What Does It Take to Add Your Second Content Source?

Grounding an assistant on one tidy corpus is the easy version. The real problem arrives with the second source, because each reaches the same vector store by a different road:

How Do You Tell Whether a Retrieval Change Helped?

The uncomfortable part of running retrieval yourself is that it fails quietly. A keyword search returning nothing is obviously broken; a semantic search that returns plausible passages and misses the right one produces an answer nobody flags.

Warning: Latency is the easiest retrieval metric to watch and the least informative. A change that halves response time while surfacing the wrong passages looks like a win on every dashboard you have.

Keep a fixed set of real questions with known-good answers and replay it against the candidate index after every change to chunking or the embedding model. Standard retrieval measures give you the shape of it: whether the right passage came back at all, and whether it landed near the top where the model will use it.

Where Does That Leave Your Retrieval Layer?

Once retrieval has a test set and an owner, it stops being a feature and becomes infrastructure. A vector database is not a fancier search index bolted onto your app; it’s the layer your search box, your assistant and next quarter’s feature release all read from. So the useful question isn’t which database to pick. It’s which of your content is reachable by meaning-based retrieval today, and which is trapped in formats a retriever quietly skips. Audit one search or assistant feature against that this week, then decide whether the layer under it is something you keep rebuilding or stand up once and reuse.

FAQ

Do I Need a Separate Vector Database or Can My Existing Database Handle This?

You have both options. Purpose-built vector databases exist, but many general-purpose databases now add vector search alongside their normal tables, so exact-match queries and semantic search live in one place. The deciding factor is usually scale and how tightly semantic search joins your structured data, not whether a dedicated engine is faster.

How Many Passages Should Retrieval Put in the Prompt?

Start small and tune the number against your own test set. Too few, and the answer misses the detail it needed. Too many and the right passage competes with filler for the model’s attention, while you pay for every extra token on every call. There is no universal number. Your test set is what will tell you when you have gone too far.

What Happens to Retrieval When My Content Changes Every Day?

Only changed content needs re-embedding, so daily churn stays incremental. The failure mode is staleness, not cost: if ingestion runs weekly and docs change daily, the assistant answers from last week’s policy. Match the schedule to how fast the content moves, and delete embeddings for retired content in the same job, since a retriever cannot tell a passage is superseded.