How to choose an embedding model and budget for changing it

What the embedding model decides

An embedding model turns a piece of text into a list of numbers, a vector, such that texts with similar meaning land close together. Every semantic search and every RAG system depends on that one property. If the model thinks two passages are similar when your users would not, no amount of downstream tuning will fix the results.

That makes the embedding model the quietest high-stakes decision in the stack. It is cheap to pick and expensive to change, because every stored vector was produced by it. Vectors from two different models cannot be compared, so switching means re-embedding the whole collection.

The good news is that the decision is tractable. There are perhaps eight criteria that matter, and a half-day test on your own documents will separate the candidates more reliably than any leaderboard.

The eight criteria

The scorecard below is the weighting we typically start from for an enterprise knowledge base. Adjust it to your situation; a consumer product with tight latency will weight the rows differently.

Domain vocabulary fit

General-purpose models are trained on the open web. If your documents are dense with product codes, legal citations, clinical terms or internal acronyms, a general model may place them poorly. This is the criterion most likely to surprise you, and it is only measurable on your own data.

Language coverage

If users write in Swedish, Korean or Hindi as well as English, you need a model trained on those languages, and you need to test cross-lingual queries explicitly. Many strong English models degrade sharply elsewhere.

Hosting and residency

Hosted embedding APIs send your document text to a third party. For public marketing content that is fine. For contracts, patient notes or HR files it may not be, and open-weight models served inside your own environment become the default. Our guide on on-prem vs cloud LLMs applies here too.

Migration cost

Changing model later means re-embedding every chunk, re-running evaluation and re-indexing. For a large collection that is days of compute and a careful cutover. Weight the criterion by how large your collection will be in two years rather than today.

Licence and lock-in

Open-weight models can be run anywhere forever. API models can be deprecated, repriced or changed. Deprecation is the quiet risk: when a hosted model is retired, you face a forced migration on the vendor's timetable.

Latency and throughput

Indexing a million chunks through a rate-limited API takes a long time. Query-time embedding adds to every request. Larger models are slower; self-hosted models depend on your hardware.

Dimensions and storage

Vector size drives index size, memory footprint and search speed in your vector database. Some model families let you truncate to fewer dimensions with modest quality loss, which can halve storage. Test it rather than assuming bigger is better.

Maximum input length

Each model accepts up to a fixed number of tokens. That ceiling constrains your chunking strategy. Longer limits give you freedom; they do not mean long chunks embed well.

Hosted APIs versus open-weight models

The first fork in the road is whether to call a provider's API or run a model yourself. Both are legitimate; the table shows where each wins.

A common and sensible path is to pilot on a hosted API, prove the use case, then move to a self-hosted model before the collection becomes large enough to make migration painful.

How to test candidates on your own data

Public leaderboards are useful for building a shortlist and useless for making the final call. The test that matters takes half a day.

1. Build a small golden set. Fifty to a hundred real questions from your users, each paired with the passage that answers it. Include hard cases: codes, abbreviations, multilingual queries.
2. Embed a representative sample. A few thousand chunks is enough to see differences. Use your real chunking.
3. Measure retrieval rather than answers. For each question, check whether the correct passage appears in the top results. Compare candidates on that single number.
4. Add a reranker and measure again. A reranker can narrow the gap between models, which may let you pick the cheaper or more private one.
5. Check the operational numbers. Index time, query latency and storage per million chunks for each candidate.

Write the results down. The same harness becomes your regression test when you change chunking or model later; evaluating LLM outputs covers the wider discipline.

Pitfalls we see

When a fine-tuned embedding model is worth it

Open-weight embedding models can be fine-tuned on pairs of queries and matching passages from your own domain. Done well, this teaches the model that "PO" means purchase order in your world, and that two part numbers one digit apart are not similar. It is worth considering when the local test shows a general model missing domain-specific matches that a reranker cannot rescue, and when you have, or can generate, a few thousand good query-passage pairs.

It is not a first step. Hybrid search and a reranker fix most of the same symptoms with far less effort, and a fine-tuned model is one more artefact to version and retrain. Treat it as the last rung on the ladder, in the same spirit as prompt engineering vs fine-tuning.

A note on images, audio and code

Text is not the only thing that embeds. Multimodal models place images and text in the same space, which lets you search product photos by description. Code-specific models understand identifiers and structure better than general ones. The criteria are the same; the test data must match the content you have.

Changing models later

You will change model eventually. Make it a routine job rather than a crisis.

Store chunk text and metadata in a system you control as well as in the vector index. Record the model name and version alongside every vector. Keep the golden set current. When the time comes, embed into a new index in parallel, run the evaluation, compare, and switch the retrieval layer over when the new index is at least as good. The old index stays until you are sure.

Done this way, a model change is a background re-index and a sign-off. Our embeddings strategy service sets this up as part of every AI knowledge base build, and the choice of store is covered in our vector database comparison.

What matters most

• The embedding model defines similarity; nothing downstream can correct a poor fit.
• Domain vocabulary, language coverage and hosting constraints usually decide the choice.
• Shortlist from leaderboards, decide from a half-day test on your own questions and documents.
• Bigger dimensions are not automatically better; measure storage and speed alongside quality.
• Hosted APIs are fast to start; open-weight models remove residency and deprecation risk.
• Pin the model version, keep chunk text outside the index and budget for re-embedding.

Related services