Embedding models, chunking and versioning chosen on your own data
The embedding decisions you make in week one decide what your search or RAG system can find for years. We make them deliberately, test them on your data and plan for the day they change.
Embeddings are the one layer you cannot patch later
Prompts can be edited in an afternoon. Rerankers can be swapped. Embeddings are baked into every vector in the index, so a weak choice means re-processing everything.
Three decisions interlock. The model fixes how meaning is represented and in which languages. The chunking decides what unit of text gets a vector, which caps how precise a retrieval can be. The versioning plan decides whether changing either is a routine job or a weekend outage. We treat the three as one design, because changing one usually changes the others. The costs show up in the vector database bill and the RAG answer quality alike.
Choosing the model
Seven criteria, in the order they usually decide the outcome
Criterion
Why it matters
What to check
Typical trade-off
Domain fit
Models trained on general web text may miss the vocabulary of law, medicine, engineering or your product.
Score on your labelled queries rather than public leaderboards.
Specialised or fine-tuned models score higher but narrow your options.
Multilingual coverage
Users may search in one language for content in another.
Include cross-language pairs in the test set.
Multilingual models can be slightly weaker on English-only tasks.
Licensing and hosting
API models send text to a provider; open-weight models stay on your infrastructure.
Data residency, contractual limits, privacy obligations.
Self-hosting keeps data in-house but adds operations work.
Dimension
Higher dimensions store more nuance and cost more memory, disk and index time per vector.
Storage at your vector count; whether the model supports truncated dimensions.
Smaller vectors are cheaper and faster, sometimes with a small quality loss.
Context length
The maximum text the model embeds in one go bounds your chunk size.
Your largest sensible chunk fits with room to spare.
Long-context models tempt oversized chunks that retrieve badly.
Cost
Paid per token for API models; per GPU-hour or CPU-hour when self-hosted.
One-off indexing cost plus ongoing query embedding and re-embeds.
Cheapest per token is not cheapest once re-embeds and storage are counted.
Latency
Query embedding sits on the critical path of every search.
Median and tail latency from your region, under load.
Larger models and remote APIs add tens of milliseconds per query.
The market has two broad camps. API models from providers such as OpenAI, Google, Cohere and Voyage are easy to adopt and improve without effort on your side. Open-weight families such as BGE, E5, GTE and Nomic run on your own hardware and keep text in your network. Neither camp wins by default. See on-prem versus cloud for the wider hosting question and the glossary for the basics.
Chunking
The same document, split three ways
A chunk is the unit that gets a vector and comes back as a result. Too big and the vector averages several ideas; too small and context is lost. The split method matters as much as the size.
Fixed windows are easy and blind to meaning. Structure-aware splits follow headings and keep context intact. Semantic splits isolate ideas at the cost of an extra model pass per document.
Our starting point
Structure-aware chunks with a size cap, small overlap, and the heading path prepended to each chunk so the vector carries its context.
Special cases
Tables are kept whole or converted to text rows. Code keeps function boundaries. Contracts split at clauses. Tickets keep question and answer together.
How we decide
Two or three configurations are indexed side by side and scored on the labelled query set. The winner ships; the runners-up are recorded for the next review. See chunking.
Metadata
What every chunk should carry besides its vector
Vectors answer "what is similar". Metadata answers everything else: who may see it, where it came from, how old it is, and whether it needs re-embedding. Skimping here is the most common reason a pilot cannot become a product.
Source system and document ID, so updates and deletions find the right vectors
Location: URL or path, page or section, for citations and deep links
Heading path and position in the document, for context and ordering
Created and updated timestamps, for recency weighting and audits
Language, so multilingual corpora can be filtered or routed
Access-control labels copied from the source, for permission-aware retrieval
Content hash, so unchanged chunks are skipped on the next sync
Embedding model name and version, and chunker version, per vector
Permissions travel with each chunk
If a document is restricted in SharePoint or Confluence, its chunks must carry that restriction. The retriever must filter on it before anything reaches the model. The secure RAG architecture page covers the full pattern.
Storage note
Rich metadata is small next to the vectors themselves. The RAG cost estimator shows how vector count and dimension drive storage, with editable assumptions.
Versioning and re-embedding
Changing the model without a maintenance window
A re-embed will happen. The plan below makes it a scheduled job with a rollback.
Always
Tag every vector
Store the embedding model, its version and the chunker version alongside each vector. You cannot migrate what you cannot identify.
Trigger
Decide to re-embed
Triggers: a model change, a chunker change, a large content drift (new product line, new language), or a measured drop on the evaluation set.
Day 1–3
Build the new index beside the old
Blue-green: a second collection or table receives the new vectors while the live one keeps serving. Throughput is throttled to protect the source systems and the embedding budget.
Day 3–4
Evaluate both
The same labelled queries run against old and new. Recall@k, MRR and, for RAG, answer quality must hold or improve before anything switches.
Day 4
Switch the alias
The retrieval service points at the new index through an alias or configuration flag. Rollback is flipping it back.
Day 11+
Retire the old index
After a soak period with monitoring, the old collection is dropped and its storage reclaimed.
Day counts are for a corpus in the hundreds of thousands of chunks and depend mostly on embedding throughput. Larger corpora stretch the build step, not the rest. The same harness used here is the one we set up under fine-tuning and evals, and the measurement approach follows evaluating LLM outputs.
What we deliver
An embeddings strategy engagement
Corpus review (week 1)
Content types, languages, sizes, update rates, access rules. Labelled query set drafted with your domain staff.
Model shortlist (week 1–2)
Two or three candidates against the criteria table, including at least one open-weight option where residency matters.
Bake-off (weeks 2–3)
Each model and chunking configuration indexed and scored on the same queries. Cost and latency recorded alongside quality.
Decision pack (week 3)
Recommended model, chunking rules, metadata schema, versioning runbook and the evaluation harness, handed over as code.
Typical duration is two to four weeks, and the output drops straight into a semantic search or RAG build. If you are weighing fine-tuning an embedding model on your own data, we cover that under fine-tuning and evals. It is rarely the first thing to try.
The one that scores best on your own labelled queries within your constraints on language, data residency, cost and latency. We shortlist two or three candidates against the criteria table on this page and run them all through the same evaluation harness. Our guide to choosing embedding models goes deeper.
What chunk size should we use?
There is no universal number. Most corpora do well with structure-aware chunks capped at a few hundred tokens, with small overlap. Tables, code and legal clauses need their own rules. We test two or three chunking configurations on the evaluation set and keep the winner.
Do we have to re-embed everything when we change the model?
Yes, vectors from different models are not comparable, so a model change means a full re-embed. The point of a versioning plan is to make that a routine, zero-downtime job rather than a crisis: build the new index beside the old one, evaluate, switch, retire.
Can we use an open-weight embedding model on our own servers?
Yes. Open-weight families such as BGE, E5, GTE and Nomic run well on modest hardware and keep data in-house. The trade-off is that you operate the model and own its upgrades. We compare that honestly against API models on your evaluation set and your data privacy requirements.
What metadata should be stored with each chunk?
At minimum: source ID, location (URL or path), heading path, position, timestamps, language, access-control labels, a content hash, and the embedding model and chunker versions. That set makes filtering, citations, permissions and re-embedding all possible later.
Talk to us about your AI project
Tell us what you are trying to do, which systems are involved and what is in the way. We reply within one business day with an opinion and a suggested next step.