Cross-Encoder vs Embedding Model Upgrades in RAG: A Safe Migration Playbook for Retrieval Quality Gains

A few quarters into running a production RAG system, most teams hit the same wall: the first version works well enough to launch, but retrieval quality starts becoming the dominant complaint surface. Users say the assistant is “missing obvious documents,” “quoting the wrong policy version,” or “answering from vaguely related chunks instead of the exact section.” Product asks for a quality jump. Engineering has three obvious levers: upgrade the embedding model, add or strengthen a reranker, or do both.

On paper, this sounds straightforward. In practice, it is one of the easiest ways to destabilize a healthy-but-imperfect RAG system.

Embedding upgrades change the geometry of your retrieval space. Nearest neighbors change, score distributions shift, chunking decisions that were acceptable under one model become less effective under another, and hybrid retrieval weighting can stop making sense overnight. Cross-encoder rerankers look safer because they leave the index unchanged, but they introduce latency, cost, operational complexity, and sometimes surprising quality regressions when the candidate set is weak. Doing both at once can produce the biggest gains—and the hardest-to-debug failures.

The safest migration mindset is not “what model is best on the benchmark?” but “what intervention improves end-to-end answer quality in our system with the least production risk?” That is a different question. It requires separating candidate generation quality from ranking quality, evaluating retrieval changes before generation washes out the signal, and rolling out with enough observability that you can explain not just whether quality changed, but why.

This article is a production-oriented playbook for deciding whether retrieval gains should come from a new embedding model, a stronger reranker, or both—and how to migrate safely. I’ll walk through the failure modes teams actually hit, how to identify whether the problem is candidate recall or ranking precision, how to run offline retrieval diffing, how to backfill and dual-run indexes, how score distributions and thresholds break, what to do about chunk re-embedding and hybrid search, and how to stage rollout guardrails so you improve relevance without introducing chaos.

A realistic failure scenario

Consider a support and internal knowledge assistant used by 8,000 employees. The first production system looks like this:

Documents are chunked into 400–700 token passages with 15% overlap.
Each chunk is embedded with a general-purpose embedding model.
Retrieval is hybrid: BM25 plus vector search.
The top 40 candidates are fused, then a lightweight reranker picks the top 8.
A generation model answers with citations.

The launch goes well. Six months later, problems accumulate:

New product names and internal acronyms are not retrieved reliably.
Queries asking for the “latest policy” often surface older but semantically similar chunks.
High-recall searches return too many broad chunks, and the reranker still selects passages that mention the concept but not the answer.
Some teams report the assistant is worse for keyword-heavy tasks after a silent embedding model refresh.

The staff response is predictable:

One group says, “Just upgrade embeddings. The newer model is better on MTEB.”
Another says, “Keep the index stable and add a stronger cross-encoder reranker.”
A third says, “We should do both and ship the best stack.”

If you take any of those paths without isolating the actual retrieval bottleneck, you can spend weeks re-embedding millions of chunks, doubling infrastructure cost, or adding 300–800 ms tail latency only to discover your answer quality barely moved.

Pattern identification: is the issue candidate recall, candidate ranking, or content representation?

Before touching models, classify the failure. In production RAG, retrieval quality problems usually come from one of four buckets:

Candidate recall failures The right chunk never enters the candidate set. No reranker can fix this. Typical symptoms:
- Correct cited chunk absent from top 50/100.
- Queries with specific identifiers, codes, table entries, or niche jargon underperform.
- Hybrid lexical retrieval outperforms semantic retrieval on exact-match tasks.
Candidate ranking failures The right chunk is present but buried. A stronger reranker often helps here. Typical symptoms:
- Gold chunk appears in top 50 but not top 5.
- Several semantically related chunks compete, but the system prefers broad overview text over exact answer passages.
- Answer quality improves dramatically when humans manually reorder retrieved chunks.
Content representation failures The chunking, metadata, or document freshness is wrong. Model upgrades may help less than expected. Typical symptoms:
- Correct answer spans split across chunk boundaries.
- “Latest version” failures trace back to missing metadata or weak document versioning logic.
- Tables, lists, and structured documents retrieve poorly because text extraction is bad.
Retrieval-policy failures The retrieval stack is mechanically correct, but thresholds, hybrid weights, or filtering logic are wrong. Typical symptoms:
- Similarity thresholds reject useful chunks after a model migration.
- BM25 dominates because score normalization is off.
- Metadata filters remove the very content users need.

This classification matters because embeddings mostly affect candidate generation and overall neighborhood structure, while cross-encoders mostly affect within-set ordering. If you use a reranker to solve a recall problem, you will be disappointed. If you re-embed to solve a ranking problem, you may incur a lot of migration risk for a gain you could have captured more cheaply.

Why the naive approach fails

The most common naive approach is to benchmark new embeddings or rerankers in isolation and assume the higher offline metric will translate into better production answers. Several things break that assumption.

First, retrieval is a pipeline, not a single model. Improving one stage can expose weaknesses in another. A better embedding model may return more semantically broad candidates, which sounds helpful, but if your reranker is weak or your chunk granularity is coarse, the final top-k can actually get less precise.

Second, score distributions are model-specific. Teams often implicitly encode assumptions like:

similarity > 0.82 means “high confidence”
top result minus second result > 0.08 means “clear winner”
vector score can be linearly combined with BM25 using fixed weights

Those assumptions break when embeddings change. Even cosine similarity behavior can shift materially across models. The absolute score means nothing outside the model/index context in which it was calibrated.

Third, hybrid search interactions are rarely robust to model upgrades. If your production stack uses reciprocal rank fusion, weighted-score blending, or query-adaptive lexical/semantic mixing, changing the embedding model changes one input distribution while leaving the others fixed. Teams think they are testing “embedding quality,” but they are really testing “new embeddings under old fusion logic.”

Fourth, generation can mask retrieval regressions. Large models are surprisingly good at answering plausibly from weak context. If you evaluate only final answer quality, you can miss retrieval degradation that later becomes a reliability issue on harder queries. Conversely, answer-level evals may show little movement even when retrieval improves, because the generation model was already compensating.

Finally, full-stack migrations are hard to debug. If you simultaneously change embeddings, reranker, chunking, and prompt, you lose causal clarity. When quality shifts, you do not know which component helped or hurt.

The better approach: treat retrieval upgrades as staged interventions

A safer playbook is to evaluate and roll out in layers:

Measure the current retrieval stack at the retrieval level.
Decide whether the bottleneck is candidate generation or ranking.
Evaluate embedding upgrades and reranker upgrades independently.
If embeddings change, backfill a second index and dual-run before cutover.
Recalibrate thresholds, fusion logic, and retrieval policy for the new score space.
Roll out through canaries with observability focused on retrieval mechanics, not just answer ratings.

In practice, this gives you three strategic options.

Option A: Upgrade the reranker first

Best when:

Gold chunks are usually in the top 20–100 already.
Failures are mostly ordering mistakes.
You want lower migration risk and no full reindex.
Corpus size is large enough that re-embedding is operationally expensive.

Advantages:

Leaves vector index unchanged.
Easy to A/B by shadow reranking the same candidate set.
Often delivers a fast win in precision@k and citation quality.

Tradeoffs:

Adds online latency and cost.
Cannot recover chunks never retrieved.
Benefit plateaus if candidate set is weak.

Option B: Upgrade embeddings first

Best when:

Gold chunks are missing from top N entirely.
Domain terminology changed, multilingual coverage is weak, or exact semantics are poor.
Current model is clearly behind on domain fit or robustness.

Advantages:

Improves recall ceiling for all downstream ranking.
No per-query reranking cost if you keep architecture simple.
Can materially improve long-tail retrieval.

Tradeoffs:

Requires re-embedding and index migration.
Breaks score calibration and hybrid weighting.
Potentially large infra cost during dual-index operation.

Option C: Upgrade both, but in sequence

Best when:

You know both recall and ranking are weak.
You have enough observability to attribute gains.
You can afford staged rollout and temporary infra duplication.

Advantages:

Highest quality ceiling.
Lets embeddings improve recall while reranker improves precision.

Tradeoffs:

Most expensive path operationally.
Hardest to debug if not staged carefully.

My practical default: if the right documents are already appearing in the candidate set with decent frequency, start with the reranker. It is the safer, cheaper experiment. If the candidate set itself is poor, invest in embeddings. If both are weak, do not switch everything at once—land the embedding migration first, recalibrate, then evaluate the incremental value of a stronger reranker on top.

Architecture for safe migration

A production-safe retrieval upgrade architecture usually includes the following components:

Query router / retrieval orchestrator
Existing embedding model + current vector index
Candidate generation subsystem with lexical and semantic retrieval
Optional cross-encoder reranker
Offline evaluation harness with labeled and weakly labeled queries
Dual-write or batch backfill pipeline for a shadow index
Online shadow testing path
Retrieval observability dashboards
Feature flags and canary controls

A concrete migration-ready architecture looks like this:

Ingestion pipeline
- Normalize documents.
- Extract metadata: source, version, date, access control tags, document type.
- Chunk using deterministic versioned chunkers.
- Store chunk text and chunk IDs in a canonical corpus store.
Embedding pipeline
- Compute embeddings for the active model into index A.
- During migration, compute embeddings for candidate model into index B.
- Preserve a stable chunk ID across indexes so retrieval comparisons are straightforward.
Retrieval service
- For each query, issue:
  - BM25/lexical retrieval
  - Semantic retrieval against active index
  - Optionally semantic retrieval against shadow index for logging only
- Fuse results.
- Optionally rerank active candidates with the production reranker.
- Optionally shadow-rerank for evaluation.
Evaluation/logging plane
- Log top-k candidates, scores, ranks, metadata, retrieval stage timings, and whether gold chunks appear.
- For shadow systems, log diff views: added chunks, removed chunks, rank deltas, source deltas.
Rollout controls
- Per-tenant, per-query-type, or percentage-based canaries.
- Budget controls for reranker usage.
- Automatic rollback if guardrails breach.

Offline retrieval diffing: the highest leverage step teams skip

Before any production cutover, run retrieval diffing on a representative evaluation set. This is more useful than a generic benchmark because it reveals how the new retriever changes actual retrieved documents in your corpus.

Build an eval set with at least these slices:

Exact-known-answer queries: identifiers, product names, policy codes
Semantic paraphrases: natural language restatements of known content
Freshness-sensitive queries: “latest,” “current,” “new policy”
Long-tail domain jargon
Multi-hop or disambiguation-heavy queries
Adversarial negatives: similar but wrong versions or neighboring policies

For each query, collect or derive:

Relevant chunk IDs or documents
Preferred freshness/version constraints
Optional ideal citation set

Then compare systems using retrieval-first metrics:

Recall@k: does any relevant chunk appear in top k?
MRR / NDCG@k: are relevant chunks ranked early?
Success@k under metadata filters
Freshness hit rate: does retrieval include the newest valid source?
Version confusion rate: how often are superseded documents ranked above current ones?
Source diversity: are results dominated by one noisy repository?

But metrics alone are not enough. Produce a qualitative diff artifact for every changed query:

Which chunks entered the candidate set under the new embedding model?
Which chunks were lost?
Did lexical-heavy queries become worse?
Are top results now broader, more recent, more exact, or more duplicated?
Does the reranker rescue weak candidates or merely reshuffle noise?

This diff review is where you discover practical truths like:

“The new embeddings improve conceptual matching, but stop surfacing SKU-style identifiers unless BM25 weight increases.”
“The stronger reranker is excellent if the correct chunk is present, but candidate depth needs to increase from 20 to 60.”
“The embedding upgrade boosted recall, but also increased duplicate sibling chunks from the same document, reducing useful context diversity.”

That is production knowledge, not benchmark knowledge.

How to decide: embedding upgrade or reranker?

A simple decision framework:

Choose reranker-first if:

Current recall@50 is already high on your labeled set.
Relevant chunks appear but rank too low.
You need a low-risk test with minimal reindexing effort.
Latency budget can tolerate additional inference.

Choose embedding-first if:

Relevant chunks often do not appear in top 50/100.
Your domain vocabulary shifted materially since launch.
Multilingual or code/text mix performance matters.
Current hybrid setup relies too heavily on lexical matching to recover failures.

Choose both in sequence if:

Recall@50 is mediocre and NDCG@10 is poor.
Corpus scale or domain complexity means you need both broad candidate generation and precise ordering.
You can run staged migration with clear attribution.

One useful heuristic: inspect “oracle reranker potential.” For each query, assume a perfect reranker could pick the best chunk from the top N candidates. If the gold chunk is already in top 50 most of the time, reranking has strong headroom. If not, embedding improvements are the prerequisite.

Dual-index backfills and shadow reads

If you change embedding models, avoid big-bang cutovers. Run a dual-index migration.

The pattern:

Keep current index A live.
Backfill new embeddings into index B.
Verify parity in document/chunk counts and metadata.
Shadow query index B in production traffic.
Compare retrieval outputs offline and from live sampled traffic.
Canary real traffic to index B only after score and fusion recalibration.

Implementation details that matter:

Stable chunk IDs Use immutable chunk IDs derived from document ID + chunk version + chunk ordinal. If chunk text changes, generate a new chunk version. Without stable IDs, diffing two retrieval systems becomes unnecessarily painful.

Versioned chunkers If you also change chunking, treat that as a separate dimension. “New embeddings + new chunking” can work great, but it destroys comparability. The safest path is:

first compare embedding models on fixed chunks
then test new chunking on the chosen embedding model

Backfill throughput and consistency For large corpora, re-embedding can take days. Plan for:

resumable jobs
idempotent writes
partial index health checks
lag dashboards by repository/source
validation of metadata parity between old and new indexes

Shadow reads For a sample of live queries, retrieve from both indexes but only use index A for the actual answer. Log:

overlap@k between old and new candidate sets
rank-biased overlap
source/version changes
gold presence for labeled subsets
latency by index

This gives you real-world distribution coverage beyond your curated eval set.

Score distribution shifts: where migrations quietly break

One of the most common production incidents after embedding migration is a silent retrieval policy failure caused by score shift.

Examples:

A hard minimum similarity threshold drops too many results because the new model’s cosine scores cluster lower.
A fallback path triggers too often because “confidence gap” heuristics no longer hold.
Hybrid weighted blending over-amplifies BM25 because vector scores are on a different practical range.

Treat score semantics as model-local. After migration, recalibrate:

Similarity thresholds Re-estimate thresholds using labeled queries. Plot precision/recall curves under the new model. Do not port thresholds directly.
Candidate depth If the new embedding model is better at recall but slightly noisier in top ranks, retrieve a deeper candidate pool and let the reranker sort it out.
Fusion weights Re-tune lexical/semantic weighting or reciprocal rank fusion parameters. In many stacks, a modest change here matters more than the raw embedding upgrade.
Deduplication rules Better semantic recall can increase sibling-chunk concentration from a single document. Apply document-level caps or diversity constraints before generation.
Fallback logic If you route low-confidence cases to keyword search, web search, or escalation, re-measure confidence signals under the new retriever.

A practical observability pattern is to maintain score histograms and percentile dashboards by query segment before and after migration:

top1 vector score distribution
top1-top2 gap distribution
number of retrieved chunks above threshold
lexical/semantic contribution ratio
duplicate chunk rate

These plots often reveal migration risk faster than aggregate metrics.

Chunk re-embedding strategy and when chunking must change

Teams often ask whether they can reuse existing chunk boundaries when upgrading embeddings. Usually yes for the first migration stage, and that is my recommendation. Keep chunks constant so you can isolate the effect of the embedding model.

However, stronger embedding models sometimes expose chunking weaknesses:

If chunks are too large, semantic retrieval finds the right document but not the answer-bearing span.
If chunks are too small, context becomes fragmented and rerankers overvalue local term overlap.
Structured documents with headers, tables, and bullet hierarchies may need layout-aware chunking.

A safe strategy:

Stage 1: re-embed existing chunks with the new model. Stage 2: if retrieval still misses answer spans, test revised chunking on a smaller subset. Stage 3: only then consider a full rechunk + re-embed migration.

Signals that chunking, not embeddings, is the real problem:

high document-level recall but poor chunk-level recall
repeated retrieval of intro sections instead of procedural steps
answer spans split across adjacent chunks that never co-occur in final context

If you rechunk, preserve lineage metadata:

old chunk ID
source document ID
chunker version
section heading path
token range

That makes debugging and rollback possible.

Hybrid search interactions: don’t tune embeddings in a vacuum

Most production RAG stacks are hybrid whether teams admit it or not. Even “semantic search” systems usually include metadata filters, lexical fallback, or exact-match boosting. This means embedding upgrades must be evaluated in context.

There are three common hybrid patterns:

Reciprocal rank fusion (RRF) Robust and simple. Less sensitive to score-scale mismatch because it operates on ranks. Good default when experimenting with embedding changes.
Weighted score blending More expressive, but brittle across model migrations because score distributions change. Requires retuning after embedding upgrades.
Query-adaptive routing Route identifier-heavy queries toward lexical, broad semantic questions toward vector search, and blended cases to both. Most performant when you have heterogeneous query types.

Production lesson: if embedding migrations repeatedly destabilize weighted blending, switch to rank-based fusion during the migration window. It reduces one dimension of uncertainty.

Also, track retrieval wins and losses by query class. In many enterprises:

code, ticket IDs, SKUs, policy numbers, and exact names favor lexical retrieval
paraphrased knowledge questions favor embeddings
cross-encoder rerankers shine when the candidate set contains both exact-match noise and semantically relevant passages

Latency and cost tradeoffs

This is where the “just add a stronger reranker” advice often runs into reality.

Embedding upgrades are mostly offline cost plus possibly larger online vector-search cost if dimensionality changes. Rerankers are online cost every query.

A rough way to think about it:

Embedding upgrade costs

One-time re-embedding of corpus
Dual-index storage during migration
Potential index rebuild / replication cost
Little or no incremental per-query model inference if queries already require embedding

Reranker upgrade costs

Per-query inference on top N candidates
Tail latency growth proportional to candidate count and model size
More infrastructure complexity if self-hosted
Tokenization and batching overhead

Cross-encoder rerankers usually provide better ranking precision than bi-encoder similarity, but the quality/latency curve is sensitive to candidate set size. If you rerank top 20 passages, you may stay within budget but miss recall gains. If you rerank top 100, quality can improve but latency and cost rise sharply.

Common production patterns:

Cheap retriever + strong reranker for high-value workflows with lower QPS
Strong embeddings + light reranker for broad enterprise assistants
Query-adaptive reranking: only invoke cross-encoder when retrieval confidence is low or ambiguity is high
Two-tier reranking: lightweight reranker on top 50, heavier reranker on top 10 for premium flows

One effective compromise is selective reranking. Trigger the cross-encoder only when signals indicate uncertainty:

low overlap between lexical and semantic candidates
small confidence gap among top candidates
high entropy across sources/documents
query class historically benefits from reranking

This reduces average cost while preserving gains where ranking matters most.

Evaluation strategy: separate retrieval evals from answer evals

To migrate safely, run both retrieval-level and answer-level evaluation, but do not collapse them into one score.

Retrieval-level evals should answer:

Did the right chunks enter the candidate set more often?
Did ranking improve among retrieved candidates?
Did freshness/version confusion improve?
Did query-class performance shift?

Answer-level evals should answer:

Did grounded answer correctness improve?
Did citation correctness improve?
Did hallucination or unsupported synthesis change?
Did latency and timeout rates change user experience?

A strong eval stack includes:

Offline labeled set A manually curated dataset of real queries with relevant chunks/documents and freshness constraints.

Weak labels from logs Use historical clicked citations, accepted answers, or human escalations as noisy relevance signals.

Counterfactual replay Run historical queries through candidate systems and compare retrieval and answer outputs under identical downstream prompts.

LLM-as-judge carefully constrained Useful for comparing final answers, but do not let it be your sole source of truth. Pair it with citation-grounding checks and human review.

Slice-based reporting Always break out results by query family, repository, language, freshness sensitivity, and traffic tier.

The most informative migration report I’ve seen is a four-panel view for each candidate stack:

retrieval recall/ndcg deltas
answer correctness/citation deltas
latency/cost deltas
top failure diff examples with annotations

That gives leadership a quality decision, not just a model preference.

Rollout guardrails

When you move from offline confidence to production rollout, define explicit guardrails before turning anything on.

Recommended guardrails:

Quality guardrails

No significant drop in recall@k on critical query slices
No increase in superseded-version citations
No increase in empty retrievals or low-context answers

Performance guardrails

p50/p95/p99 retrieval latency within budget
Reranker timeout rate below threshold
Index error rate and cache hit rate stable

Cost guardrails

Per-query inference cost within target envelope
Dual-index storage burn tracked and time-boxed
Reranker invocation rate bounded if selective reranking is used

Operational guardrails

Canary exposure by tenant or team
Kill switch to revert to old index/reranker
Shadow logging retained long enough to investigate regressions

Do not roll out only by aggregate traffic percentage if your tenants differ materially. Canary by high-value cohorts and known-sensitive workflows first.

Observability patterns that actually help debugging

The most useful retrieval observability is not “average similarity score” in a dashboard. It is the ability to reconstruct what the retriever believed and how that changed.

Log at query time:

normalized query text and query class
active retrieval configuration version
lexical candidates and ranks
semantic candidates and ranks
fused candidates and ranks
reranked candidates and scores
chunk IDs, document IDs, version metadata, source repository
retrieval stage latencies
whether answer cited retrieved chunks

Build dashboards for:

candidate set overlap old vs new system
gold chunk presence by slice
duplicate sibling chunk rate
stale-doc retrieval rate
lexical vs semantic contribution share
reranker lift over pre-rerank ranking
selective reranker trigger rate and payoff

And keep a “retrieval diff explorer” for incident response. For any query, engineers should be able to inspect:

old top-k vs new top-k
rank deltas
score deltas
source/version changes
downstream answer changes

This is invaluable when a stakeholder says, “The assistant got worse after Tuesday.”

Model/tool comparisons in practice

A battle-tested way to compare options is to think in terms of what each component is structurally good at.

Embedding model upgrades are strongest when:

your domain semantics are underrepresented in the current model
multilingual or code-mixed retrieval matters
long-tail recall is weak
you need better candidate generation across the whole corpus

Cross-encoder rerankers are strongest when:

candidate generation is acceptable but ordering is messy
queries require fine-grained relevance judgments
broad semantic candidates need exact contextual disambiguation
you care deeply about top-3/top-5 precision and citation quality

Hybrid retrieval tuning matters most when:

query types are heterogeneous
exact-match identifiers matter alongside semantic paraphrase
model upgrades changed one retrieval modality more than another

In practice, many enterprise assistants converge toward:

hybrid candidate generation
moderate candidate depth
cross-encoder reranking on selected or all candidate sets
metadata-aware freshness and version logic

The question is less “embeddings or reranker?” and more “where is the current bottleneck, and what is the cheapest safe intervention that removes it?”

A phased migration plan

If I were leading this migration for a production team, I would run it like this:

Phase 0: Baseline and failure taxonomy

Assemble eval set from production logs and stakeholder complaints.
Label failures as recall, ranking, freshness, or chunking issues.
Measure baseline retrieval and answer metrics by slice.

Phase 1: Independent component experiments

Test new embedding model on fixed chunks, no other changes.
Test stronger reranker on existing candidates.
Estimate oracle reranker headroom.
Compare quality gain per unit of latency/cost.

Phase 2: Decide migration order

If recall deficit dominates, choose embedding-first.
If ranking deficit dominates, choose reranker-first.
If both, sequence them and define success criteria for each stage.

Phase 3: Embedding migration mechanics, if chosen

Backfill shadow index with stable chunk IDs.
Run shadow reads on live traffic.
Recalibrate thresholds and hybrid fusion.
Canary by tenant/query slice.

Phase 4: Reranker rollout, if chosen

Start with candidate depth tuned from offline evals.
Consider selective reranking if cost is a concern.
Monitor reranker lift, timeout rate, and p95 latency.

Phase 5: Post-migration cleanup

Retire old index only after observability confirms stability.
Remove obsolete thresholds and migration code paths.
Refresh eval set with newly observed failure modes.

Takeaways

Retrieval quality gains in RAG do not come from picking the “best model” in the abstract. They come from correctly identifying whether your bottleneck is candidate recall, candidate ranking, content representation, or retrieval policy—and then upgrading the right layer with the right safeguards.

If the correct chunks are already in the candidate set, a stronger cross-encoder reranker is often the safest, fastest quality lever. If the correct chunks are missing altogether, an embedding upgrade is the more fundamental fix. If both are weak, stage the changes rather than flipping the entire retrieval stack at once.

The migration itself matters as much as the model choice. Dual-index backfills, shadow reads, offline retrieval diffing, score recalibration, hybrid retuning, and canary guardrails are what keep “quality improvement” from turning into a production regression.

Most importantly, evaluate retrieval as retrieval. Answer quality matters, but generation can hide retrieval problems until they become painful in edge cases and stakeholder trust erosion. Measure what entered the candidate set, what got reranked, what changed by query slice, and how freshness/version behavior shifted.

That is how you improve relevance without destabilizing production RAG: not with a benchmark-driven leap of faith, but with a staged, observable migration where every gain can be explained and every rollback can be executed quickly.