What Is Reranking in Vector Search? A Practical Guide

Your RAG endpoint’s p99 latency is within budget, yet it still returns the wrong passage. Before changing the generator, ask what is reranking in vector search, and whether the failure is candidate selection or candidate ordering.

What is reranking in vector search?

Reranking is a second pass that scores retrieved candidates and changes their order before returning search results or assembling RAG context. A common implementation uses a cross-encoder: a transformer reads the query and candidate text together and produces a relevance score. A retrieval embedding model encodes them independently. Sentence Transformers explains this distinction.

The request path is straightforward:

1. Retrieve: the vector index returns a shortlist of size N using embedding similarity.
2. Rerank: the cross-encoder scores each query-passage pair in that shortlist.
3. Select: return the highest-scoring k passages to the user or generator.

For an illustrative query about resetting an expired credential, retrieval might find general authentication documentation. Reranking can promote the specific recovery procedure if it is present. Scoring only a shortlist keeps the more expensive joint text processing bounded.

Hybrid search and neural reranking do different jobs. Hybrid search combines retrieval signals, such as sparse keyword and dense vector results. Reciprocal rank fusion combines their positions; a cross-encoder evaluates query-text pairs. They can sit in the same pipeline. Qdrant documents fusion and multistage queries.

The hard limit follows from that pipeline: reranking cannot recover a passage absent from its candidates. Diagnose low vector search recall before adding another model to a retrieval failure.

The metric that matters

Use nDCG@k on a held-out golden set, with end-to-end p99 latency as a deployment constraint. With nonnegative relevance grades, the linear-gain definition is:

DCG@k = sum(rel_i / log2(i + 1), i = 1..k)

nDCG@k = DCG@k / IDCG@k

Here, rel_i is judged relevance at rank i; IDCG is the best possible DCG for that query’s judgments. Average across queries, reporting queries with no judged relevant documents separately. This rewards useful passages appearing early, unlike average cosine similarity, which measures embedding agreement. scikit-learn documents this nDCG formulation.

Also track candidate recall@N: relevant documents retrieved divided by all judged relevant documents. Keep candidates fixed when comparing rerankers, then evaluate the complete retrieval pipeline separately. Preserve the full judgment pool for the ideal ranking; evaluating only retrieved positives hides retrieval misses.

Wiring it up

This example uses Sentence Transformers with its PyTorch backend and the documented MS MARCO MiniLM cross-encoder. Load it once per worker. Pass already authorized candidate records containing id and text; choose k and batch size in service configuration.

The CrossEncoder API returns sorted candidate indices. A Prometheus histogram records the scoring duration:

from prometheus_client import Histogram
from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "cross-encoder/ms-marco-MiniLM-L6-v2", backend="torch"
)
duration = Histogram(
    "rerank_duration_seconds", "Candidate scoring wall time"
)

def rerank(query, candidates, k, batch_size):
    if k <= 0 or not candidates:
        return []
    with duration.time():
        ranked = model.rank(
            query,
            [item["text"] for item in candidates],
            top_k=k,
            batch_size=batch_size,
            show_progress_bar=False,
        )
    return [
        {**candidates[row["corpus_id"]],
         "rerank_score": float(row["score"])}
        for row in ranked
    ]

The cost knob is N: top_k limits output, while every supplied candidate is scored. Expose the registry through the service’s Prometheus metrics endpoint. This timer excludes retrieval, worker queueing, and generation; instrument those separately. Pin the model revision in the deployment artifact.

For this classic histogram, an illustrative p99 query is histogram_quantile(0.99, sum by (le) (rate(rerank_duration_seconds_bucket[5m]))). Aggregate buckets across workers before computing the quantile. Tune bucket boundaries around your latency budget; Prometheus explains quantile approximation and aggregation.

What you’ll see

Compare baseline and reranked nDCG on identical queries. Alongside that chart, plot p50/p95/p99 latency, QPS, errors, and queue depth. This applies the Google SRE golden signals to retrieval:

Chart pattern	Operational interpretation
nDCG rises; p99 stays within budget	Candidate ordering improved at acceptable serving cost.
Candidate recall falls; nDCG falls	Inspect retrieval, filters, and corpus changes first.
Candidate recall holds; nDCG falls	Inspect the reranker, tokenizer, and text preparation.
p50 holds; p99 and queue depth rise	Investigate saturation under concurrent load.

For RAG, measure application time-to-first-token (TTFT) from request arrival, including retrieval and reranking. Generator tokens/sec alone cannot explain a delay before generation starts. Use a canary deploy with rollback thresholds for relevance and latency.

Caveats

• Truncation and batching: long inputs can be truncated, removing the answer-bearing text. Increasing batch size changes memory demand and throughput. Check input lengths and load behavior; the API documents truncation and batch-size tuning.
• False alarms: a changed query mix can shift score distributions without worsening relevance. Segment the evaluation and inspect labels before paging. SentryML’s monitoring metrics taxonomy separates drift signals from measured degradation.
• Sampling and leakage: sample shadow traffic to bound duplicate inference cost. Keep tuning queries separate from the final regression test, and avoid injecting known relevant documents into production-style candidate sets.
• Cardinality: query text, document IDs, and request IDs do not belong in metric labels. Keep them in controlled traces or evaluation artifacts. Prometheus details the cost of each additional label set.