How to Choose a Vector Database: Fix the Recall Target, Measure on Your Data, Then Shop

The question of how to choose a vector database usually arrives late. The RAG prototype works on 50,000 chunks in a notebook, product wants it on 40 million, and the retrieval step’s p99 is about to become the product’s p99. Vendor pages answer with feature grids, and a grid will not tell you what happens to recall when a tenant filter keeps 0.3% of rows, or how many gigabytes 40M vectors at 1536 dimensions need before quantization. What follows is a decision procedure: fix the constraints, name the one metric that decides it, measure it on your own corpus, then compare products.

Five constraints that make the decision

1. Corpus size and dimension. N times d times 4 bytes is the float32 floor; 40M by 1536 is roughly 246 GB before graph overhead. Under a million vectors the choice is operational, not algorithmic. Engines separate at 10M and above; memory sizing walks the arithmetic.
2. Filter selectivity. What fraction of rows does a typical filter keep? 30% is easy. A tenant_id keeping 0.1% is where naive HNSW breaks and where engines differ most.
3. Write pattern. Bulk load then read-mostly, or continuous upserts and deletes. NeurIPS 2023 added a streaming track (30M points, insert:delete:search at 4:4:1, one hour, 8 GB DRAM) because churn is a different problem from a static index.
4. The latency SLO, as a percentile. The SRE book is blunt: metrics are distributions, not averages, and targets come from what users need. Write the p99 bound down before reading a benchmark.
5. Who is on call. A Postgres team already carrying pages, a new stateful service, or a managed API with its own rate limits. This eliminates more candidates than any chart.

The metric that matters: QPS at target recall, on your data, with your filters

Recall@k for one query is the fraction of the exact top-k the approximate index returned:

recall@k = |ANN_topk ∩ exact_topk| / k

averaged over a golden query set. The selection metric is the throughput an engine holds at your target (0.95 recall@10 is a common bar), on your corpus with your filter mix, divided by monthly cost.

Why not latency alone: any engine hits any latency target if you let recall fall, because ef_search, nprobe and search_list all buy speed with recall. ANN-Benchmarks, the independent benchmark maintained by Aumüller, Bernhardsson and Faithfull, plots exactly this curve for more than 35 implementations, and their paper reports that very different algorithms land on comparable tradeoffs. The NeurIPS 2023 filtered track scores entries by QPS at 90% recall for the same reason. The catch: public curves are on glove-100 and sift-128, not your embeddings under your filters. Reproduce the curve locally.

The architectures and where each one breaks

Postgres extension. pgvector v0.8.6 ships HNSW (m 16, ef_construction 64, hnsw.ef_search 40 by default) and IVFFlat; indexed dimensions cap at 2,000 for vector and 4,000 for halfvec. Filters apply after the index scan, so a 10% selective predicate at ef_search 40 yields about 4 rows unless hnsw.iterative_scan is on. The graph must fit in shared buffers. See HNSW vs IVF.

Dedicated open-source engine. Qdrant defaults to m 16 and ef_construct 100, adds extra HNSW edges per indexed payload field so strict filters do not disconnect the graph, and drops to a full scan under full_scan_threshold (default 10,000 KB). Milvus has the widest menu: FLAT, IVF_FLAT, IVF_SQ8 (documented at 70 to 75% less memory), IVF_PQ, HNSW with SQ/PQ/PRQ variants, SCANN and DiskANN. Weaviate runs HNSW by default, a flat index for small or per-tenant collections, a dynamic index that flips to HNSW at 10,000 objects, and since v1.36 an HFresh index that keeps most data on disk.

Managed serverless. Pinecone exposes no index knobs; the limits are the design: 20,000 dimensions, 40 KB filterable metadata per record, top_k up to 10,000, and 100 requests per second per namespace for each of query, upsert, update and delete on every plan. The pgvector vs Pinecone comparison works the cost model.

Search engine with kNN. Elasticsearch keeps an HNSW graph per segment, applies filters during the graph search so exactly k hits match, and switches to brute force when the filtered set is small. Quantization is int8_hnsw, int4 (1.5x to 2x oversampling suggested), bbq (3x to 5x) and bfloat16.

In-process library. hnswlib and FAISS fit a batch reranker or a single node with no filters and no durability requirement. They are not a database.

Memory is decided by quantization, not the engine

Scalar quantization to int8 is 4x everywhere; Qdrant documents usually under 1% error on high-dimensional vectors. Binary quantization is 32x, with Qdrant claiming up to 40x speedup and Weaviate naming Cohere v3 and OpenAI ada-002 as models it suits while advising a check against your own. Product quantization reaches 64x on Qdrant and 24x on Weaviate at the cost of a training pass and slower distance math; Weaviate’s 8-bit rotational quantization is 4x at 98 to 99% recall with no training. All of them over-fetch from the compressed index and rescore against full-precision vectors. If quantized vectors still will not fit, the NeurIPS 2021 billion-scale challenge used DiskANN as the baseline for its SSD track, and its leaderboards are the closest thing to an independent benchmark at that scale.

Wiring it up

Run every candidate through one harness on a sample of your corpus. search(q, k, filt) wraps the engine client; masks are boolean arrays over corpus rows so ground truth honours the same filter.

import time
import numpy as np

def exact_topk(corpus, q, k, mask=None):
    scores = corpus @ q                      # cosine on L2-normalised rows
    if mask is not None:
        scores = np.where(mask, scores, -np.inf)
    return set(np.argpartition(-scores, k)[:k].tolist())

def bench(search, corpus, queries, k=10, filters=None, masks=None):
    hits, lat = [], []
    for i, q in enumerate(queries):
        filt = filters[i] if filters is not None else None
        mask = masks[i] if masks is not None else None
        truth = exact_topk(corpus, q, k, mask)
        t0 = time.perf_counter()
        got = search(q, k, filt)
        lat.append(time.perf_counter() - t0)
        hits.append(len(truth & set(got)) / k)
    ms = np.array(lat) * 1000.0
    return {
        "recall_at_k": float(np.mean(hits)),
        "p50_ms": float(np.percentile(ms, 50)),
        "p99_ms": float(np.percentile(ms, 99)),
        "qps_single_client": len(queries) / float(np.sum(lat)),
    }

Sweep the search parameter and record the smallest value that clears the recall target; that is the setting you cost. Repeat at filter selectivities of 100%, 10%, 1% and 0.1%. Single-client QPS is a lower bound; concurrency needs multiple clients.

What you’ll see

Good: recall climbs steeply with the search parameter and flattens above 0.95, p99 grows slowly along the curve, and the 0.1% selectivity run sits within a small factor of the unfiltered one. Set the parameter at the knee.

Bad, in order of frequency: recall collapses below 0.5 at 1% selectivity (post-filtering with no iterative scan); p99 detaches from p50 by 20x once the index no longer fits in RAM; recall is fine but QPS is a tenth of the vendor’s chart because theirs was 100-dimensional glove and yours is 1536. Fixes are catalogued in low vector search recall.

Decision rule

• Under a few million vectors with Postgres already in production: pgvector, HNSW, stop here.
• Selective filters and a team willing to run a stateful service: Qdrant, Milvus, Weaviate or Elasticsearch, chosen by the 0.1% selectivity run.
• No platform team, bursty traffic: Pinecone, with 100 RPS per namespace in the capacity plan.
• Hundreds of millions of vectors on a bounded RAM budget: quantize first; if that fails, a DiskANN-family index.

Caveats

• Vendor benchmarks are vendor benchmarks. Qdrant’s compares five engines on an 8 vCPU, 32 GB server with a 25 GB memory cap and states its own bias. Treat every one the same way, including this site’s comparisons.
• Public datasets are not your distribution. The NeurIPS 2023 out-of-distribution track pairs text queries with image embeddings precisely because the distributions differ. Measure on your own query and document pairs.
• Golden-set staleness. Ground truth from last quarter’s embedding model is wrong for this quarter’s. Recompute exact top-k whenever the model changes.
• Cardinality. Payload indexes on high-cardinality fields cost RAM and build time; Qdrant’s docs say to create them before ingest, so fix the filter schema before the load.
• This is a point-in-time number. Recall against the golden set belongs in production monitoring; SentryML’s monitoring metrics taxonomy shows where it fits. Whatever the index returns becomes model input, so read AI Sec on indirect injection in RAG pipelines before go-live.

Related across the network

• Data Poisoning in RAG Systems: A 2026 Threat Briefing — ai-alert.org
• RAG Poisoning: How Retrieval-Augmented Systems Get Compromised — ai-alert.org
• Secure RAG Architecture Best Practices for Production LLM Systems — aidefense.dev
• LLM Benchmark Fidelity: Why MMLU Won’t Predict Production Quality — aisecbench.com
• How to Secure Vector Database Access in RAG Systems — aisecreviews.com