Product Quantization Explained for Vector Search

Your retrieval service’s p99 latency is climbing, and the vector index is squeezing RAM. Here is product quantization explained for vector search: replace full embedding vectors with compact, learned codes, then search using approximate distances. That trades storage for ranking accuracy. Start with memory pressure and retrieval quality, rather than assuming compression will fix every latency spike.

How product quantization compresses vectors

Product quantization (PQ) splits a vector into subvectors and quantizes each separately. Training learns a codebook of representative centroids for each subspace. Encoding replaces each subvector with the index of its nearest centroid. The stored code is the sequence of those indices; reconstruction concatenates the selected centroids. “Product” refers to the Cartesian product of the subspace codebooks. This is lossy compression: different vectors can receive the same code. Original PQ paper.

For Faiss’s usual equal split, dimension d must be divisible by the number of subquantizers M. With b bits per subquantizer, each codebook has 2^b entries. The documented storage formulas are:

Representation	Bytes per vector
Float32 flat vectors	4 * d
PQ codes	ceil(M * b / 8)
IVF-PQ codes plus stored IDs	ceil(M * b / 8) + 8

These are vector-storage formulas, excluding shared codebooks and other index overhead. Increasing M or b lengthens each code. Scalar quantization encodes individual components; PQ encodes groups of components. Faiss index reference.

Search usually uses asymmetric distance computation: keep the query uncompressed and compare it with encoded database vectors. For squared L2 distance, D(q, code) = sum_j ||q_j - centroid_j[code_j]||². Precompute each query-subvector-to-centroid distance, then score candidates through table lookups and additions. Faiss implementation notes.

Where IVF, HNSW and OPQ fit

PQ controls representation. IVF controls which candidates get examined: it partitions vectors into inverted lists and searches nprobe lists. Faiss IndexIVFPQ normally encodes residuals relative to the assigned coarse centroid. IndexPQ scans every code, yet still returns approximate distances. HNSW uses graph traversal and can also use PQ storage. Faiss index reference.

Optimized product quantization (OPQ) learns a rotation before splitting the vectors, improving how information is distributed across subspaces. It is worth evaluating when plain PQ loses too much recall at the available code size. Faiss library paper.

The metric that matters

Use recall@k against exact search, subject to a p99 latency and memory budget. Define it explicitly:

recall@k = mean_q(|ANN_k(q) intersect Exact_k(q)| / k)

This is top-k set overlap. Some publications use recall@k for whether the single nearest neighbor appears anywhere in the returned set. Lower reconstruction error is useful, but overlap directly measures which neighbors compression loses. Faiss accuracy metrics.

Freeze the corpus, embedding model, distance metric, filters and eval queries. Keep a separate golden set of relevance judgments for RAG: reproducing exact vector neighbors does not establish that those neighbors answer the question. Treat that as a separate regression test. For dataset and tie handling, see the recall benchmark guide.

Wiring it up

With faiss-cpu, NumPy and MLflow installed, supply representative training vectors, a corpus, and held-out queries as finite float32-compatible matrices with matching dimensions. This L2 example uses the Faiss tutorial’s example parameters, which require workload-specific tuning. Configure your MLflow tracking destination before running it; MLflow’s tracking API records the parameters and retrieval metric.

import faiss
import mlflow
import numpy as np

def load_vectors(path):
    return np.ascontiguousarray(np.load(path), dtype="float32")

train = load_vectors("training.npy")
base = load_vectors("corpus.npy")
queries = load_vectors("queries.npy")
d = base.shape[1]
nlist, m, nbits, nprobe, k = 100, 8, 8, 10, 4
assert d % m == 0
assert train.shape[1] == queries.shape[1] == d
assert len(base) >= k and len(queries) > 0

exact = faiss.IndexFlatL2(d)
exact.add(base)
_, truth = exact.search(queries, k)

index = faiss.IndexIVFPQ(
    faiss.IndexFlatL2(d), d, nlist, m, nbits
)
index.train(train)
index.add(base)
index.nprobe = nprobe
_, found = index.search(queries, k)
recall = np.mean([
    len(set(a[a >= 0]) & set(t)) / k
    for a, t in zip(found, truth)
])

mlflow.set_experiment("pq-retrieval")
with mlflow.start_run():
    mlflow.log_params(dict(
        d=d, nlist=nlist, m=m, nbits=nbits, nprobe=nprobe, k=k
    ))
    mlflow.log_metric("recall_at_k", float(recall))

The exact baseline holds full vectors, so this process’s RSS is unsuitable for measuring production PQ memory. Measure the serving index separately. Replay realistic QPS and batch sizes through the service to measure p50/p95/p99 latency; mean batch time hides slow requests. Google SRE monitoring guidance.

What you’ll see

Plot recall against p99 latency, with memory attached to each configuration. Good looks like meeting the recall floor while reducing memory and staying inside the latency budget. Bad looks like a recall plateau as nprobe increases.

To investigate, set nprobe = nlist offline: remaining misses isolate PQ distortion. Compare with IndexIVFFlat to isolate losses from list selection. Check tied distances before declaring a regression. These diagnostics follow the Faiss FAQ.

When compression is the limit, evaluate larger codes or OPQ. Another option is retrieving more candidates and reranking with original vectors. Budget their storage and fetch latency; reranking cannot recover a candidate absent from the shortlist. Faiss reranking notes.

Caveats

• Distribution changes: codebooks need representative training data. Resolve insufficient-training-sample warnings before evaluating quality. An embedding-model change or a different corpus distribution can require a new index build. Validate a shadow deploy before a canary deploy. Faiss training guidance.
• False alarms and evaluation leakage: query-mix changes can move aggregate recall. Keep fixed queries alongside fresh samples, and reserve untouched queries for final acceptance after tuning.
• Sampling cost: exact search scans the corpus. Run sampled quality checks offline, away from the serving latency budget.
• Cardinality: keep query IDs and document IDs out of Prometheus labels. Each label combination creates another time series. Prometheus instrumentation guidance.

For ongoing alert design, use the monitoring and drift coverage at sister publication SentryML.

Related across the network

• Hugging Face Security Incidents and Supply Chain Risks — ai-alert.org
• Monitoring LLM Outputs in Production: Anomalies and Drift — aidefense.dev
• Best AI Monitoring Tools 2026: LLM Observability Compared — aiincidents.org