Cosine Similarity vs Dot Product Explained: When They Rank the Same and When They Don't

A retrieval service starts returning the same handful of long documents for almost every query the day after an embedding model swap. Recall@10 on the golden set drops, the ANN index never logged an error, and p99 latency is unchanged. The usual cause is the metric, not the index: the collection was built for inner product, the new model does not emit unit-length vectors, and the longest vectors in the corpus now win every search. This is cosine similarity vs dot product explained from the operational side: the formulas, the single condition under which they are the same function, and the ways a pipeline quietly violates that condition.

The formulas, and the one case where they agree

Dot product is the sum of elementwise products: a . b = sum(a_i * b_i). Cosine similarity divides that by both magnitudes: cos(a, b) = a . b / (||a|| * ||b||). Rearranged, a . b = ||a|| * ||b|| * cos(theta), so dot product is cosine scaled by the two vector lengths. If both vectors have L2 norm 1, the denominator is 1 and the two numbers are identical.

PyTorch’s implementation is the formula plus a guard: torch.nn.functional.cosine_similarity computes x1 . x2 / (max(||x1||, eps) * max(||x2||, eps)) with eps=1e-8 by default, so a zero vector returns 0 rather than NaN.

The ranking consequence is what matters in production. For a fixed query q, ordering candidates d_i by q . d_i does not depend on the query’s norm; it is a constant factor across every candidate. The Faiss wiki states this directly: with METRIC_INNER_PRODUCT, “the norm of the query vectors does not affect the ranking of results.” What changes the order is the norm of each stored vector. If every stored vector has norm 1, dot-product ranking and cosine ranking are the same list. If they do not, dot product ranks by cos(theta) * ||d_i||, and long vectors get a bonus that has nothing to do with the query.

Euclidean distance collapses into the same picture. The Faiss wiki gives ||x - y||^2 = 2 - 2 * <x, y> for normalized vectors, so L2 nearest neighbour, maximum inner product and cosine all produce the same order once everything is length 1. That is why most engines implement cosine as normalize-then-dot. Qdrant says so in its collection docs: “Cosine similarity is implemented as dot-product over normalized vectors. Vectors are automatically normalized during upload.” hnswlib exposes the seam in its formulas: ip is 1.0 - sum(Ai*Bi) and cosine is the same expression divided by sqrt(sum(Ai*Ai) * sum(Bi*Bi)). OpenAI’s embeddings guide notes its vectors are normalized to length 1, so “cosine similarity can be computed slightly faster using just a dot product.”

The metric that matters

The question is not “cosine or dot.” It is two questions: are the stored norms constant, and does the score function match what the embedding model was trained with. Pinecone’s index docs put the second one plainly: “for the most accurate results, choose the similarity metric used to train the embedding model for your vectors.” Sentence-Transformers models carry this as similarity_fn_name, which accepts cosine, dot, euclidean or manhattan and defaults to cosine when unset.

The first question has a measurable answer. Track the L2 norm distribution of vectors per ingest batch: p50, p99, and the ratio max_norm / min_norm. When that ratio is close to 1, cosine and dot product rank identically and the choice is a performance detail. When it is not, the two metrics disagree, and the disagreement grows with the spread. This beats watching a score histogram because dot-product scores are unbounded and shift with every model version, while a norm ratio is a property of the stored data you can alert on.

Norms are not always noise. In a two-tower recommender trained on inner product, item norm typically encodes something like popularity, and maximum inner product search is the intended objective; Faiss documents METRIC_INNER_PRODUCT as the recommendation-system case. Cosine throws the magnitude away. That is a feature for text retrieval with a model trained on cosine, and a silent bug for a model trained on dot product.

Wiring it up

The audit is a few lines of PyTorch: confirm the norm spread, then measure how far top-k under each metric drifts apart on a sample of real queries.

import torch
import torch.nn.functional as F

def metric_audit(db: torch.Tensor, queries: torch.Tensor, k: int = 10) -> dict:
    norms = db.norm(dim=1)
    dot = queries @ db.T
    cos = F.normalize(queries, dim=1) @ F.normalize(db, dim=1).T
    top_dot = dot.topk(k, dim=1).indices
    top_cos = cos.topk(k, dim=1).indices
    overlap = torch.tensor([
        len(set(a.tolist()) & set(b.tolist())) / k
        for a, b in zip(top_dot, top_cos)
    ])
    return {
        "norm_p50": norms.median().item(),
        "norm_p99": norms.quantile(0.99).item(),
        "norm_ratio": (norms.max() / norms.min()).item(),
        "topk_overlap_mean": overlap.mean().item(),
    }

# FAISS: cosine is "normalize both sides, then inner product"
import faiss
xb = db.numpy().astype("float32")
faiss.normalize_L2(xb)
index = faiss.IndexFlatIP(xb.shape[1])
index.add(xb)

In pgvector the same choice is an operator plus an index class, and the sign convention is the trap. <=> is cosine distance; <#> is the negated inner product, because Postgres only supports ascending-order index scans; the matching operator classes are vector_cosine_ops and vector_ip_ops. The README also recommends inner product as the fastest exact-search option for length-1 vectors.

CREATE INDEX ON chunks USING hnsw (embedding vector_ip_ops);

-- <#> is NEGATIVE, so ascending ORDER BY returns the best match first
SELECT id, (embedding <#> $1) * -1 AS score
FROM chunks
ORDER BY embedding <#> $1
LIMIT 10;

Weaviate follows the same convention and returns -dot(a, b) as its dot distance, so a smaller distance still means a closer match.

What you’ll see

Healthy: norm p50 and p99 sit on top of each other, norm_ratio is close to 1, and top-k overlap between cosine and dot product on sampled queries is near 1.0. The cosine histogram is bounded to [-1, 1]; the dot-product histogram is unbounded and only means something relative to its own model version.

Unhealthy has a signature. Norm p50 steps on the day of a model, tokenizer or chunker change that altered input lengths, and top-k overlap drops with it. A small set of “hub” documents starts appearing for unrelated queries; those are the high-norm vectors. The norm series belongs next to whatever embedding drift monitoring you already run; sentryml’s monitoring taxonomy covers where input-distribution metrics fit. Metric mismatch is also one of the first items to rule out in the low-recall checklist on this site.

Caveats

• Normalizing at index time but not at query time does not break ranking, since the query norm is a constant factor. It does break any threshold: the score is no longer a cosine, so a cutoff like score > 0.8 silently stops meaning what it meant. Normalize both sides or neither.
• Inner product is not a metric. hnswlib’s README says it outright: “an element can be closer to some other element than to itself.” Index structures that assume a metric can misbehave on raw inner product; the LSH family needs an asymmetric transform, the contribution of Shrivastava and Li (2014), to reduce maximum inner product search to near-neighbour search.
• Quantization reintroduces norm spread. A vector reconstructed from PQ or scalar-quantized codes is no longer exactly unit length, so dot product over quantized storage approximates cosine rather than equalling it.
• Cosine is not calibrated. Steck, Ekanadham and Kallus (2024) show that for embeddings from regularized linear models, cosine similarity “can yield arbitrary and therefore meaningless ‘similarities’.” Do not read 0.83 as “83% similar” across models or versions.
• Cardinality. Emit the norm audit as per-batch quantile gauges, never a per-vector series. A burst of exact-zero cosine scores usually means empty chunks reaching the encoder, since PyTorch’s eps guard maps a zero vector to 0.

Related across the network

• LangChain RAG Pipeline: Setup to First Answer — langchainguide.com
• RAG Chunking Strategy: Picking Chunk Size and Overlap — ragstackguide.com
• RAG Pipeline Architecture: Components and Build Order — ragstackguide.com
• LangChain Building Blocks: Chains, Tools and Agent Control — langchainguide.com
• LangChain vs LangGraph vs LlamaIndex Compared — langchainguide.com