Embeddings & Vector Search
How meaning becomes geometry. Drag a query around a 2D vector space and watch cosine similarity rank the neighbors.
An embedding is a list of numbers — a vector — that captures the meaning of a piece of text. The crucial property: texts with similar meaning get vectors that point in similar directions. This turns the fuzzy question “are these two things related?” into the precise, computable question “what is the angle between these two vectors?”
Real embeddings live in hundreds or thousands of dimensions, which no one can picture. But the intuition survives perfectly in two dimensions. Drag the query point around and watch which documents it points toward — the ranking is pure geometry.
Notice the clusters. Words about animals sit near each other because their meanings are related, so their vectors point the same way. Drag the query into the “code” cluster and python (language), compiler, and variable shoot to the top — not because of shared letters, but because of shared meaning. That’s the leap beyond keyword search: an embedding query for “feline” would rank “cat” highly even though they share no characters.
Cosine similarity: the ranking function
The standard way to compare embeddings is cosine similarity — the cosine of the angle between two vectors. It ranges from 1 (identical direction, i.e. same meaning) through 0 (unrelated) to −1 (opposite). Because it measures direction and ignores magnitude, a short document and a long one about the same topic still score as similar.
import numpy as np
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))From similarity to a vector database
Computing cosine similarity against millions of documents one by one is O(n) per query — too slow at scale. A vector database (Pinecone, Weaviate, pgvector, FAISS, and friends) solves this with an approximate nearest neighbor (ANN) index. It trades a tiny bit of accuracy for a massive speedup, answering “give me the 10 closest vectors” in milliseconds even over billions of entries.
The typical workflow:
- Chunk your documents into passages.
- Embed each chunk with an embedding model and store the vectors, with the original text as metadata.
- At query time, embed the query and ask the index for its nearest neighbors.
The rules that keep it correct
- Use the same model to embed documents and queries. Vectors from different models live in incompatible spaces and can’t be compared.
- Chunk thoughtfully. Too large and a chunk mixes many topics, blurring its vector; too small and it loses the context needed to be meaningful. A few hundred tokens with slight overlap is a common starting point.
- Similarity ≠ correctness. The nearest neighbor is the most semantically similar passage, not necessarily the one that answers the question. That gap is exactly what the next lesson, retrieval-augmented generation, is built to manage.
Why this is the backbone of applied AI
Embeddings are what let you connect a language model to your data. Search, recommendations, deduplication, clustering, classification, and — most importantly — retrieval for RAG all rest on this one idea: meaning becomes geometry, and geometry is something a computer can rank.
Pattern recognition
Variations
Worked problems
Implement cosine similarity
Write cosine similarity using only the Python standard library. Return 0.0 if either vector has zero length.
Approach. Cosine is dot product divided by both vector norms. Guard against zero vectors because division by zero is not a useful similarity score.
Show solution
import math
def cosine_similarity(a, b):
if len(a) != len(b):
raise ValueError("vectors must have the same dimension")
dot = sum(x * y for x, y in zip(a, b))
norm_a = math.sqrt(sum(x * x for x in a))
norm_b = math.sqrt(sum(y * y for y in b))
if norm_a == 0 or norm_b == 0:
return 0.0
return dot / (norm_a * norm_b)
print(round(cosine_similarity([1, 1, 0], [1, 0, 0]), 3))The example prints about 0.707: the vectors point partly in the same direction. Many vector stores normalize embeddings at write time so cosine can be computed as a dot product.
Complexity. O(d) time and O(1) extra space for dimension d.
Choose a chunking strategy
You are indexing a 200-page employee handbook. Some policies span several paragraphs, while users usually ask one specific policy question. Pick an initial chunk size and overlap, and explain what you would measure.
Approach. Chunks should be small enough to retrieve a focused answer but large enough to preserve the policy context. Overlap protects facts that cross boundaries.
Show solution
Start around 400–700 tokens per chunk with 50–100 tokens of overlap. Split on semantic boundaries first: headings, sections, paragraphs, then token count. Store metadata for handbook version, section title, page, and URL.
Measure:
- whether known questions retrieve the supporting section in top
k, - answer citation accuracy,
- empty-result rate,
- average prompt tokens after retrieval,
- cases where the answer was split across chunks.
def sliding_chunks(tokens, size=600, overlap=80):
if overlap >= size:
raise ValueError("overlap must be smaller than size")
step = size - overlap
for start in range(0, len(tokens), step):
chunk = tokens[start:start + size]
if chunk:
yield chunkComplexity. O(n) chunks over n tokens. Chunking is an eval problem: tune it against retrieval misses, not aesthetics.
Reason about dimensionality
A team stores 10 million embeddings with 1,536 floating-point dimensions, using 4 bytes per float. Estimate raw vector storage. Then explain why reducing dimensions is not automatically safe.
Approach. Multiply vectors by dimensions by bytes. Then separate storage math from retrieval quality: dimensions carry information learned by the embedding model.
Show solution
def raw_vector_gb(num_vectors, dimensions, bytes_per_value=4):
total_bytes = num_vectors * dimensions * bytes_per_value
return total_bytes / (1024 ** 3)
print(round(raw_vector_gb(10_000_000, 1536), 1))Raw vectors take about 57.2 GiB before index overhead, metadata, replication, and write-ahead logs. HNSW graph edges can add substantial memory.
Reducing dimensionality can save memory, but it may collapse distinctions the model uses for ranking. Evaluate recall@k and downstream answer quality after any projection, truncation, quantization, or model change.
Complexity. O(1) for the estimate. The production cost is the evaluation run that proves recall did not regress.