Cheat sheet · Data & storage
Vector search
The problem
An online store's help centre has 2 million article chunks, and customers ask in their own words: "can I get my money back for a broken kettle". Keyword search misses the answer when it uses other words. We add embeddings and cosine similarity, see why exact kNN is too slow, trade a little recall for milliseconds with HNSW, shrink memory with int8, handle filters, combine BM25 and vectors with reciprocal rank fusion, and feed the result to an LLM.
What you learn
- What an embedding is and how cosine similarity finds text with the same meaning but no shared words
- Why exact kNN costs N x d per query, and how HNSW trades recall, latency and memory
- Filters with approximate search, and hybrid BM25 + vector ranking with reciprocal rank fusion
- Why vectors from different model versions must never be compared, and how to re-embed safely
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_user(["Help chat"])
n_api["Search API"]
n_text[("Keyword index")]
n_indexer("Indexer")
n_docs[("Articles DB")]
n_model>"Embedding model"]
n_vdb[("Vector index")]
n_llm>"LLM"]
n_user -->|"call"| n_api
n_indexer --> n_docs
n_indexer --> n_text
n_api -->|"call"| n_model
n_api --> n_vdb
n_indexer --> n_model
n_indexer --> n_vdb
n_api --> n_text
n_api -->|"call"| n_llmMermaid: request sequence
sequenceDiagram
participant n_api as Search API
participant n_text as Keyword index
participant n_model as Embedding model
participant n_vdb as Vector index
participant n_indexer as Indexer
participant n_llm as LLM
Note over n_api,n_text: The customer and the article use different words
n_api->>n_text: q = #34;singan choynak uchun pulimni qaytarib olsam bo'ladimi#34;
Note over n_api,n_model: Turn meaning into 768 numbers
n_api->>n_model: POST /v1/embeddings HTTP/1.1
n_model-->>n_api: HTTP/1.1 200 OK
Note over n_api,n_model: Similar meaning, small angle
n_api->>n_model: cos(a, b) = (a . b) / (#124;a#124; #124;b#124;) normalized vectors: cos = a . b
Note over n_api,n_vdb: Compare the question with all 2 million vectors
n_api->>n_vdb: exact kNN 2 000 000 vectors x 768 dims, float32
Note over n_api,n_vdb: Walk a graph instead of reading everything
n_api->>n_vdb: index = HNSW(M=16, ef_construction=200) build: 38 min
Note over n_indexer,n_vdb: Store each number in 1 byte instead of 4
n_indexer->>n_vdb: vector storage per chunk, 768 dims
Note over n_api,n_vdb: The filter removes almost everything the graph found
n_api->>n_vdb: filter: lang = #34;uz#34; AND in_stock = true matches 8% of chunks
Note over n_api,n_vdb: Choose the plan by how much the filter keeps
n_api->>n_vdb: plan by how many chunks the filter keeps
Note over n_api,n_text: Vectors miss exact codes: bring BM25 back
n_api->>n_text: q = #34;KT-2041 choynak qopqog'i yopilmayapti#34;
Note over n_api,n_llm: Give the best 5 chunks to an LLM, with sources
n_api->>n_llm: retrieve the top 5 chunks (hybrid, 14 ms) -#62; 2 100 tokens of contextD2
direction: right
n_user: "Help chat" {
shape: rectangle
}
n_api: "Search API" {
shape: rectangle
}
n_text: "Keyword index" {
shape: cylinder
}
n_indexer: "Indexer" {
shape: rectangle
style.multiple: true
}
n_docs: "Articles DB" {
shape: cylinder
}
n_model: "Embedding model" {
shape: cloud
}
n_vdb: "Vector index" {
shape: cylinder
}
n_llm: "LLM" {
shape: cloud
}
n_user -> n_api: "call"
n_indexer -> n_docs
n_indexer -> n_text
n_api -> n_model: "call"
n_api -> n_vdb
n_indexer -> n_model
n_indexer -> n_vdb
n_api -> n_text
n_api -> n_llm: "call"PlantUML: request sequence
@startuml
participant "Search API" as n_api
database "Keyword index" as n_text
boundary "Embedding model" as n_model
database "Vector index" as n_vdb
participant "Indexer" as n_indexer
boundary "LLM" as n_llm
note over n_api, n_text : The customer and the article use different words
n_api -> n_text : q = "singan choynak uchun pulimni qaytarib olsam bo'ladimi"
note over n_api, n_model : Turn meaning into 768 numbers
n_api -> n_model : POST /v1/embeddings HTTP/1.1
n_model --> n_api : HTTP/1.1 200 OK
note over n_api, n_model : Similar meaning, small angle
n_api -> n_model : cos(a, b) = (a . b) / (|a| |b|) normalized vectors: cos = a . b
note over n_api, n_vdb : Compare the question with all 2 million vectors
n_api -> n_vdb : exact kNN 2 000 000 vectors x 768 dims, float32
note over n_api, n_vdb : Walk a graph instead of reading everything
n_api -> n_vdb : index = HNSW(M=16, ef_construction=200) build: 38 min
note over n_indexer, n_vdb : Store each number in 1 byte instead of 4
n_indexer -> n_vdb : vector storage per chunk, 768 dims
note over n_api, n_vdb : The filter removes almost everything the graph found
n_api -> n_vdb : filter: lang = "uz" AND in_stock = true matches 8% of chunks
note over n_api, n_vdb : Choose the plan by how much the filter keeps
n_api -> n_vdb : plan by how many chunks the filter keeps
note over n_api, n_text : Vectors miss exact codes: bring BM25 back
n_api -> n_text : q = "KT-2041 choynak qopqog'i yopilmayapti"
note over n_api, n_llm : Give the best 5 chunks to an LLM, with sources
n_api -> n_llm : retrieve the top 5 chunks (hybrid, 14 ms) -> 2 100 tokens of context
@endumlPlantUML: components
@startuml
left to right direction
agent "Help chat" as n_user
component "Search API" as n_api
database "Keyword index" as n_text
collections "Indexer" as n_indexer
database "Articles DB" as n_docs
cloud "Embedding model" as n_model
database "Vector index" as n_vdb
cloud "LLM" as n_llm
n_user --> n_api : call
n_indexer --> n_docs
n_indexer --> n_text
n_api --> n_model : call
n_api --> n_vdb
n_indexer --> n_model
n_indexer --> n_vdb
n_api --> n_text
n_api --> n_llm : call
@endumlRecap
Six rules for vector search.
1) An embedding model turns text into a vector; cosine similarity finds the same meaning without shared words. 2) Exact kNN costs N x d per query; use it only for small sets. 3) HNSW trades a few points of recall for milliseconds; ef_search is the query-time knob, M the build-time one. 4) Quantize to int8 and rescore the top candidates to cut memory without losing recall. 5) Filter during the search, pick the plan by selectivity, and combine vectors with BM25 through reciprocal rank fusion. 6) A vector is only comparable with vectors from the same model version: store the version, and re-embed into a new index before switching.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Recall@10 (golden set) | 41% | 93% |
| Search p95 | 12 ms | 14 ms |
| Index memory | 6.1 GB | 1.8 GB |
| Empty results | 0% | 0% |
Steps
- Case The customer and the article use different words
The help chat searches articles with BM25, as in the full-text search lesson.
- Fix Turn meaning into 768 numbers
An embedding model maps a piece of text to a vector: here 768 numbers.
- Case Similar meaning, small angle
Cosine similarity measures the angle between two vectors: 1 means the same direction, 0 means unrelated.
- Problem Compare the question with all 2 million vectors
The vectors go into a vector index, and the indexer embeds every new or changed chunk.
- Fix Walk a graph instead of reading everything
HNSW links each vector to its nearest neighbours in a layered graph.
- Fix Store each number in 1 byte instead of 4
The index needs 6.4 GB of RAM, and RAM is the expensive part.
- Problem The filter removes almost everything the graph found
Uzbek-speaking customers must get Uzbek articles about products in stock.
- Fix Choose the plan by how much the filter keeps
The filter moves into the search.
- Fix Vectors miss exact codes: bring BM25 back
"KT-2041 choynak qopqog'i yopilmayapti" (the KT-2041 lid does not close).
- Fix Give the best 5 chunks to an LLM, with sources
Retrieval-augmented generation is one more step on top: the top 5 hybrid chunks, about 2 100 tokens, go into the LLM prompt with an instruction to answer only from them and cite them.
- Recap Recap
Six rules for vector search.
Terms
- Embedding
- A vector of numbers a model makes from a text or an image, for example 768 numbers. Things with similar meaning get nearby vectors; vectors from different models or versions cannot be compared.
- Cosine similarity
- The cosine of the angle between two vectors: 1 means the same direction, 0 means unrelated. For vectors of length 1 it is just the dot product.
- HNSW
- A layered graph for approximate nearest neighbour search: each vector links to its neighbours and a search walks the graph towards the query. ef_search sets recall and latency, M sets memory.
- Reciprocal rank fusion (RRF)
- A way to merge several ranked lists: a document gets 1 / (60 + rank) from each list and the terms are added up. Scores need no common scale, which makes it handy for combining BM25 and vector results.
- Re-embedding
- When the embedding model changes: encode every document again with the new model into a new index, then switch it through an alias together with the query model. Otherwise old vectors and new queries live in different spaces.