Limits of vector search

Limits of vector search

a new GDM paper shows that embeddings can’t represent combinations of concepts well

e.g. Dave likes blue trucks AND Ford trucks

even k=2 sub-predicates make SOTA embedding models fall apart

www.alphaxiv.org/pdf/2508.21038

btw even adding a reranker won’t help if you’ve already dropped the relevant results in the first stage embedding retrieval

agentic search DOES work, but now you’re relying on an expensive LLM to resolve simple boolean logic

multi-vector (late interaction) search like ColBERT also works, because it handles the predicate logic in cheaper latent space, but storage costs are a lot higher because, well it’s multi-vector

(fwiw Qdrant and a few other vector DBs support multi-vectors)

huggingface.co/jinaai/jina-...

you really need to capture the query and decompose it into multiple sub queries

e.g. maybe get a 1B-3B LLM to rewrite the query into a DSL (e.g. a JSON breakdown of the various components and concepts in the query)

and then push that logic into the database engine itself

alternatively, sparse approaches like SPLADE do this in latent space but use inverted indices (regular full text search, exact matches)

arxiv.org/abs/2107.057...

imo if search is done perfectly, you effectively drive your LLM context to infinity

but it’s very much not a solved problem

to illustrate how underdeveloped this space is — research from 5 years ago still seems like the best ideas (contrast that to LLMs)