Building a cheap RAG router: can geometry tell us when a question is hard?
At Tessa we built one of the strongest knowledge-base search engines around Orlo. Some questions are easy. Some are hard. The hard ones need slow, expensive machinery. But we can't afford to run that machinery on every question, because our answers power Tonic's live call-center conversations and have to land in under five seconds. This post is about a fun idea we researched for telling the easy questions from the hard ones, cheaply, using nothing but the geometry of an embedding space.
The problem
Picture a support agent on a call. The customer asks something, the agent types it, and an answer needs to appear fast enough that the conversation doesn't stall. That's our constraint: sub-5-second answers, in real time. Anything slower, and the human on the line is left unhappy.
Our search runs on RAG (retrieval-augmented generation). The idea is simple: instead of hoping a language model already knows the answer, you first retrieve relevant documents from a knowledge base, then hand them to the model as context so it can answer from real, current information. For a knowledge base of support articles, policies, and product docs, RAG is exactly the right tool.
It works beautifully, as long as the answer lives in one document. "What's our refund window?" is a single-hop question, one lookup, one passage, done. Fast, cheap, accurate.
The trouble is multi-hop questions, where the answer requires stitching facts together across several documents. "Does the warranty on the model that was replaced by the X200 cover water damage?" You can't answer that in one lookup first you have to find what was replaced by the X200, then find that older model's warranty terms. Naive RAG does a single retrieval and generates on whatever comes back, so when the evidence is scattered you get garbage in, garbage out.
The state of the art is expensive
The modern fix for multi-hop is agentic retrieval: instead of a single lookup, an agent plans, retrieves, checks whether it has enough, and retrieves again in a loop (interleaving retrieval with chain-of-thought reasoning is the canonical version). It works the current top systems on 2WikiMultiHopQA are almost all graph or agentic pipelines like HippoRAG 2 and GFM-RAG, and published 2026 benchmarks put agentic systems at 25 to 40 percent fewer irrelevant retrievals on multi-hop and ambiguous queries. But that quality comes at a steep price in exactly the two dimensions we care about:
- Cost. An agentic pipeline runs 3 to 5 LLM calls per query instead of 1, and broadly costs 10x what a naive pipeline does for the same job.
- Latency. This is the dealbreaker for us. Naive retrieval is quick, but adding agentic decomposition can blow latency out completely one recent study measured naive RAG at 0.03s per query and the full decomposition pipeline at 18.9s per query. Even modest agentic loops add 5 seconds on top of the baseline. For a system with a 5-second ceiling, that's not an option.
And here's the kicker: most questions don't need any of it. For a simple factual lookup, agentic RAG is overkill it returns the same answer as naive RAG, just slower and pricier.
The router idea
So the idea is clear: put a router in front. If a question is single-hop, send it down the fast, cheap, accurate path. If it's multi-hop, then pay for the agentic machinery. The industry knows that routing complex queries selectively rather than running everything through the expensive path can reduce costs by 40% and latency by 35%.
The catch is the router itself. It has to be fast and cheap, or it just becomes another tax on every query. A big classifier model would defeat the purpose. So the real question is what's the cheapest possible way to tell single-hop and multi-hop apart?
Most routers learn this from the text directly. We asked ourselves, is the answer already sitting in the geometry of the retrieval space itself? If it is, the router is free.
What is an embedding space?
To understand the idea, it helps to know where documents live within a retrieval system.
An embedding turns a piece of text into a point in a high-dimensional space, arranged so that meaning becomes position. Texts about similar things land near each other while texts about unrelated things land far apart. "Refund policy" and "return window" end up as neighbors; "refund policy" and "battery life" end up far away.
Crucially, directions carry meaning too. The relationships between points encode semantic structure. And the same machinery handles every scale: a single word, a sentence, or a whole document each collapses to a single point in the same space. A query is just another point in the embedding space, and retrieval simply answers the question "which document points are nearest to this query point?"
Meaning becomes position: documents cluster by topic, and a query (★) lands among its nearest neighbors.
A classic example comes from the original Word2Vec embedding space where the offset that turns man into woman is essentially the same offset that turns king into queen, so . Semantic meaning is carried not only by where a point sits, but by the directions between points.
The gender offset that separates man/woman is the same offset that separates king/queen.
What is a Voronoi diagram?
Once retrieval is "find the nearest point," this is a well-known piece of mathematics the Voronoi diagram (see also Aurenhammer's survey).
Imagine cell-phone antennas scattered across a map. Wherever you stand, your phone connects to the nearest antenna. Draw the borders where "nearest antenna" switches from one to another, and you've carved the map into regions, one region per antenna, each containing exactly the points closest to it. That partition is a Voronoi diagram.
Each site owns the cell of all points closer to it than to any other site; borders fall where the nearest site changes.
Now swap antennas for documents. Every document is like an antenna (a Voronoi site in the lingo), and the embedding space is the map. Each document "owns" the region of space closer to it than to any other document. When a query arrives, nearest-neighbor retrieval is exactly and provably just asking which Voronoi cell does this query fall into? The borders between cells are where two documents compete to be the closest match.
In modern embedding spaces vectors are normalized to the unit sphere, so the cells become geodesic patches, but the logic is identical: each document owns the region nearest to it.
This is the whole picture any modern knowledge-base search runs on, whether or not anyone draws the diagram explicitly (which is really hard when you have thousands of dimensions).
A fun idea: the boundary hypothesis
Here's the intuition that started this. A single-hop question has its answer in one document, so it should land comfortably deep inside that document's cell, far from any border. A multi-hop question needs evidence from several documents, so maybe it lands near a boundary, pulled toward two or three competing cells at once.
If that's true, it's a gift. A query near a boundary has a small margin the gap between its closest and second-closest document is tiny, because they're tied for the win. And the margin is essentially free to compute; we get the top matches anyway during retrieval. IF small margin then it's multi-hop so route to the agent. If big margin then it's single-hop so answer fast. A router for almost nothing.
That's the boundary hypothesis: multi-hop questions sit closer to Voronoi boundaries than single-hop ones. We tested it on 2WikiMultiHopQA, a standard multi-hop benchmark (leaderboard).
Results
At first, it looked beautiful. Multi-hop questions really did have smaller margins than single-hop ones a clear, statistically strong effect that held up even when we scaled from a toy corpus to ~50,000 documents and thousands of questions. The neighborhoods around multi-hop queries were messier too, exactly as predicted. We were ready to believe it.
Then it fell apart under extreme methodological scrutiny. The effect turned out to be an artifact of how we built our test questions: our single-hop comparison questions were quietly "cheating" by naming the exact document they targeted, which trivially made their margins look big. Once we fixed the test to be fair, the effect didn't just shrink it reversed. With clean comparisons, multi-hop questions showed larger margins, sitting farther from boundaries, not closer.
Euclidean top-1/top-2 margin by reasoning category. For compositional and inference questions the multi-hop distribution (blue) shifts toward larger margins the opposite of what the boundary hypothesis predicted.
So the nice idea was wrong. Multi-hop questions aren't poised between their supporting documents at all. What seems to happen instead is that one document usually the one tied to the entity the question explicitly names dominates the neighborhood, while the rest of the needed evidence sits off somewhere else entirely. The "bridge" you have to cross to finish the answer doesn't live in the local geometry. It lives in the links between documents, which a nearest-neighbor lookup simply can't see.
The "named" first-hop document is a top retrieval about 85% of the time; the "bridge" document the answer really hinges on is retrieved only about 5% of the time.
One compositional question laid out in the retrieval space. Rays from the query (★) reach the named document (green, ringed), but the bridge document (red diamond) sits near the query yet is reached by no ray the link lives in the graph, not the local geometry.
But... and this is what saves the idea; the geometry still knows. Hop count is identifiable from geometric features. It's just not the boundary alone that carries the signal it's a combination of margin, neighborhood spread, and the raw magnitude of the embedding vectors. A simple, cheap classifier built only on these geometric features separates single-hop from multi-hop questions with 0.91 ROC-AUC. Fold in question length and it climbs to 0.99. That's a router light enough to sit in front of every query without breaking our latency budget.
One of the cheap signals the classifier leans on: in this embedding space the vector norm falls off with query length, and multi-hop questions (blue) occupy a distinct region of the plot.
Conclusion
The headline-grabbing version of this idea "multi-hop questions live on the boundaries" turned out to be false, and falling for it early was a good reminder to test your assumptions. But the practical payoff survived the disappointment. Cheap geometric features really can tell easy questions from hard ones, well enough to route them.
It's not a silver bullet. It won't catch every hard question, and the cleanest signal showed up partly because of how our embeddings were set up, so mileage will vary across models and domains. It's also worth noting that the "weaker" query sets were far more natural for real human usage. But for the problem we actually have with Orlo in Tonic, keeping live call-center answers under five seconds without paying agentic prices on every single query a geometry-based router is cheap, fast, and good enough. Sometimes that's exactly the bar you need to clear.
References
- 2WikiMultiHopQA SOTA leaderboard — wizwand.com
- Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
- Ho et al. (2020), Constructing a Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2WikiMultiHopQA) — ACL Anthology
- Yang et al. (2018), HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering — ACL Anthology
- Karpukhin et al. (2020), Dense Passage Retrieval for Open-Domain Question Answering — ACL Anthology
- Trivedi et al. (2023), IRCoT: Interleaving Retrieval with Chain-of-Thought Reasoning — ACL Anthology
- Jeong et al. (2024), Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity — ACL Anthology
- Jiménez Gutiérrez et al. (2025), From RAG to Memory: Non-Parametric Continual Learning for LLMs (HippoRAG 2) — arXiv:2502.14802
- Luo et al. (2025), GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation — arXiv:2502.01113
- Aurenhammer (1991), Voronoi Diagrams — A Survey of a Fundamental Geometric Data Structure — ACM Computing Surveys
- Nussbaum et al. (2024), Nomic Embed: Training a Reproducible Long Context Text Embedder — arXiv:2402.01613
Based on our paper "A Formal and Empirical Study on the Geometry of Multi-Hop Query Identification." Code and Lean proofs: github.com/kutessa/boundryrepo
T. Hastor et al. (2026), A Formal and Empirical Study on the Geometry of Multi-Hop Query Identification. KDIR 2026. ResearchGate · Preprint on osf.io
Author: Tarik Hastor