Image credit: Li and colleagues, 2024, Figure 1. Original figure padded to 16:9. Cost panel shows relative token use.
Pasting a long PDF into a chatbot and asking it to search a folder can look similar. Both can produce an answer based on documents. But the routes differ. One puts the document into the model's input. The other first selects material that appears relevant, then asks the model to answer from that selection.
A huge context window makes the first route more practical. It does not make the second redundant. Research comparing them shows a tradeoff between access to the full document and the cost of processing it. The useful question is which information a particular answer needs, and how reliably the system can find and use it.
Retrieval augmented generation, usually shortened to RAG, combines retrieval with a model that writes an answer. Search can cover a collection of files or web material. The retrieved passages become evidence in the prompt rather than requiring the model to rely only on information learned during training. Modern uses vary, but a 2020 paper by Patrick Lewis and colleagues established an influential version of the approach.
That system paired the BART text generator with Dense Passage Retrieval, or DPR. A dense retriever represents questions and passages as numerical vectors and searches for relevant matches. The authors built their main index from December 2018 Wikipedia, split into separate chunks of 100 words. That produced about 21 million retrievable documents, meaning passages rather than 21 million Wikipedia articles.
The motivation included factual accuracy, the ability to update knowledge and the difficulty of showing where an answer came from. Meta's publication summary describes combining the generator's learned knowledge with an external Wikipedia index. Keeping some knowledge outside the model meant researchers could change the evidence without retraining the whole generator.
Their index swap experiment made that distinction concrete. The researchers assembled questions about 82 world leaders whose positions changed between December 2016 and December 2018. With the matching Wikipedia index, their model answered 70% correctly for the 2016 leaders and 68% for the 2018 leaders. With mismatched indexes, accuracy fell to 12% for 2016 leaders using the 2018 index and 4% for 2018 leaders using the 2016 index.
That was a controlled demonstration of updating an answer source, not proof that any search result overrides a model's assumptions. The paper reported new best results at the time on Natural Questions, WebQuestions and CuratedTrec. Its RAG Sequence model scored 44.5 exact match on Natural Questions, where exact match checks whether an answer matches an accepted reference.
The authors also evaluated generated Jeopardy questions. Table 4 reports that human evaluators preferred RAG's factuality in 42.7% of comparisons, preferred BART in 7.1%, and rated both good in 11.7%. These are judgments about generated questions, not the percentage of quiz questions a chatbot answered correctly. The accompanying prose gives a different figure for both being factual, so the figures here follow the table.
Long context offers a different way to supply evidence. Google's February 2024 Gemini 1.5 announcement described a private preview reaching one million tokens, alongside a standard window of 128,000 tokens. Tokens are the units a model processes, which can be words or pieces of words. A sufficiently large window lets a user supply a lengthy source without first reducing it to selected passages.
Google reported finding a hidden piece of text 99% of the time in a test extending to one million tokens. The Gemini 1.5 technical report also documents very high recall in these needle in a haystack evaluations. Finding a deliberately planted fact is useful evidence of recall, but it is not the same task as weighing scattered clues or answering an ambiguous question about a whole book.
Fitting a document into the input therefore does not establish that a model will use every relevant part correctly. The site's earlier explanation of why chatbots lose track in long conversations covers that distinction in more detail. For this comparison, the important point is that retrieval and long context each introduce different ways to miss evidence.
A 2024 study by Zhuowan Li and colleagues compared the two directly. It tested Gemini 1.5 Pro, GPT 4o and GPT 3.5 Turbo across nine datasets. Seven came from LongBench, including NarrativeQA, Qasper, MultiFieldQA, HotpotQA, 2WikiMultihopQA, MuSiQue and QMSum. Two came from Infinity Bench, covering English question answering and multiple choice questions.
The default retrieval setup divided documents into chunks of 300 words and supplied the five highest ranked chunks. Contriever supplied the main results, with Dragon used in a separate comparison. The long context route supplied the full context within the model's limits. This tested specific systems and prompts, not every possible version of RAG against every long context model.
The paper's prose and Table 1 do not always agree. Its prose reports average long context advantages of 7.6%, 13.1% and 3.6% for Gemini, GPT 4o and GPT 3.5 respectively. Table 1 instead lists average scores of 49.70 versus 37.33 for Gemini, 48.67 versus 32.60 for GPT 4o, and 32.07 versus 30.33 for GPT 3.5. Those table values are the comparison used here.
These averages combine task measures such as answer overlap, multiple choice accuracy and summary overlap. They should not be read as a single real world accuracy percentage. The study also found an important exception. On the two longer Infinity Bench datasets, retrieval outperformed the full context route for GPT 3.5 Turbo, whose 16,000 token window could not accommodate the much longer inputs.
An earlier study by Peng Xu and colleagues in 2023 had reached a more favorable conclusion about retrieval. It found that retrieval with a 4,000 token window could match a model extended to 16,000 tokens, and that retrieval could improve longer context models too. Li and colleagues suggest their different results may reflect stronger models and longer windows. The studies show why a conclusion needs its model and setup attached.
Retrieval can lose questions before the answer model starts reasoning. Li and colleagues identify four recurring problems. Some questions need several reasoning steps, with one answer needed to locate the next piece of evidence. Others are too general for a narrow search, or long and complex enough to confuse the retriever. Implicit questions may require understanding material that does not directly mention the words in the question.
Their spaceship example illustrates the last problem. A question asks what caused a shadow behind a spaceship, but the passage revealing the cause does not explicitly mention the shadow. Matching the question to a few apparently similar passages can miss the explanation. The paper relates this kind of failure to NarrativeQA stories, where dialogue and dispersed clues matter.
Barnett and colleagues describe seven engineering failure points from three case studies. Their names are "Missing Content", "Missed the Top Ranked Documents", "Not in Context - Consolidation strategy Limitations", "Not Extracted", "Wrong Format", "Incorrect Specificity" and "Incomplete". These distinguish an absent answer from one that exists but ranks too low, disappears while passages are combined, or reaches the model without being used correctly.
The remaining failures concern how the model responds, including ignoring the requested format, choosing the wrong level of detail or leaving out part of the answer. A larger window does not fix missing source content. Better retrieval does not guarantee faithful writing. The site's explanation of chatbot hallucinations covers why retrieval does not cure invented answers, while its citations explainer shows why a displayed reference still needs checking.
Li and colleagues proposed combining the approaches through Self Route. The first stage supplies retrieved passages and lets the model answer or declare the question unanswerable from that material. If it declines, a second stage supplies the full context. The method tries to reserve the larger input for questions that need it, although it depends on the model recognizing when the selected evidence is insufficient.
The motivation was overlap. The paper reports identical predictions for 63% of queries, including incorrect predictions. For 70%, the absolute difference between their evaluation scores was below 10 on a scale multiplied by 100. Agreement therefore suggested an opportunity to avoid some large prompts, but did not establish that either answer was trustworthy.
Table 1 shows the compromise. Self Route scored 46.41 for Gemini versus 49.70 with full context, 48.89 versus 48.67 for GPT 4o, and 35.32 versus 32.07 for GPT 3.5. Its reported token shares relative to full context were 38.39%, 61.40% and 38.85% respectively. These are token usage figures, not a measured universal reduction in bills or response time.
The same table lists Gemini's answerable rate as 76.78%, rather than the roughly 82% or 81.74% stated elsewhere in the prose. The paper also describes different performance averages and cost reductions in its narrative. The table values above are retained without combining them with those claims. Self Route was a useful compromise in this experiment, not an identical accuracy substitute in every case.
In the study's retrieval size experiment, five chunks gave the lowest average token cost for Self Route. More chunks made the first stage larger but could reduce the number of questions sent to full context. Some individual datasets favored one chunk instead. Five is therefore a tested starting point, not a universal rule for how much evidence an answer needs.
For readers, a sensible approach is to use a full document when the task depends on its overall structure, a broad summary or relationships across distant passages. Search is a useful starting point when a question targets a particular fact across a large collection. If a narrow search misses the answer, broaden the evidence rather than treating the first result as the collection's final word.
For teams building assistants, test both approaches on representative questions. Include questions with no answer in the sources, and check whether the system admits that. Inspect which passages were selected, whether the answer follows them and whether its references support the claims. Evaluate the hybrid route too, including cases where it should have asked for more context but answered prematurely.
Huge context windows make it possible to read more source material at once. Retrieval helps decide which material to read in the first place. Their strengths overlap, but their costs and failure points differ. A system can search first and read more when necessary. The research supports choosing the amount of evidence a question requires rather than assuming that either a longer prompt or a search step makes the answer reliable.
Comments