How AI learns to answer from your own documents.
The method was published in 2020 by researchers at Facebook AI Research, University College London and New York University, and is now the standard way companies connect language models to their own knowledge. Six years of evidence show what it fixes, what it does not, and which decisions belong to management rather than to the technology team.
Die Insights erscheinen auf Englisch.
Most companies that describe an “AI project” are describing a search problem. The model is bought or rented; what has to be built is the path from a question in plain language to the documents that already hold the answer.
That path has a name, a date and, six years on, a documented set of failure modes. Retrieval-augmented generation — RAG — moves the accuracy of an answer out of the model and into the company’s own document base.
The design was published in 2020 and has changed less than the models around it
Patrick Lewis and eleven co-authors submitted “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” on 22 May 2020; it appeared at NeurIPS 2020. Large models store factual knowledge in their parameters, the authors observed, but cannot access and manipulate it precisely. Pairing a generative model with a dense vector index of Wikipedia set the state of the art on three open-domain question-answering tasks and produced “more specific, diverse and factual language” than a parametric baseline.
IBM Research defines it as “an AI framework for retrieving facts from an external knowledge base to ground large language models on the most accurate, up-to-date information”; its director of language technologies, Luis Lastras, calls it “the difference between an open-book and a closed-book exam”. NVIDIA compares retrieved passages to “footnotes in a research paper, so users can check any claims”.
The pipeline is short: documents are cut into passages and converted into vectors, the question the same way, and the closest passages are retrieved, ranked and passed to the model. Nothing is retrained — when a contract is amended, the index is updated, not the model.
The retrieval step, not the model, sets the ceiling on the answer
What the model has to work with is decided before it writes a word, and that step can be measured. The difficulty starts with cutting documents up. Anthropic, in engineering work published on 19 September 2024, gives the standard example: a passage reading “The company’s revenue grew by 3%” is useless once separated from the sentence naming the company and the quarter. Across its evaluation datasets, the baseline pipeline failed to place the right passage in its top twenty results 5.7% of the time. Adding a short generated context to each passage before indexing brought that to 3.7%, a keyword index to 2.9% and a reranking step to 1.9% — a 67% reduction overall. Retrieval failure is a quantity a company can measure on its own documents before opening an assistant to customers.
A grounded answer and a correct answer are not the same thing
The first preregistered evaluation of commercial retrieval systems comes from law, where errors leave a record. In a 2025 study in the Journal of Empirical Legal Studies, Varun Magesh, Daniel E. Ho and four co-authors at Stanford and Yale ran 202 preregistered queries against commercial legal research tools built on retrieval. Lexis+ AI, the highest-performing system tested, answered 65% of queries accurately; Westlaw’s AI-Assisted Research was accurate on 42% and hallucinated nearly twice as often as the other legal tools; Ask Practical Law AI was incomplete on more than 60%. Across the tools, hallucination rates ran between 17% and 33%. Retrieval improves performance, the authors report, but “RAG systems are no panacea”.
Their vocabulary is worth borrowing. Correctness asks whether a statement is right; groundedness asks whether the source it cites supports it. A response whose cited source does not support it is misgrounded, and counts as a hallucination. It is the failure most likely to survive internal review: the answer arrives with a reference attached.
A study presented at ICLR 2025 and summarised by Google Research explains why weak retrieval can be worse than none. Measuring whether retrieved passages held enough information to answer at all, the authors found that models answer anyway rather than abstain: the Gemma model gave incorrect answers on 10.2% of questions with no context, and 66.1% when the context supplied was insufficient. More retrieved material is not the same as more usable information — but it does make a model more likely to answer.
The asset a company builds is the document base, not the model
Eurostat reported on 11 December 2025 that 20.0% of EU enterprises with ten or more employees used AI in 2025, against 13.5% a year earlier, and that the most used category, at 11.8%, was technologies analysing written language. The gap by size is wide — 17% of small firms against 55% of large ones — but the work being automated is, above all, document work.
Three consequences follow for the corpus. The first is authority: retrieval treats every indexed version as equally true, so three versions of a warranty policy produce three defensible answers. The second is access control: a search index does not inherit the permissions of the file server it came from. Microsoft’s documentation for Azure AI Search sets out two deliberate routes to document-level authorisation — built-in access control lists, or a security filter the developer builds — and warns that in the second, “there’s no authentication or authorization through the security principal”, only a string used to include or exclude a document. An index built without it makes every document readable by anyone who can reach the assistant. The third is maintenance: the index must follow the documents — a permanent responsibility, not a project milestone.
Two legal duties apply before the first assistant reaches a customer
For a Swiss company the rule already applies: the Federal Data Protection and Information Commissioner confirmed on 8 May 2025, restating a position of 9 November 2023, that the Federal Act on Data Protection “is formulated in such a way as to apply to all types of technology and is therefore also directly applicable to the use of AI-supported data processing”. Manufacturers, providers and users must make the purpose, functionality and data sources of that processing transparent; individuals have a right to know whether they are corresponding with a machine; high-risk processing requires a data protection impact assessment. A corpus of contracts, service tickets and correspondence is, in most companies, a corpus of personal data.
For any assistant whose output reaches the European Union, Article 50 of the AI Act has applied since 2 August 2026. Providers of systems that interact directly with people — the Commission names chatbots, AI agents and avatars — must ensure users are told they are dealing with an AI system unless that is obvious; generated content must be marked in machine-readable form, with a grace period to 2 December 2026 for systems already on the market. Under the Act’s definitions, a firm that develops an assistant and puts it into service under its own name acts as its provider, not only its deployer, and providers outside the Union are in scope where the output is used inside it. Article 99(4)(g) caps the fine for breaching Article 50 at 15 million euros or 3% of worldwide annual turnover, whichever is higher.
The objection that documents cannot leave the country has had an answer since 2 September 2025, when EPFL, ETH Zurich and the Swiss National Supercomputing Centre released Apertus, an open multilingual model licensed for commercial use and available through Swisscom and Hugging Face. Where a model can be hosted decides where the index may sit.
What a management team can decide this quarter
- Write the questions before choosing the tool: thirty to fifty real ones, with answers agreed by their owners, are the evaluation set for comparing offers.
- Score two things separately: was the answer correct, and does the passage it cites support it.
- Make abstention acceptable: “this is not in the documents” must be a permitted answer, and tested.
- Repair the corpus before enlarging it: one authoritative version per document, a named owner, a removal rule.
- Mirror access rights into the index before the first query, not after the first incident.
- Record the processing — purpose, data sources, and the disclosure the user sees — as Swiss law requires.
Two dates frame the coming months: 2 December 2026, when the AI Act’s grace period for marking generated content ends, and the end of 2026, by which the Federal Department of Justice and Police is to submit a Swiss consultation bill implementing the Council of Europe AI Convention.
Whichever models a company is running by then, a curated, permission-aware document base does not have to be rebuilt with them.