This analysis organizes the evidence behind does Your Knowledge Base Really Need RAG?, then explains the practical implications, trade-offs and current limits.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
Should RAG be used for a knowledge base of 200 documents? Before investing in vector libraries and embedding pipelines, I ran three sets of controlled experiments. Result: RAG is not required for small data sets, but keyword searching is not an alternative—semantic ambiguity can completely break retrieval.
Should I plug the knowledge base of 200 documents into it directly or build a RAG?
I have a knowledge base of 200 documents, including 64 core documents (14 project technical documents + 50 WordPress articles) and 136 public AI technical documents. The goal is to build a question-and-answer system that allows users to query the content of these documents using natural language.
The intuitive reaction is "Retrieval-Augmented Generation (RAG)" - building embedding, vector database, and retrieval pipeline. But before committing to those construction costs, a more fundamental question is:Does my data volume really require a RAG?
In order to answer this question, I designed three sets of baseline experiments, using the simplest method to first find out where the upper and lower limits are.
experimental design
Test questions: 20 questions, four types
20 test questions are designed, divided into four types to test different retrieval abilities:
- Type A — Single document direct answer(5 questions): The answer exists in a certain document and does not need to be integrated across documents. Test basic information positioning abilities.
- Type B — Cross-document synthesis(5 questions): The answer requires integrating information from multiple documents. Test whether the system can piece together scattered information.
- Type C — Reasoning and Judgment(5 questions): The answer is not written directly in the document, but needs to be inferred based on the content. Test depth of understanding.
- Type D — Questions outside the knowledge base(5 questions): The answer is not in the knowledge base. Test whether the system can honestly say "I don't know" instead of making up the answer.
Each question is worth 1-5 points, and will be self-evaluated by AI and then reviewed manually.
Three sets of Baseline
Baseline A-R1: Context Stuffing, 14 documents— Insert the complete contents of 14 project documents directly into the prompt and let AI answer each question one by one. About 78 KB / 7,300 words.
Baseline A-R2: Context Stuffing, 64 documents— cram all 64 documents into prompt. About 415 KB / 18,300 words / estimated 120,000-150,000 tokens. The 1M context window of Claude Opus 4.6 is used.
Baseline B: keyword search (64 documents)—Cut 64 documents into 1,583 paragraphs, use TF-IDF (term frequency-inverse document frequency) weighted keyword search to find the top-5 paragraphs, and use only these 5 paragraphs to answer. Simulate the simplest RAG scheme.
Context Stuffing scored the highest score of 93, and keyword search scored 59 points.
Overall score at a glance
| method | Total score (/100) | average | Type A | Type B | Type C | Type D |
|---|---|---|---|---|---|---|
| A-R1: Context Stuffing 14 files | 88 | 4.40 | 4.80 | 3.80 | 4.00 | 5.00 |
| A-R2: Context Stuffing 64 files | 93 | 4.65 | 4.80 | 4.60 | 4.40 | 4.80 |
| B: Keyword search 64 files | 59 | 2.95 | 3.80 | 1.80 | 2.40 | 3.80 |
64 stops performs better than 14 – more data is better
64 gears (93 points) perform better than 14 gears (88 points), and the improvement mainly comes from the Type B cross-file problem (3.80 → 4.60). The 50 WordPress articles add critical context to many issues—such as MBA background documented in the articles, split decision logic, and performance optimization processes—that are missing from the 14 project documents.
Intuitively, one would worry that "too much stuffing will interfere with the AI", but the actual measurement is the opposite.More relevant information gives the AI more clues to cross-reference, and Claude's 1M context window has no obvious quality degradation when processing 150K tokens.
Type B only scored 1.80 points for keyword search in a mixed Chinese and English scenario.
A keyword search score of 59 may seem “okay” on the surface, but when taken apart, the problem is serious.Type B cross-document questions only scored 1.80 points(Full score is 5), there are three questions that directly score 0-1 points. The reason is semantic ambiguity:
- Check "Technical Selectiondecision making》→ Match an article about writing AI decision makingSupported unrelated articles
- Check "Blog-Writer" SplitEvolution" → Match to "" in the game marketing articledistinguishinto two types"
- Check "Act 1 ProjectAlignmenttarget"→ Matched one article"ProjectOlder articles on "Management Tools Comparison"
This is no accident. In a knowledge base with a mixture of Chinese and English and diverse themes, the same Chinese word has completely different meanings in different contexts. Keyword search cannot distinguish between these contexts and will match as long as the superficial words match, producing a large number of false positives.
Chinese bigram segmentation produces a large number of meaningless tokens, exacerbating matching noise.
Baseline B uses bigram (two-character segmentation) for Chinese tokenization. However, Chinese bigram will generate a large number of meaningless tokens - for example, "from the initial single tool to the split" will be cut into "from the most", "initial", "original", "single" and "single"... These fragments may be accidentally matched in any article, seriously diluting the truly meaningful keyword signal.
Human Review: Is AI Self-Assessment Trustworthy?
The three sets of baselines are all AI self-scoring scores, which themselves need to be verified. I selected 5 questions for manual review (one question for each type, and selected the question with the largest change in scores or the most worthy of verification), and compared the AI answers one by one with the original text of the knowledge base.
Results: Among the 15 ratings, 12 (80%) review suggestions were the same as the self-assessment, 3 may have deviations of ±1 point, and the direction of deviation was inconsistent (either higher or lower), and there was no systematic bias.
The more important conclusion is that even if all deviations are adjusted, the three-party rankings remain unchanged.R2 (93→91) > R1 (88→89) >> B (59). The conclusion is robust.
462K tokens don’t fit into a 200K context window — that’s why RAG exists
The total of 200 documents is about 764K characters, which is estimated to be about 462K tokens, which is 2.3 times that of the standard 200K context window. Even using Claude Opus 4.6 with 1M context window, it will occupy nearly half of the context. Coupled with context rot (accuracy and recall decrease as the number of tokens increases) and cost issues, Context Stuffing is not feasible at this scale.
When the amount of data exceeds what the context window can handle, you need to retrieve it first and then inject it instead of stuffing it all in.
What this means
The conclusions of the three sets of experiments can be condensed into three sentences:
First, small datasets do not require RAG.With 64 documents and less than 150K tokens, Context Stuffing directly scored 93 points. It is better to save the construction cost of embedding, vector database and retrieval pipeline and just plug them in directly.
Second, keyword hunting is not an alternative to RAG.In a knowledge base with a mixture of Chinese and English and diverse topics, semantic ambiguity will completely break down keyword search in cross-document and reasoning problems. What you need is semantic search (vector search), not better keyword search.
Third, do the baseline first and then the system.If I had jumped into building the RAG pipeline from the beginning, I wouldn’t have known that Context Stuffing was good enough on small datasets, or what the failure modes of keyword searches looked like. Baseline experiments don’t take much time, but they let you know where the limits are and where the problems are before you invest in the cost of building it.
Before you build a system, run it in the simplest way and see what the data tells you.
Practical questions and boundaries
Is a Context Stuffing score of 93 good enough? Why do you need RAG?
A score of 93 was achieved at a scale of 64 documents. When the file size grows to 200 (462K tokens), Context Stuffing cannot physically fit it in. And even if it can be squeezed in, the cost of sending 462K tokens for every query is unacceptable. RAG solves the problem of scale, not quality.
Are AI self-assessment scores trustworthy?
In this experiment, 80% of the self-assessments were consistent with manual review, with the deviation within ±1 point and no systematic deviation. But this does not mean that all scenarios can be based solely on self-assessment – there are only 5 questions in this review sample. Formal evaluation systems should have a greater proportion of manual review. The key point is: self-evaluation can be used as a tool for rapid iteration, but it cannot be used as the final quality standard.
Can these conclusions be generalized to other knowledge bases?
The core conclusions (small data sets do not require RAG, keyword searches have semantic ambiguity issues) are universal. However, the specific score threshold will vary depending on the characteristics of the data - keyword searches in pure English knowledge bases may perform better, and knowledge bases in specialized fields may require fewer documents to trigger context decay. It is recommended to run a baseline on your own data and use your data to make judgments.
What to take away
The article's value is in the evidence and trade-offs behind does Your Knowledge Base Really Need RAG?, not in treating the conclusion as universal.