This analysis organizes the evidence behind enterprise RAG: From Go/No-Go to Evaluation, then explains the practical implications, trade-offs and current limits.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
The company has a bunch of internal documents and wants to do smart Q&A. The first question is not "how to build a RAG" but "do I need a RAG?" This article uses actual measurement data to go through the complete process from evaluation, construction, diagnosis to optimization - 200 documents, 20 test questions, and five sets of controlled experiments. The final conclusion: The value of RAG is not that it is better than Context Stuffing, but that it can handle the scale that Context Stuffing cannot handle.
Whether the amount of data you have can fit into the Context Window determines whether you need RAG.
RAG (Retrieval-Augmented Generation) is currently one of the most popular directions for enterprise AI introduction. But before investing in the construction costs, there’s a more fundamental question:With the amount of data you have, do you really need RAG?
The knowledge base of many companies is actually not that big. 50 internal documents, 100 technical documents - these can be directly plugged into the context window of LLM and used, without the need to build any vector database or retrieval pipeline.
The following is a record of my entire evaluation process using actual data.
Context Stuffing scored 93 points in the small data set, while keyword search only scored 59 points in the Chinese environment.
Context Stuffing: Stuff the file directly into it
The simplest solution: insert the complete contents of all files into the LLM prompt and let it answer directly. No need to build any pipeline.
Actual test results:
- 14 documents (~25K tokens): 88 points (out of 100)
- 64 documents (~121K tokens): 93 points — More data is better because cross-document questions have more context to cross-reference
A score of 93 is pretty good. If your knowledge base is within this scale, direct Context Stuffing is the best solution, saving the construction cost of the entire RAG pipeline.
Keyword search is seriously ineffective in Chinese multi-topic environment
If the amount of data is too large to fit in, the most intuitive alternative is keyword search - use TF-IDF weighting to find relevant paragraphs, and only stuff the most relevant paragraphs into the prompt.
Actual test results:59 points.
On the surface it looks "ok", but when you take it apart the problem is serious. The cross-document questions are only 1.80 points (out of 5), and three questions directly get 0-1 points. The root cause is semantic ambiguity—the same Chinese word has completely different meanings in different contexts, and keyword searches cannot distinguish them. Check "Technical Selectiondecision making"Match to AIdecision makingFor supporting articles, check "Blog-WriterSplit"Matching to game marketing"distinguishinto two types”.
Conclusion: Keyword search is seriously ineffective in Chinese multi-topic environments. What you need isn't better keyword search, it's semantic search.
Six technical decisions for building a RAG system: Choose something that can run first in the PoC stage
After confirming the need for RAG, every step in the construction process requires selection. A brief description of six key trade-off decisions:
| Decision items | Choose | Core trade-off |
|---|---|---|
| Chunking Strategy | Mixed (title + fixed length) | A mixed Chinese and English knowledge base needs to take into account both file structures. |
| Chunk Size | 500 characters + 100 overlap | Too big contains noise, too small loses context, 500 is the balance point for Chinese |
| Embedding Model | multilingual-MiniLM (384 dimensions) | Not the best, but free, lightweight, bilingual support in Chinese and English |
| vector database | ChromaDB | Local free is suitable for PoC. For production environment, you need to switch to cloud solutions such as Pinecone. |
| search strategy | Hybrid (Vector + BM25 + RRF) | Semantic search and exact matching are complementary and better than pure vectors or pure BM25 |
| Top-K | First 5 then 10 | Less will miss cross-file answers, more will introduce noise. |
There is no single right answer for every decision, it depends on the characteristics of your data. But there is a common logic:In the PoC stage, first choose the one that can run, confirm that the direction is correct, and then choose the one that can run well.
Generation Failure = 0: All quality problems are on the retrieval side, not in LLM
The first version of RAG ran a 20-question test and scored 77 points. Lower than Context Stuffing's score of 93.
If you only look at the total score, the conclusion seems to be "RAG is not good enough." But this conclusion has no action value—you don’t know what to change. So I did a question-by-question quality diagnosis: marking 100 chunks (20 questions × 5 chunks) for relevance and classifying failure modes.
The results are very clear:
| failure mode | Proportion | Description |
|---|---|---|
| Retrieval Failure | 33% | Search finds completely unrelated passages |
| Coverage Failure | 27% | Found some relevant information but incomplete information |
| Generation Failure | 0% | LLM’s answering ability is not a bottleneck at all |
| No failures | 40% | Search and answer are both good |
Generation Failure = 0 is the most important finding.It means: don't spend time upgrading LLM, invest all your resources in improving retrieval. This diagnosis directly determines the direction of optimization - what you don't do is just as important as what you do.
Query Rewriting + Top-K adjustment, score increased from 77 to 88
Based on the diagnostic results, I chose two optimizations:
Query Rewriting:Before searching, let the system rewrite the user question into 2-3 versions, describing the same information need from different perspectives, and then search the merged results separately. Solve the problem of "there is a gap between the language of user questions and the language of the knowledge base".
Top-K increased from 5 to 10:Give more paragraphs a chance to be included and solve the problem of "correct paragraphs exist but are excluded when ranking between 6-15".
Reranker and Metadata Filtering were evaluated simultaneously,decide not to do. The reason is that the 20-question test set is too small, and if too many variables are changed at the same time, the source of the effect will be unclear, making it easy to overfitting. Controlling variables is more important than chasing the highest score.
Optimize results
| indicator | Before optimization | After optimization | improve |
|---|---|---|---|
| total score | 77 | 88 | +14.3% |
| Retrieval Failure | 5 questions | 2 questions | -60% |
| No failed questions | 6 questions | 11 questions | +83% |
| Type A (single file) | 4.60 | 5.00 (full score) | +0.40 |
| Type B (cross-file) | 3.20 | 4.20 | +1.00 |
Only the pre-retrieval processing (Query Rewriting) and parameters (Top-K) were changed, and LLM, Embedding Model, and vector database were not changed, and the score was raised from 77 to 88. The diagnostic conclusion that "the problem is on the retrieval side" is verified.
Best solution for different data sizes: small data sets do not require RAG
| method | total score | Can handle scale | Construction cost | Applicable scenarios |
|---|---|---|---|---|
| Context Stuffing | 88-93 | < 150K tokens | zero | Small knowledge base, the easiest way to use it directly |
| keyword search | 59 | any size | low | Not recommended for Chinese multi-theme environments |
| RAG v1 (basic configuration) | 77 | No upper limit | in | Starting plan for a large knowledge base |
| RAG v2（+ Query Rewriting） | 88 | No upper limit | in | Recommended solutions for large knowledge bases |
Simple decision tree:
- Data volume < 100K tokens: Direct Context Stuffing. No need to build any pipeline, the effect is the best.
- 100K-200K tokens:Context Stuffing might be ok (if you have 1M context window), but starts to have cost and context rot pressure. You can start evaluating the RAG.
- > 200K tokens:RAG is the only feasible solution. The actual measurement can achieve 95% of the effect of Context Stuffing (88 vs 93), and can process 3.8 times the amount of data.
The real value of RAG is not "better than Context Stuffing", but "can handle scales that Context Stuffing cannot handle".Within the scope of the context window, Context Stuffing is almost certainly better than RAG. Only when the amount of data exceeds a critical point does RAG become a necessary option.
What this means
After going through this entire process, the biggest experience is:The quality management of AI systems is completely different from traditional software.
When a bug occurs in traditional software, look at the log to find the root cause, and run tests to confirm after fixing it. A “bug” in an AI system isn’t black and white — it’s “get 2 points for this question instead of 5.” You need a different set of quality diagnostic methods: marking search quality, classifying failure patterns, and deriving optimization directions from failure patterns. This methodology has nothing to do with technology, but management capabilities.
Another counter-intuitive conclusion is:When evaluating AI solutions, it is much more efficient to build the simplest baseline first than to build the most complete system first.If I had jumped into building the RAG pipeline from the beginning, I would not have known how powerful Context Stuffing is on small data sets, nor would I have known the failure mode of keyword search in the Chinese environment. These baselines don't take much time, but they define the standards that RAG needs to exceed and also reveal the weaknesses of each approach.
For enterprises, the correct order for importing RAG is:Verify the requirements first (is the amount of data really too big for you?) → then build a baseline (how effective is the simplest solution?) → then build a RAG → do quality diagnosis → targeted optimization.Skipping the first two steps and building a RAG directly is the most common waste.
Guide to series of articles
This article is an overview of the complete process. If you want to dive into a specific topic, here are the other six articles in the series:
Things you should know before RAG: A complete comparison of four data retrieval methods
Vector search, BM25, Hybrid Search, principles of knowledge graphs and trade-off. Suitable for readers who want to understand what the bottom layer of RAG is doing.
Does your profile really need RAG? Three groups of Baseline actual tests will tell you
Complete experimental data of 88-93 points for Context Stuffing and 59 points for keyword search. Why RAG is not needed for small datasets.
Six technical decisions you need to make before building a RAG Pipeline
Selection records of Chunking, Embedding, vector database, and search strategies. Write down the considerations and sacrifices for every decision.
RAG actual measurement: The data generated using 200 files is different from what you think
Correct interpretation of RAG 77 points vs Context Stuffing 93 points. Optimal solution selection at different scales.
How did the RAG system score 77 come about? A complete Retrieval quality diagnosis
Break down the retrieval quality of 100 chunks question by question. Methodology for discovery and failure mode classification of Generation Failure = 0.
RAG optimization practice: Query Rewriting how to increase the score from 77 to 88
Optimized decision-making process after diagnosis. Why Query Rewriting, Why Not Reranker, Limitations of 20 Question Sample.
200 Chinese documents account for 462K tokens, which is 2.3 times the upper limit of Context Window
Context Stuffing works well but has a physical limit - LLM's context window cannot hold too many files. Where is the critical point?
This knowledge base has 200 documents totaling approximately 764K characters and an estimated 462K tokens. The standard context window is 200K tokens, and 462K is 2.3 times the upper limit. It is physically impossible to fit them all in. Even using a 1M context window, it takes up nearly half of the space plus context rot (accuracy decreases when the number of tokens increases), which is not practical.
The judgment method is very simple, and there is no need to actually run an excessive prompt:
| Data size | Estimate tokens | vs 200K cap | Context Stuffing Feasibility |
|---|---|---|---|
| 14 documents | ~25K | 13% | Totally feasible and effective |
| 64 documents | ~121K | 60% | Feasible, close to the upper limit |
| 200 documents | ~462K | 230% | Not feasible |
Chinese token efficiency is about 3-4 times worse than English(A Chinese character accounts for about 1.5 tokens), which is easily overlooked when estimating. 50 Chinese articles account for 350K tokens, but only 25% of the number of documents.
Practical questions and boundaries
How many engineering resources do companies need to introduce RAG?
The PoC stage can be done by one person. The RAG pipeline code itself is not complicated and can be run in just a few hundred lines of Python. What really requires investment is not writing programs, but data organization (ensuring that the file format is consistent and of good enough quality) and quality assessment (designing test questions, making diagnoses, and deciding on optimization directions). Engineering teams are needed to operate and scale production environments.
Are these data applicable to other industries?
Methodology applies, specific numbers do not. This knowledge base is a mixture of Chinese and English AI technical documents. Your knowledge base may be purely Chinese legal documents or English customer service records. The optimal parameters for each data characteristic are different. However, the process of "do baseline first → build RAG → diagnose → optimize" is universal. It is recommended to run it on your own data and use your data to make judgments.
Are there any other options besides RAG?
Yes. If your data is highly structured (e.g. database, knowledge graph), Text-to-SQL or graph queries may be more suitable than RAG. If your Q&A scenario is very fixed (such as FAQ), simple intent classification + template reply may be enough. RAG is most suitable for scenarios of "large amounts of unstructured documents + open questions and answers". Don’t think of RAG as a one-size-fits-all solution—understand the characteristics of your scenario first, and then choose a solution.
What to take away
The article's value is in the evidence and trade-offs behind enterprise RAG: From Go/No-Go to Evaluation, not in treating the conclusion as universal.