This analysis examines 200 Files Revealed About RAG Data Preparation and separates supported implications from claims that still depend on context or further evidence.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
Most people think that RAG must be better than traditional solutions. The actual test data tells you otherwise - RAG scored 77 points and Context Stuffing scored 93 points. But RAG is the only solution that can handle the scale of 462K tokens. This article uses data from four sets of experiments to break down what options to choose at different scales.
Experimental configuration
Knowledge base: 200 documents (64 project documents + 136 public AI technical documents), totaling about 764K characters and an estimated 462K tokens. After hybrid segmentation (Chunking), 1,880 chunks are generated.
RAG pipeline configuration: paraphrase-multilingual-MiniLM-L12-v2 is used for embedding vectors (embedding, 384 dimensions), ChromaDB is used for vector storage, and hybrid search (Hybrid Search) uses vector + BM25 + RRF (k=60) to select top-5 chunks.
Test questions: 20 questions, four types - Type A single document direct answer, Type B cross-document synthesis, Type C reasoning and judgment, Type D questions outside the knowledge base (testing the ability to refuse to answer). Each question is worth 1-5 points. AI self-assessment will be followed by manual review and verification.
RAG 77 points vs Context Stuffing 93 points: four sets of experimental results
| method | Data volume | Total score (/100) | Type A single file |
Type B Across files |
Type C reasoning |
Type D refuse to answer |
|---|---|---|---|---|---|---|
| R1: Context Stuffing 14 files | 14 levels / ~25K tokens | 88 | 4.80 | 3.80 | 4.00 | 5.00 |
| R2: Context Stuffing 64 files | 64 levels / ~150K tokens | 93 | 4.80 | 4.60 | 4.40 | 4.80 |
| B: Keyword search 64 files | 64 files / top-5 paragraphs | 59 | 3.80 | 1.80 | 2.40 | 3.80 |
| RAG Hybrid 200 gears | 200 files / top-5 chunks | 77 | 4.60 | 3.20 | 3.20 | 4.40 |
RAG uses 2% of the information to achieve 83% effectiveness — scale is the key
It’s normal for RAG to lose to Context Stuffing
RAG 77 points vs Context Stuffing 93 points, a difference of 16 points. On the surface, RAG looks "worse", but this comparison is unfair. Context Stuffing completely stuffs all files into the prompt. LLM can "see" all the information, and the answer quality is naturally the highest.The problem is that it won't fit.
The 64-level Context Stuffing already accounts for about 150K tokens, which is close to the upper limit of the 200K context window. The 462K tokens of 200 levels are 2.3 times the upper limit of 200K. It is physically impossible to do Context Stuffing. Even with a 1M context window, the cost of sending 462K tokens per query and context rot (accuracy decreases as the number of tokens increases) make it impractical.
The correct interpretation is: RAG uses 2% of the information volume (5 chunks vs all files) to achieve 83% effect (77 vs 93), and is the only solution that can handle this scale.
Hybrid Search is 31% better than pure keyword search
RAG Hybrid scored 77 points vs. pure keyword search scored 59 points, an increase of 31%. The biggest improvement is the cross-file issue (Type B: 3.20 vs 1.80). The semantic understanding of vector search makes up for the semantic ambiguity blind spot of BM25.
But Hybrid doesn’t completely solve the problem either. There is still a gap between Type B's 3.20 points and Context Stuffing's 4.60 points. The reason is that top-5 chunks are not enough for cross-document questions — the answer might be spread across 8-10 paragraphs.
Highlights and Weaknesses of RAG
Highlights: Q7 fact-checking mechanism gets full marks
This is the question where Hybrid Search works best. The system finds complementary information from three different files - article-writer's CLAUDE.md has verification steps, multi-agent articles are designed with human-machine division of labor, and content-pipeline articles have verification dimension definitions. The three chunks are put together to completely cover the entire fact-checking mechanism.
This is the value of Hybrid Search: Vector search finds semantically relevant "fact-checking" passages, BM25 finds passages containing precise terms, and RRF combines the two results, resulting in higher coverage than either search alone.
Weakness 1: BM25’s keyword trap
Q8 asked "Which capabilities of the target job position does the Act 1 project align with?" RAG only scored 1 point. The reason is that BM25 was misled by the word "Act" and matched an article about the US Chip Act (CHIPS and Science).Act) article, and an article on "AI Manhattan Project". 3 out of 5 chunks are completely irrelevant.
This is a side effect of BM25 in Hybrid Search: it will push high-frequency but contextually incorrect matches into the candidate pool of RRF. If the correct paragraph is not found on the vector search side, the final result will deviate.
Weakness 2: Difficulty retrieving abstract questions
Q6 asked "Technology selection decisions and reasons" and only got 2 points. The problem is that "technology selection decision" is an abstract concept, and there will not be a paragraph in the document that clearly says "this is a technology selection decision." Relevant information is scattered in multiple paragraphs in multiple files such as HANDOFF, CLAUDE.md, troubleshooting, etc., and each paragraph only covers one aspect. The Top-5 limitation allows the system to only capture the tip of the iceberg.
Best solutions at different scales
| Data size | Recommended plan | Reason |
|---|---|---|
| < 50 documents / < 50K tokens | Context Stuffing | It works best if you plug it in directly, no need to build any pipeline |
| 50-100 copies / 50K-150K tokens | Context Stuffing (if it fits) | It is still feasible under 1M context window, and the effect is better than RAG |
| 100-200 copies / 150K-500K tokens | It depends on the situation | If you have 1M context window and can accept the cost, you can continue to stuff it; otherwise you need RAG |
| > 200 copies / > 500K tokens | RAG（Hybrid Search） | Context Stuffing is not physically feasible, RAG is the only option |
The key judgment point is not only "whether it can be plugged in", but also cost and context attenuation. Even if 1M context window can fit 462K tokens, the API fee of sending so many tokens for each query, as well as the accuracy degradation as the number of tokens increases, need to be taken into consideration.
What this means
The most important conclusion of this set of experiments is not that "RAG is worse than Context Stuffing", butDifferent solutions are suitable for different scales, and there is no one-size-fits-all solution。
If your knowledge base is small enough to fit into the context window, just stuff it into it instead of spending time building a RAG pipeline. If your knowledge base is too big to fit in, then RAG is not optional but required — a score of 77 is not perfect, but a score of 0 (cannot answer because it doesn’t fit in) is worse.
Within RAG, Hybrid Search is 31% better than pure keyword search. This is not a "slight improvement" but a "qualitative difference." Keyword search crashes to 1.80 on cross-file issues, and Hybrid pulls at least 3.20. Semantic retrieval is not "better" but "necessary".
Run baseline first and then build the system.If I had jumped into RAG from the beginning, I wouldn’t have known how strong Context Stuffing is on small data sets, or what the failure modes for keyword searches look like. Run it with the simplest method first and let the data tell you where the problem is. It is much more efficient than building the system with your eyes closed.
77 points is not the upper limit, but v1 baseline — four optimization directions
77 points is not the upper limit of RAG, but the baseline of v1. Clear optimization path:
Add Top-K.Increase it from 5 to 10 to see how much it improves for cross-file problems (Type B) and inference problems (Type C). The price is more noise and more token consumption.
Add Reranker.After RRF merging and before sending to LLM, a cross-encoder is used to refine the top-K results. The calculation amount of Reranker is larger than that of embedding, but it looks at the query-document pair instead of independent embedding, and the accuracy is higher.
Adjust Chunk Size.Test the performance difference between 300 and 800 characters. 300 might make the exact query more accurate, 800 might make the context across paragraphs more complete.
Upgrade Embedding Model.Upgrade from multilingual-MiniLM (384 dimensions) to bge-m3 (1024 dimensions) to see how much score improvement the improvement in semantic accuracy can bring.
Practical questions and boundaries
Is RAG 77 a good score?
As v1 baseline it is a reasonable starting point. A score of 4.60 for the single file question (Type A) is close to the level of Context Stuffing. Weaknesses focus on cross-file and inference issues, and these happen to be the directions with the most room for optimization (increasing Top-K, adding Reranker). 77 points is the floor not the ceiling.
Context Stuffing 93 points, why not just use it?
Because it cannot scale. The score of 93 was achieved on the basis of 64 documents. When the file size grows to 200 copies (462K tokens), Context Stuffing cannot fit into the 200K context window. Even with a 1M context window, the cost and context decay make it impractical. RAG solves the problem of "my knowledge base is growing, and I need a solution that can scale."
How to solve the keyword trap of BM25?
In the short term, you can use stopword filtering (add highly ambiguous words such as "Act" to the stop word list). In the mid-term, a Reranker can be added to allow the cross-encoder to reorder after seeing the complete query-document pair and downgrade the results that do not match the context. In the long term, query expansion can be considered - using LLM to rewrite user questions into multiple variations to increase search coverage.
What to take away
The article's value is in the evidence and trade-offs behind what 200 Files Revealed About RAG Data Preparation, not in treating the conclusion as universal.