This analysis organizes the evidence behind query Rewriting Raised This RAG Score from 77 to 88, then explains the practical implications, trade-offs and current limits.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
The previous article diagnosed that 60% of the quality problems in the RAG system come from the retrieval end, Generation Failure = 0. What to do next in this record: Evaluate three optimization directions, decide to do only two of them, and after execution, the score is pulled from 77 to 88 (+14.3%), and Retrieval Failure is reduced by 60%. A complete "diagnosis → decision → execution → verification" loop.
Three optimization directions: do Query Rewriting and Top-K, do not do Reranker and Metadata Filtering
Based on the failure mode, there are three clear optimization directions. But not everyone should do it.
Query Rewriting - decide to do
Corresponding questions: Among the 5 Retrieval Failure questions, the root cause of 3 questions is "there is a gap between the language of user questions and the language of the knowledge base".
The user asked about "technology selection decision", but the knowledge base said "complete headless CMS architecture" - semantically related but the wording is completely different. The user asked "Act 1 project alignment target position", but BM25 matched "Act" to "CHIPS and Science Act".
Solution: Add a layer of "translation" before the search - let LLM rewrite the user question into 2-3 versions, each version describing the same information need from a different perspective. The rewritten versions are searched separately, and the results are merged to remove duplicates. It is equivalent to using multiple search queries to cover the same question, greatly increasing the probability of finding the correct paragraph.
The expected effect is high, the implementation cost is low, and the risk is small. Do.
Top-K increases from 5 to 10 – decide to do; Reranker – decide not to do
Corresponding question: In Question 4 Coverage Failure, the correct paragraph exists in the knowledge base but is ranked between 6th and 15th. Top-K=5 excludes them.
Increasing Top-K to 10 is the most straightforward solution — giving more paragraphs a chance to be included.do.
Reranker (using cross-encoder to refine the top-20) can theoretically further improve the accuracy, but I decidedDon't do it. The reason is that I only have a test set of 20 questions. If the three variables of Query Rewriting + Top-K + Reranker are adjusted at the same time, the score increases and I can’t tell which one contributed. The sample size of 20 questions is too small, and changing too many things at the same time will easily lead to overfitting (overfitting) - you think you are optimizing the system, but in fact you are just fitting these 20 questions.
Controlling variables is more important than chasing the highest score.
Metadata Filtering — decide not to do it
Corresponding problem: The search for reasoning questions is interfered by the content of the article. If the search scope can be limited first (only architecture files are searched, not articles), the accuracy will be improved.
The reasons for not doing it are the same as above: there are too few 20 questions, and Metadata Filtering needs to do intent classification first (to determine whether the question is an "architectural problem" or a "general problem"), which itself is another subsystem that needs to be debugged. In the PoC stage, optimize the maximum leverage first, confirm it is effective, and then add layers.
Query Rewriting’s four rewriting strategies, each targeting different retrieval failure modes
For each user question, the system will generate three rewritten versions. Including the original question, a total of four queries were used for hybrid search (Hybrid Search), and the results were combined and deduplicated to obtain the top-10.
There are four rewriting strategies, targeting different retrieval failure modes:
Strategy 1: Add architecture file keywords.Add "CLAUDE.md HANDOFF.md SKILL" to the rewrite. This is equivalent to telling BM25 to "go to the architecture document" and solves the problem of vector search being attracted by the content of the article. This is the single most effective strategy, directly contributing +5 points.
Strategy 2: Expand the internal terminology of the project.Rewrite "Act 1" as "Initial stage of the gwarket project" to prevent BM25 from being interfered with by homographs.
Strategy Three: Add technical terms.Add "Deploy Hook rebuild" when rewriting the "automatic listing process" so that searches can match specific technical descriptions.
Strategy 4: Remove distracting question words.Keep nouns and concepts, and remove words such as "how", "why" and "which" that are not helpful for searching.
RAG v2 vs Context Stuffing: The remaining 5 points difference comes from data quality, not algorithm
RAG v2 scored 88 points, tying R1 Context Stuffing (88 points in 14 gears) and only 5 points behind R2 (93 points in 64 gears). However, RAG v2 can handle 200 files/462K tokens, which is 3.8 times the amount of data of R2.
The remaining 5-point gap is concentrated in two questions:
Q8 (2 points vs R2’s 3 points): "Act 1 Align job vacancies with capabilities". BM25 is still interfered with by "Act" to the chip bill. And there may be no PRD files in the knowledge base at all - this is not an algorithm problem, it's a data coverage problem. Adding a PRD file will solve the problem.
Q13 (3 points vs R2’s 4 points): "Adjustment of expanded content strategy". The semantics of "AI technology industry analysis" and "enterprise digital transformation" in the question are too strong, and even after rewriting, the search still leads to industry analysis articles. This requires more advanced technology (HyDE or intent-based routing) to solve.
Conclusion: The remaining gaps are mainly due to data quality issues (lack of PRD) and extreme semantic pollution, not to the capabilities of the RAG pipeline itself.
What this means
Optimize Retrieval for the highest ROI
The practical significance of Generation Failure = 0 is:Don’t waste time switching to a stronger LLM. The quality bottleneck of the RAG system is on the retrieval end. Spending budget and energy on Query Rewriting, search strategy optimization, and knowledge base quality will be far better than upgrading LLM.
The data this time directly proves this: only changing Query Rewriting + Top-K, leaving LLM, Embedding Model, and vector database unchanged, the score dropped from 77 to 88.
"Deciding what not to do" is as important as "deciding what to do"
Reranker is theoretically useful, and Metadata Filtering is also theoretically useful. But doing three optimizations at the same time on the 20-question test set, you can’t tell where the effect comes from. Worse, you might be overfitting those 20 questions — the system performs well on those 20 questions, but not on a different set of questions.
In the PoC stage, only change one or two variables at a time, verify the effect with data, and then decide what to change next. This is much better than doing it all in one go and not knowing why it works (or why it doesn’t).
Limitations of the 20-question sample
I have to be honest: the test set of 20 questions is too small. An improvement of +11 points is significant, but if a different batch of 20 questions were used, the improvement might be different. If it is a production environment, a test set of at least 100-200 questions is needed to cover a wider variety of problem types and edge cases.
But 20 questions are enough to verify the direction of the PoC stage - it tells you that Query Rewriting is worth investing in, Retrieval is the right optimization direction, and Generation does not need to be touched. These directional conclusions will most likely not be reversed on a larger test set.
Conclusion of the last round of diagnosis: All quality problems point to the search end
The conclusion of the last round of quality diagnosis is clear:
- Generation Failure = 0: LLM Answering ability is not a bottleneck. You can answer well as long as you are given the right information.
- Retrieval Failure = 5 questions (33%): Completely irrelevant paragraphs found
- Coverage Failure = 4 questions (27%): Some relevant passages were found but the information is incomplete.
- All quality problems point to the retrieval side, and improving Retrieval is the only lever
Now that the problem is clear, the next step is to decide how to change it.
Optimization results: 8 questions improved, 0 questions regressed, Retrieval Failure reduced by 60%
Six-party comparison table
| method | total score | Type A single file |
Type B Across files |
Type C reasoning |
Type D refuse to answer |
Can handle scale |
|---|---|---|---|---|---|---|
| R1: Context Stuffing 14 files | 88 | 4.80 | 3.80 | 4.00 | 5.00 | 14 files / ~25K tok |
| R2: Context Stuffing 64 files | 93 | 4.80 | 4.60 | 4.40 | 4.80 | 64 files / ~121K tok |
| B: Keyword search 64 files | 59 | 3.80 | 1.80 | 2.40 | 3.80 | 64 gears / top-5 |
| RAG v1：Hybrid Top-5 | 77 | 4.60 | 3.20 | 3.20 | 4.40 | 200 files / ~462K tok |
| RAG v2：Rewrite + Top-10 | 88 | 5.00 | 4.20 | 3.80 | 4.60 | 200 files / ~462K tok |
key figures
| indicator | v1 | v2 | improve |
|---|---|---|---|
| total score | 77 | 88 | +14.3% |
| Type A average | 4.60 | 5.00 (full score) | +0.40 |
| Type B average | 3.20 | 4.20 | +1.00 |
| Retrieval Failure | 5 questions | 2 questions | -60% |
| No failed questions | 6 questions | 11 questions | +83% |
Scores improved on 8 questions and declined on 0 questions. The biggest improvement is Q6 (technical selection), from 2 points to 4 points. After Query Rewriting added the "CLAUDE.md HANDOFF.md SKILL" keyword, the search successfully found gwarket's architecture description instead of semiconductor acquisition analysis.
Retrieval Precision Changes
The strict precision of Type B (cross-document questions) increased from 24% to 36% (+12%), and the corresponding answer quality increased from 3.20 points to 4.20 points (+1.00).The positive correlation between Precision and answer quality has been verified again in v2.
Practical questions and boundaries
How much delay and cost does Query Rewriting add?
Going from 1 search to 4 searches per question (original + 3 reworded versions), a 4x increase in search volume. But at the scale of 1,880 chunks, a single search itself only takes milliseconds, and 4 times is not noticeable. The rewriting itself uses rules (not calling LLM), so it does not increase API costs. If LLM rewriting is used instead, there will be an additional delay and expense of an LLM call per question, and ROI needs to be evaluated.
Why does Type A get perfect scores but Type C only gets 3.80?
Type A means "the answer is in a certain document", and the search only needs to find that paragraph. Type C is a reasoning question, which requires you to understand the whole system and then make inferences. However, the search can only give you 10 fragments - which is equivalent to asking you to guess the entire picture by only looking at 10 puzzle pieces. This is an essential limitation of chunk-level retrieval and cannot be solved by Query Rewriting. Requires document-level retrieval or larger context window.
If optimization continues, what is the theoretical upper limit?
Adding PRD files (addressing Q8) expects +2 points, adding HyDE or intent-based routing (addressing Q13) expects +1-2 points, adding Reranker expects +1-2 points. The theoretical upper limit is about 92-95 points, which has the potential to exceed Context Stuffing's 93 points - and can handle 3.8 times the amount of data.
What to take away
The article's value is in the evidence and trade-offs behind query Rewriting Raised This RAG Score from 77 to 88, not in treating the conclusion as universal.