This analysis explains how to Diagnose RAG Retrieval Quality through the source's decisions, evidence and operating constraints.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
The RAG system scored 77 points, which was lower than Context Stuffing’s 93 points. But "low 16 points" is not a conclusion, but a starting point for diagnosis. After dissecting the retrieval quality of 100 chunks question by question, we found that: 60% of the quality problems come from the retrieval end, and LLM's answering ability is not a bottleneck at all. This article shows the complete diagnostic methodology and optimization derivation.
Why do we need quality diagnosis?
After the RAG system was launched, the first reaction of most people was to look at the total score. With a score of 77, lower than Context Stuffing, the conclusion seems to be "RAG is not good enough". But this conclusion has no action value—you don’t know what to change.
Is it because LLM’s answering ability is not good enough? Did you search and found the wrong thing? Or did you find the right thing but not complete enough? Each reason corresponds to a completely different optimization direction.It is a manager's instinct to look at the total score, and splitting the total score is what managers should do.
Diagnostic method: mark the relevance of each Chunk one by one
20 questions × 5 chunks per question = 100 chunks. Determine the relevance of each chunk to the problem one by one:
- ✅ Totally relevant: This chunk contains the core information needed to answer the question
- ⚠️ Partially related: Has some useful context, but not the core answer
- ❌ Not relevant: It has nothing to do with the problem at all, it’s just noise
After marking, calculate the retrieval precision (Retrieval Precision): how many of the 5 chunks for each question are ✅.
The lower the precision, the worse the answer quality - data support
| Question type | Strict Precision (✅) | Loose Precision（✅+⚠️） | average answer score |
|---|---|---|---|
| Type A — Single file | 36% | 76% | 4.60 |
| Type B — across files | 24% | 48% | 3.20 |
| Type C — Reasoning | 16% | 28% | 3.20 |
| Type D - outside the knowledge base | 0% | 16% | 4.40 |
Precision is highly related to answer quality.Type A's 36% precision corresponds to 4.60 points, and Type C's 16% precision corresponds to 3.20 points. The lower the Precision, the worse the answer quality. This is no coincidence — it tells you:Improving search accuracy can directly improve the quality of answers.
The Type D data also makes sense. The precision is 0% (there is no answer in the knowledge base), but the answer got 4.40 points (correct rejection). This means that LLM is very good at "judging insufficient information" - even if it gets a bunch of irrelevant chunks, it can still say "I don't know".
60% of quality problems come from retrieval, and Generation Failure is zero
Excluding 5 Type D questions (correct refusal is expected behavior), the remaining 15 questions are classified as failure modes:
| failure mode | Number of questions | Proportion | Description |
|---|---|---|---|
| No failures | 6 | 40% | Search and answer are both good |
| Retrieval Failure | 5 | 33% | The retrieved chunks are completely irrelevant and cannot be answered by LLM |
| Coverage Failure | 4 | 27% | Some relevant paragraphs found, but the information is incomplete |
| Generation Failure | 0 | 0% | No. As long as you provide the right information, LLM's answers will be of good quality. |
The most important finding: Generation Failure = 0.
This means that LLM’s answering ability is not a bottleneck at all. All quality problems come from the retrieval end - either the wrong thing is found (Retrieval Failure), or the thing found is not complete enough (Coverage Failure). 60% of quality problems point in the same direction:Improving Retrieval is the only lever to improve overall performance.
In-depth diagnosis of failure cases
Case 1: Q8 — 1 point (BM25 is misled by "Act" + vector is attracted by "Target")
The question asked was "Which competencies of the target job position does the Act 1 project align with?" Vector search matched "the ability to align targeted job openings" to "the strategy of the AI Manhattan Project"target” and “Maintaining technological leadershipAbility”. BM25 matched "Act" to "CHIPS and Science Act". 4 out of 5 chunks are completely irrelevant.
There are three root causes: BM25 homograph interference, vector search is misled by universal semantics ("goal", "ability"), and direct descriptions of Act 1 and job vacancy alignment may not exist in the knowledge base. Even if you increase the Top-K to 20, the correct passage is not included in any rankings — because the semantic signals contained in the question itself skew the search direction.
Case 2: Q6 — 2 points (correct paragraph exists but ranked 10-15)
The question asked about "technical selection decisions and reasons." There is an exact corresponding paragraph in the knowledge base ("Complete Headless CMS Architecture" of HANDOFF.md), but vector search ranks it at Rank 10-15. Because the semantics of "technology selection decision" are matched to "major decisions in the industrial competitive landscape" (a semiconductor article) and "choosing appropriate project management tools" (a tool evaluation article) - the embedding model captures the general semantics of "decision" and "selection", but ignores the qualification of "gwarket's technical architecture".
The solution to this problem is clear: increase Top-K to 15 to get the correct paragraph. Or use Query Rewriting to rewrite the "technology selection decision" into "what technical architecture is used on the gwarket website" to directly avoid the interference of common semantics.
Case 3: Q13 — 3 points (semantic pollution)
The question is "If we want to expand content strategy from AI technology industry analysis to enterprise digital transformation, what adjustments need to be made to tools and processes?". The core of the question is "tool and process adjustment", but 70% of the words in the question describe "expanding from what topic to what topic". The result is that search engines are dominated by descriptive words such as "AI technology industry analysis" and "enterprise digital transformation", matching "articles covering these topics" (semiconductor analysis, smart manufacturing, AI weekly reports) instead of "architectural documents describing tools and processes."
The data on the vector distance is very convincing: the distance of a Qualcomm-Intel acquisition analysis article is only 0.2892 (very close), and the really needed "source list and filtering rules" paragraph is not in the top-20 at all.Semantically "similar", logically completely unrelated.
Each optimization direction corresponds to a specific failure mode
Optimization 1: Query Rewriting (expected +5-8 points)
Corresponding questions: 3 of 5 Retrieval Failure questions.
Method: Before searching, let LLM rewrite the user question into a more specific search query - break the abstract question into specific sub-questions, expand the internal terms of the project ("Act 1" is rewritten as "Phase 1 Project"), and remove distracting descriptive words. If Q6 is rewritten as "What is the technical architecture of the gwarket website", you will almost certainly find the correct paragraph. The same reason applies when Q13 is rewritten as "Source List and Filtering Rules for Internet Information Experts".
Optimization 2: Increase Top-K to 10 + Reranker (expected +5-8 points)
Corresponding questions: 4 Questions Coverage Failure.
Method: The initial search has taken the top-20, but in the end only the top-5 were sent to LLM. Increase the final feed number to 10 to give more paragraphs a chance to be included. In addition, the cross-encoder is used as a reranker to refine the top-20 - the reranker looks at the query-document pair rather than independent embeddings, which has higher accuracy and can downgrade paragraphs that are "semantically similar but logically unrelated".
Optimization 3: Metadata Filtering (expected +2-3 points)
Corresponding question: Retrieval deviation for Type C reasoning questions.
Method: First classify the problem by intention - if it is an "architectural problem", limit the search scope to CLAUDE.md, HANDOFF.md, SKILL.md and other architecture files. In this way, you will not be disturbed by the content of articles in the knowledge base. Q11 asked about the Newsletter subscription function. You can directly find the deployment restriction description of code-conventions SKILL by restricting the search for architecture files.
theoretical upper limit
| Status | score | Description |
|---|---|---|
| Current RAG | 77 | v1 baseline |
| + Query Rewriting | 82-85 | Resolve Retrieval Failure |
| + Top-K 10 + Reranker | 87-93 | Resolve Coverage Failure |
| + Metadata Filtering | 90-97 | Close to or exceed Context Stuffing’s score of 93 |
What this means
The most valuable thing about this diagnosis is not the specific number;The diagnostic method itself。
"Mark chunk correlations question by question → calculate precision → classify failure modes → deduce optimization directions from failure modes" - This process can be used on any RAG system. You don’t need to guess “whether the LLM is not good enough” or “whether the embedding model needs to be changed”, the data will tell you where the problem lies.
The data this time is very clear:Generation Failure = 0。All quality issues come from Retrieval. This means that switching to a stronger LLM won't help, but improving your search strategy will directly translate into an improvement in answer quality. For managers, this is the rationale for resource allocation—spend budget on Retrieval optimization, not LLM upgrades.
Hybrid Search is not “better by adding it”. It has a complementary effect on 40% of the questions, but on the other 13% of the questions, the good results of BM25 are diluted by vector noise. The value of technical solutions is not based on the best case scenario, but on the average case and worst case scenario.
Hybrid Search only has complementary effects on 40% of the questions
Hybrid Search combines vector search and BM25, which is expected to be complementary: vector search handles semantics, and BM25 handles exact matching. But actual data shows that complementarity does not work every time.
| situation | Number of questions | Description |
|---|---|---|
| The two are complementary (1+1 > 2) | 3 | Each found different correct paragraphs, and the merged version is more complete. |
| Vector is better, BM25 contributes nothing | 3 | BM25 Can't find something or find noise |
| BM25 is better, but diluted by vector | 2 | BM25 has correct results, but is crowded out by vector noise during RRF merging |
| Both are worse | 4 | I can't find the correct paragraph, and I can't save it by merging it. |
| Both are good | 3 | Useful passages found |
The complementary effect was only effective in 40% of the questions (6/15).For another 2 questions, BM25 clearly found the correct result, but it was diluted to Rank 4-5 by a large amount of noise from vector search during RRF merging, which affected the final quality. Hybrid Search is not a panacea — when neither side can find the correct passage, merging just mixes two sets of incorrect results together.
Homograph Weaknesses of BM25
The most typical failure case: The question asked "Which capabilities of the target job position does the Act 1 project align with?" BM25 matched "Act" to an article about the US Chip Act (CHIPS and Science).Act) article. BM25 is a bag-of-words model, it cannot distinguish between "Act 1 = first phase of the project" and "Act = bill". This homograph interference is the most classic failure mode of BM25.
Practical questions and boundaries
Why is Generation Failure 0? Is LLM too strong?
Not exactly. Claude Opus 4.6 is indeed a strong model, but the more important reason is that this test question is not designed to specifically test the Generation ability scenario (such as questions requiring complex logical reasoning or mathematical calculations). In the context of knowledge base Q&A, the main task of LLM is to "organize answers according to given paragraphs", which is a relatively simple task for large language models. If you switch to a weaker model, Generation Failure may occur.
Is Retrieval Precision 20% too low?
A strict Precision of 20% may seem low, but there are two factors to consider. First, Type D’s 5 questions should not have correct chunks, which lowered the overall average. Second, the relaxed Precision (which includes partially relevant ⚠️ chunks) is 52%, meaning that more than half of the chunks are at least somewhat helpful. What really needs improvement is the strict Precision of Type B and Type C (24% and 16%), which is the focus of optimization.
Does the diagnostic method require manual labeling of each chunk? Not very time consuming?
20 questions × 5 chunks = 100 marks, takes about 1-2 hours to actually do. For the first quality diagnosis after a system goes online, the investment is worth it. Afterwards, an automated evaluation mechanism can be established - using LLM to make preliminary judgments on chunk correlation, and then sampling for manual review. But be sure to do it manually the first time, because you need to see failure cases with your own eyes to build up your intuition about the system's behavior.
What to take away
The practical value lies in the decisions and constraints behind to Diagnose RAG Retrieval Quality, not in copying one implementation without its context.