Enterprise RAG Evaluation

The question

When enterprise RAG quality is poor, is the cause the data, retrieval method, answer generation or the evaluation design?

Approach

1. Establish non-RAG baselines before adding system complexity.
2. Test retrieval methods on the same document set and questions.
3. Diagnose failed questions instead of relying on one aggregate score.
4. Change one retrieval decision at a time and rerun the evaluation.

Result

Testing with 200 documents and 20 questions isolated retrieval failures and improved the measured RAG score from 77 to 88.

Evidence

Seven linked experiments, 200 documents and 20 evaluation questions.

Do not optimise RAG before locating the failure

Enterprise RAG teams often choose a vector database, embedding model or framework before proving that retrieval is the real problem. This project established baselines with 200 documents and 20 questions, then changed only the failure point supported by evidence.

Leave fashionable techniques out until the evidence asks for them

Reranking, hybrid search and larger infrastructure were not added without a diagnosed need. Controlled changes made each improvement attributable.

From deciding whether RAG is needed to locating the failure

Keyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.

1. Baseline: 59 / 88 / 93
2. RAG v1: 77
3. Diagnose 100 retrieved passages
4. Targeted optimisation: 88

The score improved only after the failure was isolated

RAG v1 did not automatically beat every baseline. Targeted retrieval changes raised the score from 77 to 88 and reduced failed questions from five to two.

Evidence behind the case

Seven Chinese experiment articles preserve the baseline, failure analysis and optimisation sequence. English editions remain pending editorial and SEO review.