Project · Experimental research

Enterprise RAG Evaluation

A sequence of controlled experiments that separates data, retrieval, answer-generation and evaluation failures in an enterprise RAG system.

Do not optimise RAG before locating the failure

Enterprise RAG teams often choose a vector database, embedding model or framework before proving that retrieval is the real problem. This project established baselines with 200 documents and 20 questions, then changed only the failure point supported by evidence.

Leave fashionable techniques out until the evidence asks for them

Reranking, hybrid search and larger infrastructure were not added without a diagnosed need. Controlled changes made each improvement attributable.

From deciding whether RAG is needed to locating the failure

Keyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.

StageWorkResult

BaselineKeyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.59 / 88 / 93

RAG v1Keyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.77

Diagnose 100 retrieved passagesKeyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.

Targeted optimisationKeyword search, long-context baselines and RAG were tested on the same material. One hundred retrieved passages were manually labelled before query rewriting and Top-K were adjusted.88

The score improved only after the failure was isolated

RAG v1 did not automatically beat every baseline. Targeted retrieval changes raised the score from 77 to 88 and reduced failed questions from five to two.

Evidence behind the case

Seven Chinese experiment articles preserve the baseline, failure analysis and optimisation sequence. English editions remain pending editorial and SEO review.