This analysis organizes six Decisions to Make Before Building a RAG Pipeline into a practical comparison of evidence, decisions and current limits.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
Building a RAG pipeline is not difficult, but the difficulty lies in making selection decisions at every step. How to cut Chunking, which model to use for Embedding, which vector database to choose, and how to configure the search strategy - a complete record of the consideration process of six technical decisions, reasons for selection and trade-off.
Why are technical decisions more important than implementation?
The RAG pipeline code itself is not complex. Use Python to write a pipeline of "File → Segment → Embedding → Save Vector Database → Search → Group Prompt → LLM Answer" and it can be run in a few hundred lines.
But there is a selection to be made at every step, and there is a trade-off in every choice. If you choose the wrong link, the subsequent effects will have a ceiling no matter how you adjust them. This article examines the six key decisions I made when building the RAG pipeline. The focus is not on what I finally chose, but onWhy did you choose this way? What did you sacrifice? Under what circumstances should you choose another option?。
The pipeline structure looks like this: the knowledge base document is segmented by the chunker, indexed by the indexer and stored in ChromaDB. When querying, the most relevant paragraphs are found through hybrid search (Hybrid Search) and vector + BM25 + reciprocal rank fusion (RRF, Reciprocal Rank Fusion), forming a prompt and giving it to the LLM for answer.
Decision 1: Chunking Strategy—Hybrid Cutting
what was considered
Four mainstream cutting methods:
- fixed length: Cut into a section every N characters. Simple violence, but it could cut an entire concept in half.
- Cut by title: Using Markdown
##The title serves as a dividing point. Semantic integrity is preserved, but paragraph sizes can vary significantly if the content under the heading is too long or too short. - Semantic cutting: Use embedding to detect semantic turning points. Gives the best results but is computationally expensive and prone to over-cutting on short files.
- hybrid: Cut according to the title first, and then cut the paragraphs exceeding the upper limit with a fixed length. Take into account semantic completeness and paragraph size consistency.
What did you choose and why?
choosehybrid(Markdown title + fixed 500 characters). The reason is that my knowledge base has two completely different structures of files: Chinese blog articles have long paragraphs and few titles, and English technical documents have short paragraphs and many titles. Pure fixed length will break the title structure of English technical documents, and pure title cutting will produce too large chunks for long Chinese paragraphs. A hybrid strategy can do both.
what was sacrificed
There is an extra layer of logic than a purely fixed length, and some paragraphs may be cut off unnaturally at the title. But in a knowledge base that mixes Chinese and English, this is the most balanced choice.
Decision 2: Chunk Size — 500 characters + 100 characters Overlap
what was considered
The chunk is too large: Each paragraph contains too many topics, and it is easy to match the noise in the paragraph instead of the core content when searching. And large chunks occupy more context windows.
Chunk is too small: context is lost, a paragraph may only contain fragments of a sentence, and the semantics are incomplete. Even after searching for it, I don’t know what it is talking about.
The role of overlapping sections (Overlap): adjacent paragraphs overlap part of the content to ensure that the cut off sentences are complete in at least one paragraph.
What did you choose and why?
500 characters + 100 characters overlap as a starting point. The reason is that the amount of information per character in Chinese is higher than that in English (one Chinese character is approximately equal to the semantic amount of 2 English characters), and 500 characters in Chinese is approximately equal to the amount of information in 1,000 characters in English. In addition, the minimum chunk threshold is set to 50 characters. Fragments below this length will be merged into the previous section to avoid meaningless indexes.
what was sacrificed
The context of long paragraphs may be chopped up. If a technical concept spans 800 characters, the 500-character chunk will cut it into two pieces, and only half will be found during the search. This is the focus of subsequent optimization—testing the performance difference between 300 and 800 characters.
Decision 3: Embedding Model — multilingual-MiniLM
Four alternatives
| model | Dimensions | Chinese support | cost | Remarks |
|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | weak | free local | English is the most popular, but Chinese is less effective |
| multilingual-MiniLM-L12-v2 | 384 | good | free local | 50+ language support, widely used by the community |
| bge-m3 | 1024 | Excellent | free local | Best effect but large model and slow speed |
| jina-v3 | 1024 | Excellent | API payment | Works well but requires external API calls |
What did you choose and why?
chooseparaphrase-multilingual-MiniLM-L12-v2. Four reasons: First, the Chinese and English bilingual support is good, and my knowledge base is a mixture of Chinese and English; second, the 384-dimensional vector is lightweight enough, and 1,880 chunks can be processed in 22 seconds on the CPU (85.8 chunks/s); third, it is completely free for local execution and does not require an API key; fourth, it is widely used by the community, and the chance of being cheated is low.
what was sacrificed
The effect is not as good as bge-m3 or jina-v3. A 384-dimensional vector is not as expressive in semantic details as a 1024-dimensional vector. But in the proof-of-concept (PoC) stage, speed and freeness are more important than effectiveness—run it first, and then upgrade the model after confirming that the pipeline is feasible.
Decision 4: Vector database — ChromaDB
three candidates
| Plan | Features | Suitable for the scene | cost |
|---|---|---|---|
| FAISS | Pure library, you have to manage the storage yourself | Embed into existing systems | free |
| ChromaDB | Lightweight database, running locally, built-in persistence | PoC, small project | free |
| Pinecone | Cloud hosting, no operation and maintenance | Production environment, large scale | Pay |
What did you choose and why?
chooseChromaDB（PersistentClient）. Reasons: Locally free, Python native integration is good (a collection can be created with one line of code), built-in persistence does not require you to manage data storage, and the performance is fully sufficient in the PoC scale (< 10K files). ChromaDB takes up only 11.2 MB with 1,880 chunks.
what was sacrificed
Not suitable for large-scale deployment in production environments. If the knowledge base grows to millions of files in the future, you will need to switch to Pinecone, Weaviate or Qdrant. But during the validation phase, spending money on cloud services is a waste—confirm that RAG works in your scenario before investing in infrastructure.
Decision 5: Search Strategy — Hybrid Search (Vector + BM25 + RRF)
three options
- Pure vector search: Only semantic matching of embedding is used. It is good at synonyms and queries with similar meanings, but may be inaccurate for specific nouns (API names, version numbers).
- Pure BM25: Match only keywords. It's strong on precise nouns, but has a serious problem with semantic ambiguity - the same Chinese word has completely different meanings in different contexts.
- Hybrid: Vector + BM25 find top-20 each, and use RRF (k=60) to merge into the final top-K.
What did you choose and why?
chooseHybrid. In the previous baseline experiment, pure keyword search only scored 59 points (out of 100) on the mixed Chinese and English knowledge base, and the cross-file problem collapsed to 1.80 points. Semantic ambiguity is fatal. Hybrid allows vector search to handle semantics and BM25 to handle exact matching. RRF only uses rankings but not scores when merging, bypassing the problem of different magnitudes of scores for the two searches.
BM25 is implemented using rank_bm25 (BM25Okapi), which is lightweight and pure Python and does not require additional services. The k=60 of RRF uses the value recommended in the original paper and is not tuned for this knowledge base.
what was sacrificed
One more layer of BM25 computation and RRF merging than pure vectors, but no latency at all at 1,880 chunks. The real sacrifice is that BM25’s index does not support real-time updates, and needs to be rebuilt every time a new file is added—acceptable in the PoC stage, but may need to be replaced with ElasticSearch in the production environment.
Decision 6: Top-K — Use 5 first
Three options of trade-off
- Top-3: Small amount of information and low noise. Suitable for questions whose answers focus on a single paragraph. But cross-document questions are almost impossible to answer completely in just 3 paragraphs.
- Top-5:Balance point. Most single file problems are sufficient, and some cross-file problems can also be covered.
- Top-10: Large amount of information and high cross-file coverage. But there’s also a lot of noise—irrelevant passages can mislead the LLM.
What did you choose and why?
Initial selectionTop-5. Vector search and BM25 each select the top-20 as the candidate pool for RRF. After RRF is merged, the final top-5 is selected and sent to LLM. 5 is a safe starting point — in the previous baseline, the top-5 keyword search could already handle most Type A problems, but it was not enough for cross-file problems.
what was sacrificed
If the answer is spread over more than 5 paragraphs (common with Type B cross-document questions), key information will be missed. This is the first direction for subsequent optimization — increase it to top-10 to see if cross-file issues are improved.
From file to queryable status in less than a minute — Pipeline measured data
| indicator | numerical value |
|---|---|
| Number of knowledge base files | 200 (199 processed successfully, 1 skipped due to Windows long file name) |
| Total number of chunks | 1,880 |
| average chunk length | 308 characters |
| Embedding dimensions | 384 |
| Embedding takes time | ~22 seconds (CPU, 85.8 chunks/s) |
| ChromaDB size | 11.2 MB |
The entire pipeline goes from file to queryable status in less than a minute. This is the advantage of choosing a lightweight solution in the PoC stage - iterate quickly and adjust immediately when problems are discovered, without having to wait for half an hour of embedding to be completed.
What this means
There is a common logic among the six decisions:Choose those who can run first, then choose those who run well.
Embedding uses multilingual-MiniLM instead of bge-m3 because the pipeline needs to be verified first. Use ChromaDB instead of Pinecone for the vector database because PoC should not spend money on infrastructure. Chunk size starts with 500, because you need a baseline to know what will happen if you increase or decrease it.
Behind every "use this first" there is a judgment of "when should I change it". When the semantic accuracy of multilingual-MiniLM is insufficient, upgrade bge-m3. When ChromaDB's query performance slows down with millions of files, switch to Pinecone. Increase to top-10 when top-5 misses too many cross-file issues.
Technology selection is not a one-time decision, but a series of "currently best + known upgrade paths." What I'm afraid of is not choosing the wrong one. What I'm afraid of is not knowing when to change it after I choose it.
Practical questions and boundaries
Do these options apply to all RAG projects?
No. These selections are for a 200-document, mixed Chinese and English, PoC stage knowledge base. If your knowledge base is pure English, has millions of documents, and needs to be updated immediately, the best choice for each decision will be different. But the decision-making framework is universal: first understand the characteristics of your data, and then choose an option.
Why not just use the best models and tools?
Because the goal of the PoC stage is to verify feasibility, not to pursue the best results. If you use the best model but have problems with the pipeline design, the effect will be the same. First use a lightweight solution to run through the entire road, confirm where the bottleneck is, and then upgrade to target the bottleneck. This saves time and money rather than going all-in with the top specs from the beginning and then finding out you went in the wrong direction.
How much better is Hybrid Search than pure vector search?
In my tests, Hybrid Search (77 points) outperformed pure keyword search (59 points) by 18 points. The comparison with pure vector search has not been done yet, but it can be inferred from baseline data that pure vector will be weaker than Hybrid in precise noun search. The cost of Hybrid (an extra layer of BM25) is negligible at the scale of thousands of chunks, so there are almost no disadvantages in choosing Hybrid by default.
What to take away
The article's value is in the evidence and trade-offs behind six Decisions to Make Before Building a RAG Pipeline, not in treating the conclusion as universal.