A lot of "RAG in 10 minutes" tutorials follow the same shape: load some documents, chunk them, embed them, throw them in a vector store, and write a prompt template. It works well enough as a demo. It tends to fall apart the moment real users ask real questions.
The model is the easy part
By the time your retrieved context reaches the LLM, most of the decisions that determine quality have already been made:
- Chunking strategy. Fixed-size chunks are easy to implement and frequently wrong — they split a table from its header, or cut a procedure in half right before the step that matters.
- Embedding choice and domain fit. A general-purpose embedding model will quietly underperform on domain-specific terminology (legal contracts, medical notes, internal codebases) compared to one tuned or selected for that domain.
- Retrieval strategy. Pure vector similarity search misses exact-match cases — product codes, error messages, specific names — that a keyword or hybrid search would catch instantly.
- Re-ranking. The top-k results from a vector search are a list of "probably relevant" candidates, not a list of "actually relevant" ones. Skipping re-ranking means feeding the model noise alongside signal and hoping it sorts it out.
None of this is a prompting problem. You can write the most carefully engineered prompt in the world and it won't compensate for retrieving the wrong three paragraphs.
Treat it like a data pipeline, because it is one
The teams that get good results from RAG tend to treat it the way they'd treat any other data pipeline: with monitoring, with evaluation sets, and with the assumption that every stage can silently degrade. Concretely, that means:
- Building a small, hand-labeled set of realistic questions and expected source passages, before writing a single prompt.
- Measuring retrieval quality (precision/recall against that set) separately from generation quality. A system can retrieve perfectly and still generate a bad answer, or retrieve badly and still get lucky — you want to know which failure you're looking at.
- Re-evaluating the pipeline whenever the underlying documents, embedding model, or chunking logic changes — the same discipline you'd apply to a schema migration.
The takeaway
If your RAG system isn't performing well, the first place to look isn't the system prompt. It's the four steps that happen before the model is ever called. Fix retrieval, and a mediocre prompt will often produce a good answer. Fix only the prompt, and a broken retrieval layer will still hand the model the wrong information — just phrased more politely.
