What RAG actually looks like when it works inproduction.

Every consulting call about AI starts the same way these days. Someone has seen a demo. The demo was beautiful. They want that demo, but for their company. They have docs, contracts, support tickets, a product catalog — whatever — and they want a chatbot that knows about it.

The demo, almost always, was made by hooking up OpenAI's API to a vector database and a few hundred PDFs. It took an afternoon. It is also nothing like a production RAG system.

The demo vs the reality

Here is what a demo RAG system does well:

• Answers obvious questions about content that was clearly written down in one place
• Hallucinates confidently when it cannot find the answer
• Costs nothing to run because no one is using it
• Has no concept of permissions, freshness, or accountability

Here is what a customer actually wants:

• Answers questions where the answer is spread across five documents that contradict each other
• Says "I don't know" when it doesn't know, instead of inventing answers
• Costs less than the person it is replacing
• Respects who is allowed to see what
• Tells them when its source documents are outdated
• Logs every answer so a human can verify and correct it

The gap between these two lists is where 90% of the work lives. The model is the easy part. Retrieval, evals, permissions, and feedback loops are where you spend the next eight weeks.

Retrieval is the hard part

Pick the right chunks and even a small model gives you good answers. Pick the wrong chunks and even Claude Opus will confidently misinform you, because that's what's in the context.

What we have learned the hard way:

Chunking is content-specific

The default LangChain recursive text splitter is fine for blog posts. It is terrible for contracts (where a clause might be 300 words and meaningless without the preceding definitions) and worse for tables (which it shreds). Every RAG project we have done has needed custom chunking logic per document type. For Northwind Legal we wrote a clause-aware chunker that respects contract section boundaries. For a hospital we wrote one that preserves drug-dose tables intact.

Pure semantic search misses obvious matches

Embeddings are good at meaning. They are surprisingly bad at proper nouns, part numbers, and dates. "Show me the SOW for project Atlas signed in March" should return the SOW for project Atlas, but pure cosine similarity often surfaces three unrelated documents that talk about "Atlas" or "March" abstractly. Hybrid search (semantic + BM25) closed this gap in every project we've done. It's not optional.

Rerankers earn their keep

Pull 30 candidates with hybrid search, then rerank with Cohere or a cross-encoder, then send the top 5 to the model. This three-stage pattern was the single biggest accuracy lift we got on the Civic Bank churn explanation system. The reranker catches false positives that retrieval can't distinguish.

Evals or you are flying blind

The thing nobody warns you about: once your RAG system is in production, every prompt change, every chunking tweak, every retrieval parameter is now risky. You will spend the rest of the project asking "did I just make this worse?" — and you cannot answer without evals.

Our minimum eval setup for any production RAG project:

• 50–200 hand-labeled question-answer pairs from real users (not synthetic)
• Automated retrieval metrics: recall@k, MRR
• LLM-judge accuracy scoring on a held-out set, with human spot-checks weekly
• A dashboard showing accuracy drift week-over-week

It feels like overkill until your first regression. Then it feels like the cheapest thing you ever built.

Cost and latency are product decisions

A RAG call that takes 8 seconds is fine for an internal research tool. It is a disaster for a customer-facing chatbot. A call that costs $0.20 is fine when a lawyer is using it. It is unviable for a support agent answering 10,000 questions a day.

We decide cost-and-latency targets before we pick the model. Then we work backwards: how much context can we afford? How many retrievals? Do we need a cheap classifier in front to route easy queries to a small model? On the Helio Health voice booking system we used a tiny model to classify intent (book / cancel / question / handoff) and only routed the "question" branch through full RAG. Average cost dropped 70% and we kept the accuracy on the path that mattered.

What we would do differently

If we were starting fresh on a RAG project today:

1. Build the eval set first. Even before retrieval. Even before picking a model. The eval set forces you to define what "good" looks like in concrete terms, and it pays back ten times over.
2. Use hybrid retrieval from day one. Don't wait for accuracy problems to discover that pure vector search isn't enough.
3. Add the reranker before adding more documents. Bigger corpora make weak retrieval worse, not better.
4. Plan for feedback loops in the UI. A thumbs up/down button and a "this answer was wrong" path. Without this you cannot improve the system over time.
5. Be ruthless about scope. "Answer any question about our company" is a project that never finishes. "Help an RM understand why this customer was flagged" is a project that ships.

If you're building something like this and want to talk it through, drop us a note. We've made most of the mistakes and we're happy to share which ones to skip.