What building a production RAG system for legal documents actually requires

Most legal “AI search” demos die the first week a real firm uses them. The model sounds fluent. The citations are almost right. A partner pastes one of those almost-right paragraphs into a memo, and that is the last time anyone trusts the product.

We built Law4U as a retrieval system first and a language model second. The language model is allowed to talk only after the documents have been carved, stored, and fetched in a way a lawyer can audit.

Chunking is the product

A 120-page judgment is not a bag of paragraphs. It has a caption, a bench, a procedural history, holdings, and a disposition. If you split on token count, you cut a ratio in half and you lose the heading that told you which court said it.

Structure-aware splitting keeps those units together. We walk the document the way a clerk would: detect headings, keep numbered paragraphs intact, and attach the nearest parent heading to every chunk. The embedding then represents “what this holding is about,” not “page 47, middle.”

That extra heading metadata is what lets the retriever prefer a Supreme Court holding over a similar-sounding high-court remark.

Hybrid retrieval beat pure vectors

Pure nearest-neighbour search is good at “this feels like that.” Legal research also needs “this is the exact section.” Statute numbers, party names, and reporter citations are sparse signals. A vector model will happily return a neighbouring section that discusses the one you asked for.

We run both:

• keyword / text search for citations, section numbers, and proper names
• vector search for conceptual similarity
• a merge that refuses to drop a high-confidence keyword hit just because the embedding was lukewarm

The merge is boring on purpose. Boring is what survives a Monday morning when someone searches Section 138 and expects 138, not a neighbour that mentions cheques.

If the retriever cannot point at a span, the generator does not get to speak. “Cited” means a passage the user can open, not a vibe.

What “cited” has to mean

A footnote that says “see generally” is not a citation in this product. Each generated sentence that depends on the corpus has to carry the chunk id, the document id, and the page or paragraph range. The UI shows those as openable sources. If the model cannot attach one, the sentence is dropped or rewritten as the model’s own reasoning — labelled as such.

That rule is slower than “just stream the answer.” It is also why a lawyer will keep using the tool after the first wrong answer.

Isolation is not optional

Firms do not share a corpus. Tenants do not share a vector index. Uploads are tagged at the edge, queries are filtered before they ever reach the model, and a missing tenant claim is a hard failure, not a default to “all documents.”

The throughline is the same one we use on IoT platforms: take a messy real-world input and make it something other software can depend on. A legal PDF is just a noisier sensor.

If you are scoping a retrieval product, start with the document tree and the citation contract. The model is the last thing you pick.