Our voice agents at TeleVox AI answer business phone calls with retrieved knowledge inside a roughly 0.7-second response budget. On a live call there is no regenerate button. A wrong answer about opening hours or pricing is worse than no answer, because the caller believes it. That constraint forced us to treat RAG as an engineering discipline rather than a demo, and most of what we learned applies to any business knowledge base — support bots, internal assistants, customer-facing chat.
Here is how we build RAG systems that answer correctly, and where they fail when you don't.
Chunking that respects document structure
The default approach — split every document into fixed 500-token windows with overlap — is where most RAG quality problems start. Fixed windows cut tables in half, separate a heading from the paragraph that explains it, and split a policy's rule from its exception.
The rules we follow:
- Split at structural boundaries first. Headings, list items, table rows. Fall back to token windows only inside oversized sections.
- Keep tables and FAQ pairs atomic. A pricing table split across two chunks produces confidently wrong prices.
- Prepend a breadcrumb to every chunk.
Refund Policy > Enterprise Plans > Exceptionscosts a few tokens and rescues retrieval for chunks whose text alone is ambiguous. "This does not apply after 30 days" — what doesn't? - Store source and section with each chunk, so answers can cite where they came from and updates can delete precisely.
Chunk size is a tradeoff, not a constant. Small chunks retrieve precisely but strip context; large chunks carry context but dilute the embedding. We start around 300–500 tokens for prose, atomic for structured content, then let evals move it.
pgvector, because your data is already in Postgres
We run retrieval on pgvector inside Supabase Postgres. Not because it benchmarks best — dedicated vector databases will beat it at extreme scale — but because at the tens or hundreds of thousands of chunks a real business knowledge base contains, it is fast enough, and it removes an entire class of problems:
- One database. No sync pipeline between your source of truth and a second vector store, so no drift between them.
- Row-level security applies to retrieval. Our tenants' knowledge bases are isolated by the same RLS policies that protect the rest of their data. A retrieval bug cannot leak one customer's documents into another customer's answers.
- Transactions. Updating a document and its embeddings is atomic. Delete-then-insert in one transaction means retrieval never sees half-updated state.
An HNSW index gets you a long way:
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE tenant_id = $2 -- RLS enforces this anyway
ORDER BY embedding <=> $1
LIMIT 10;
We also keep a tsvector column and merge keyword hits with vector hits. Embeddings are weak on exact strings — SKUs, error codes, people's names — and keyword search catches what cosine similarity misses.
The retrieval failures you will actually hit
Demos fail in three predictable ways. Production systems need an answer for each.
Stale documents
The pricing page changed; the old embedding is still in the index. Retrieval now returns both versions and the model picks one. Treat ingestion as sync, not append: hash each source document, and when the hash changes, delete every chunk from that source and re-ingest inside a single transaction. Append-only ingestion is the most common cause of confidently outdated answers in RAG systems.
Near-duplicate chunks
Business content repeats itself — the same guarantee on five product pages, boilerplate headers, templated sections. All five near-copies land in your top ten and crowd out the one chunk that actually answers the question. Deduplicate at ingestion (embedding similarity above roughly 0.95 is a duplicate) and diversify at query time — maximal marginal relevance is simple to implement and noticeably improves multi-fact answers.
Questions that span chunks
"What's the difference between the Pro and Enterprise SLA?" needs two chunks that may not both surface for one query embedding. Two mitigations work reliably. Retrieve small, feed big: match on a precise chunk, then hand the model its whole parent section. And decompose comparison questions into sub-queries, retrieving separately for each side. Both add latency, which is why our voice stack applies them selectively — a phone caller will not wait out a five-hop retrieval chain.
Evals before shipping, not after
You cannot tune what you do not measure, and "we tried some questions and it seemed good" is not measurement.
Before a knowledge base goes live we build a golden set: 50 to 200 real questions with known correct answers, sourced from actual support tickets and call transcripts, not invented by the team that wrote the docs. Then we score two things separately:
- Retrieval hit rate. Is the chunk containing the answer in the retrieved set? If not, no prompt engineering will save you — fix chunking and search first.
- Groundedness. Does the generated answer stay within the retrieved text? An LLM judge given the answer and its sources works well enough here, spot-checked by a human.
The golden set must include questions the knowledge base cannot answer. The correct behavior is "I don't know" or an escalation — for a voice agent, a transfer to a human. Systems never tested on unanswerable questions learn to guess.
Re-run the suite on every re-ingestion. Retrieval regressions from an innocent docs update are real and otherwise invisible.
When fine-tuning is the wrong answer
Clients regularly ask us to fine-tune a model on their data so it "knows the business." For factual knowledge this is usually the wrong tool:
- Knowledge changes. Prices, policies, staff. RAG updates in one transaction; a fine-tune is a training run.
- Fine-tuning does not stop hallucination. It teaches form and style far more effectively than it stores facts. The model will answer in your voice and still invent your prices.
- No citations. RAG can show its sources; model weights cannot.
Fine-tuning earns its place for tone, output format, and domain phrasing — an agent that always speaks in your brand voice. Facts belong in the retrieval layer, where you can update, audit, and delete them.
What this looks like end to end
Structure-aware chunking, hybrid pgvector retrieval under RLS, sync-not-append ingestion, and a golden-set eval gate is not an exotic stack. It is a few weeks of disciplined engineering, and it is the difference between a demo and a system a business can put in front of customers — in chat through our AI solutions practice, or on live phone calls through our voice AI work, where the same retrieval layer has to perform inside a sub-second response budget.
If you are building a knowledge base for support, internal ops, or a voice agent — or fixing one that answers confidently and wrongly — get in touch. We start with your documents and your real questions, not a framework.