What "production AI" actually means here
A chat wrapper is a weekend. The hard part of shipping an LLM feature is everything around the model call: deciding what goes in the context window, proving where an answer came from, keeping the cost per question predictable, and deciding what the product does on the day the model API is down.
This demo is small on purpose — one document, one question box — but it makes all four of those decisions explicitly, and you can read every one of them in the source.
Retrieval: why there is no vector database
The obvious build is embeddings in a vector store. I didn't do that, and the reason is worth stating plainly because it's the kind of trade-off that comes up in real projects.
Retrieval here is TF-IDF scoring across the chunks of a single document. Every chunk is scored against the question, and the top three go into the context window. Within one document that is a strong baseline: the corpus is small, the vocabulary overlap between a question and its answer passage is usually high, and lexical matching handles the proper nouns, product codes and numbers that embeddings are famously mediocre at.
What it buys:
- →No embedding API call on upload, so ingestion is instant and free
- →No vector database to run, back up, or pay for
- →The demo works on a fresh clone with no API keys at all
- →Retrieval is deterministic and debuggable — you can see exactly why a chunk scored
What it costs: it will not match a paraphrase that shares no words with the source, and it does not scale to retrieval across thousands of documents, where embeddings genuinely win. That's the upgrade path, and the retrieval function is the only thing that changes.
The point isn't that TF-IDF is better. It's that "RAG" is not synonymous with "vector database", and picking the cheaper mechanism when it clears the bar is usually the senior decision.
Chunking is where quality is won or lost
Chunks are 500 characters with 100 characters of overlap, and the split points
are pushed to a sentence boundary — the chunker looks ahead 50 characters
for a . , ? or ! and breaks there, as long as that lands past the
halfway mark of the chunk.
Two deliberate details:
- →Overlap means a fact that straddles a boundary still appears whole in one of the two chunks. Without it, retrieval reliably fails on exactly the sentences that span a split.
- →Sentence-aware breaks stop the model being handed a fragment that begins mid-clause. A chunk that starts with "…which is why the deadline moved" is worse than useless: it's confidently misleading.
Chunks under 20 characters are dropped rather than stored, so trailing whitespace and page furniture never compete for a retrieval slot.
Citations: making the answer checkable
The retrieved chunks are numbered and injected as [1], [2], [3], and the
system prompt requires the model to cite those markers and answer only from
the supplied context. The chunk IDs travel back to the client alongside the
streamed answer, so each marker is a link to the passage it came from.
This is the part that makes an AI feature usable in a business setting. An uncited answer is a claim a user has to take on faith, and no one signs off on a quote or a contract clause on faith. A cited answer is a claim they can check in two seconds. The citation is not decoration — it is the feature.
Every question, the full answer, and the list of cited chunk IDs are written to the database. That gives you the audit trail you will want the first time someone asks "why did it say that?", and it's the raw material for evaluating retrieval quality later.
Cost and latency
The model is Claude Haiku, capped at 1024 output tokens, with exactly three chunks of context. That combination is what makes the cost per question predictable rather than open-ended: context is bounded by construction, not by how large the uploaded document happens to be. A 400-page PDF and a one-page memo cost the same to answer.
Responses stream token-by-token. The stream is teed — one branch renders in the browser, the other accumulates in memory and writes the finished answer to the database. The user sees first tokens immediately; the audit record is written once, after completion, without a second model call.
Haiku is chosen over a larger model deliberately. Synthesising three short passages into a cited paragraph is not a reasoning-heavy task, and paying frontier-model prices for it is how AI features end up with unit economics that don't work.
What happens when the model is down
This is the question most AI demos have no answer to.
If there is no API key — or, in production, when the provider is unavailable —
the endpoint does not error. It falls back to keyword extraction over the
same retrieved chunks: it pulls the sentences from the top chunk that share
terms with the question, returns them with the same [1] [2] [3] citation
structure, and streams them word-by-word so the interface behaves identically.
The answer is visibly worse, and it says so. But the product still works, the citations are still correct, and the user still gets to the passage they needed. Degrading to a worse answer beats a spinner and a 500.
That fallback also means the whole thing runs on a laptop with no credentials, which is why the demo can be open to the public without a key or a rate-limit nightmare.
The upgrade path, honestly
If this became a real product, the order of work would be: embeddings and a vector store once retrieval spans more than one document; a hybrid of lexical and semantic scores rather than replacing one with the other; an eval set of question/passage pairs so retrieval changes can be measured instead of eyeballed; and a cache on repeated questions, which in document Q&A is a large share of real traffic.
None of that changes the shape above. Chunking, citation, bounded context and a degradation path are the parts you have to get right first, because they're the parts that decide whether anyone trusts the output.