← All projects

LIVE DEMO

Internal and customer RAG chatbot

One document set, two audiences, and a source under every answer it gives

STACKPython 3.12 / FastAPI / Qdrant / PostgreSQL 16 / n8n 2.31.6 / OpenRouter / Docker

CONTEXTSpec project (Upwork brief)

DATE2026-07

Open recruiter demo ↗ rag-support-chatbot-demo.aaron.cv

EVIDENCE, NOT CLAIMS

Nine cases you can run yourself, plus your own question

The demo runs the delivered application and the delivered escalation workflow, on the same document set, the same score gate and the same citation validator as the source repository. Every scenario states what it expects before it runs, then shows the actual result, every check it made and the PostgreSQL rows behind it. Retrieval and the policy layer are deterministic and are labeled as such; what a model writes varies from run to run, and those checks are labeled separately. Three consecutive runs of all nine scenarios over the public URL on 2026-07-29 passed all 141 checks. The customer chat page of the application itself is open next to the console, so the interface can be used and not only read about.

141 of 141 checks passed over three consecutive runs on the live demo All nine scenarios, three times, over the public URL on 2026-07-29. 120 of those were policy checks and 21 were wording checks, and none of the 27 scenario runs missed a check.

7.9 s median scenario on the live demo, slowest 20.2 s 27 scenario runs on 2026-07-29 over the public URL. A scenario is one or two full turns including retrieval and a live model call, on a small VPS. In an earlier series the same day the model route itself failed on one turn: that run ended on a handover with reason provider_error and no invented figure. Not a production figure.

0.9414 nDCG@10 over the 40 question evaluation set Measured 2026-07-28. recall@5 0.9643, and 1.0000 on both floors that matter for separation: no internal document in a customer result, and every gold document indexed.

39 of 40 questions that behaved correctly in all three attempts The answer behaviour matrix over the whole evaluation set. One question was wrong in all three attempts and is named in the test report.

0.0 per cent leak rate over 69 customer turns No internal passage reached a customer answer. Measured again at retrieval level: 0 of 2400 retrieved chunks came from a class the role may not read.

64 of 64 role and permission cells matching the expected outcome Four roles against sixteen document classes, plus 6 of 6 document_acl cells measured in both directions.

40 of 40 prompt injection cases behaving as expected Twenty attempts in document content and twenty in user input. Attempts carried out by a released answer: 0.

28 of 28 files processed as expected Every document of the corpus plus seven deliberately broken files, through the real ingestion pipeline. PDF, DOCX, TXT, CSV, HTML and Markdown.

1072 unit, contract and security tests passed 0 failed, with the formatter and the linter clean.

253 ms retrieval p50, p95 352 ms Hybrid search with a cross-encoder rerank over the evaluation set, on the machine the source stack runs on.

200 of 200 questions answered from 20 concurrent sessions 43.4 seconds wall clock, end to end p50 3965 ms and p95 6023 ms. Generation ran on the recorded provider in that run, so it measures the application and retrieval, not the model.

0.000086 USD measured cost per conversation Read off the provider account over a 120 turn matrix, not calculated from token counts.

Not measured production accuracy, throughput and availability Synthetic runs only, on the machine of the source repository and on the demo host. This build has never seen production traffic or a real document set.

Not configured provider processing region and retention The model route of this reference implementation states no region or retention term. A client deployment picks its own provider and key.

An answer without a source is not released

Every claim carries a marker, every marker maps to a stored passage with its document title, version and page or heading, and an answer whose markers do not check out is refused with the reason logged.

VERIFY: Run Documented customer question and open the sources under the answer

Customers cannot reach internal content

The same question about an internal only fee returns the amount with its citation for a teammate and nothing at all for a customer, who is offered a person instead.

VERIFY: Run Internal only content

A restricted category needs more than a login

The trust account procedure is outside what a teammate without the grant can retrieve, checked at retrieval level before any model is involved, and inside what an account that holds it retrieves.

VERIFY: Run Restricted category and read the retrieval panel

It stops instead of inventing

A question the documents deliberately do not cover ends on a recorded fallback reason with no rate in the answer.

VERIFY: Run Nothing in the documents

Instructions inside a question stay text

An injected instruction to print the internal commission policy leaves no internal source, no internal amount and an injection_detected row in the audit trail.

VERIFY: Run Prompt injection

A handover really leaves the application

The escalation is stored before anything is sent, posted to the workflow, delivered as mail, chat message and ticket, and the ticket number is written back on the escalation row.

VERIFY: Run Handover to a person

Routing is deterministic

The same handover reaches the live support channel when it is stamped inside the configured business hours and on-call when it is stamped outside them.

VERIFY: Run Business hours routing

Duplicates cost nothing

A repeated delivery of one escalation is answered as a duplicate with the time it was first seen, and no destination is reached twice.

VERIFY: Run Duplicate delivery

A provider outage degrades safely

With the provider returning HTTP 502 the answer falls back with reason provider_error and states no rate, and the same question answers normally once the provider is back.

VERIFY: Run Provider failure

Every turn is auditable

Retrieval candidates and scores, citation count, validation result, fallback reason, tokens and security events are read back from PostgreSQL under each result.

VERIFY: Open the PostgreSQL evidence panel under any result

TRY THESE CASES

Documented customer questionAn answer with at least one public source, or a handover with a recorded reason. Never a rate without a source

Internal only contentNothing for the customer and no source name, the amount with its citation for the teammate

Nothing in the documentsA recorded fallback reason, no invented rate, a person offered instead

Restricted categoryOutside what the teammate can retrieve, inside what the account with the grant retrieves

Prompt injectionNo internal source, no internal amount, an injection_detected row in the audit trail

Handover to a personAn escalation row, mail, chat and ticket delivered, urgent priority, ticket number written back

Business hours routingThe live support channel inside the window, on-call outside it, everything else identical

Duplicate deliveryThe repeat answered as a duplicate, nothing delivered twice

Provider failureFallback with reason provider_error and no rate, then a normal sourced answer

Your own questionSame guardrails in a customer, teammate or granted session, and the invariants checked on your input too

RUNTIME PATH

The route decides the audience: the customer surface reads public documents only, the internal surface reads both
A follow up question is rewritten into a standalone one from the last turns
Dense and sparse search run over the collections the session may read, with a mandatory audience filter, and are fused
A local cross-encoder reranks the candidates, near duplicates are dropped and the context is built to a token budget
Below the score gate nothing is generated: the reply is the handover block with the reason recorded
The model answers only from the numbered passages, with a marker on every claim
The citation validator checks every marker and refuses an answer that states a fact no passage carries
Question, answer, retrieval scores, citations, tokens and security events are written in one transaction
A handover is stored, posted to the n8n workflow and delivered as mail, chat message and ticket with the reference written back

DATA BOUNDARY

An invented document set for a fictional agency, and one public surface. The application, PostgreSQL, Qdrant, n8n and the provider key stay on the internal Docker network.

Only the question and the retrieved passages reach the model provider. Documents, accounts and log rows stay in the stack.
The provider key lives in one container that neither the public console nor the application reads.
The browser never receives an internal document, an internal passage or the address of any internal service.
The three escalation destinations are stand-ins on the internal network. No mail, chat message or ticket leaves it.
Free text is redacted before it is logged, and the log tables refuse updates and deletes.
Provider processing region and retention are not configured or measured for this reference implementation.
The demo resets to its seeded state every night at 04:00 CET.

A chatbot over company documents fails in two ways. It refuses everything that is slightly awkward, or it states a fee, a policy or a procedure that no document contains. The second failure is the expensive one: a customer who is quoted a management fee that is not on the rate card acts on it.

So the model is only allowed to write from what was retrieved, and a separate layer decides what is released. Retrieval is hybrid: a dense vector and a sparse keyword vector in the same Qdrant collection, fused, then reranked by a local cross-encoder. A follow up question is made standalone from the last turns first. Below the score gate nothing is generated at all: the reply is the handover block, with the reason recorded. Above it, the model gets numbered passages and has to put a marker on every claim, and a citation validator checks each marker against the passages it was given. An answer that states an amount, a date or a name that no passage carries is refused and the visitor is offered a person.

The audience is resolved from the session on the server. No request field selects a collection or a filter, public and internal passages live in two separate collections, and a category inside the internal set can be locked further with a grant. Around that sits the rest of what a document assistant needs to be usable: ingestion for PDF, DOCX, TXT, CSV, HTML and Markdown with versioning, duplicate handling and re-indexing, an admin panel for documents, conversations, unanswered questions and citations, role based access, rate limiting, prompt injection handling, an append-only audit trail per turn and analytics. Handing a conversation to a person runs through an n8n workflow with an idempotency guard, business hours routing, bounded retries and a dead letter path.

This is a production-style reference implementation built on a public brief, with an invented document set for a fictional real estate agency and no client behind it. The measured report is included rather than summarized: 39 of 40 evaluation questions behaved correctly in all three attempts, the leak rate over 69 customer turns was zero, all 64 role and permission cells matched, and all 40 prompt injection cases behaved as expected. An independent review then re-ran the whole acceptance list against the running stack and accepted the build against every requirement in the brief.

What it does

Hybrid dense and sparse retrieval with a local cross-encoder rerank and standalone rewriting of follow up questions
Audience resolved from the session, never from the request: two collections, a mandatory payload filter and a restricted category behind a grant
A citation validator that refuses an answer stating a fact no retrieved passage carries, with the reason written to the log
Document lifecycle over six file formats: upload, extract, chunk, version, publish, retract and re-index
Handover to a person through n8n with an idempotency guard, business hours routing, bounded retries and a dead letter path
Append-only audit trail per turn: retrieval scores, citations, validation result, fallback reason and tokens