Problem statement

Scientific literature grows faster than any researcher can read it, and the questions that matter span a whole corpus rather than one paper. Keyword search hands back a ranked list and leaves the reader to open, skim, compare, and assemble the answer. This system returns the cited answer directly, streamed, over a corpus of NLP research papers, from one Docker Compose command, on CPU or GPU.

The task is open-domain question answering over a pre-indexed corpus. Given a natural-language query, retrieve the top-7 passages from a 47,810-chunk index built on FAISS and BM25 (drawn from the QASPER benchmark, 5,049 NLP paper QA pairs), then write a grounded, cited answer with a local model. When the retrieved context falls below the confidence threshold, the pipeline stops rather than generating from thin evidence.

Application architecture

The stack is 2 services under Docker Compose: a FastAPI backend on port 8080 and a Chainlit chat UI on port 8001. Both containers run with network_mode: host, which gives the backend direct access to Ollama on the host. The frontend waits on a health-gated backend (condition: service_healthy), polling /health every 30 seconds with a 300-second start period so the first-boot artifact download has time to finish.

The FAISS indices run about 230 MB (dense.index at 147 MB, metadata at 31 MB, sparse.pkl at 51 MB). gdown fetches them from Google Drive on first boot and writes them to a mounted volume, so every restart after that is instant. The backend detects CUDA and picks HuggingFace Transformers on GPU or Ollama on CPU, with no configuration change from the user.

The RAG pipeline, in 4 stages

Stage 1. Hybrid retrieval. FAISS dense search (SPECTER2 embeddings, 768-dim, IndexFlatIP, 47,810 vectors) fuses with BM25 sparse search through Reciprocal Rank Fusion (k=60) to return the top-50 candidates. A query under 10 words triggers HyDE: the model writes a hypothetical passage that becomes the dense query, which lifts recall on short or ambiguous inputs.

Stage 2. ColBERT v2 reranking. Document token embeddings are computed offline, so query time costs only a MaxSim over token embeddings. That buys cross-encoder precision at retrieval scale, cutting the top-50 down to the top-7.

Stage 3. The CRAG gate. Three outcomes: Correct sends the passages through to generation, Incorrect stops generation and returns a low-confidence warning, and Ambiguous produces a hedged answer. calibrate_crag.py in the research repo sets the threshold.

Stage 4. Streamed generation. Llama-3.1-8B-Instruct, through Ollama on CPU or HuggingFace Transformers on GPU, writes a chain-of-thought answer with inline [Doc N] citations and 6 turns of conversation memory. An async queue pushes tokens to an NDJSON streaming endpoint.

CI/CD pipeline

Every push to master triggers a GitHub Actions workflow that authenticates to GHCR with the automatic GITHUB_TOKEN, so there are no secrets to manage, then builds and pushes both Docker images under 2 tags: :latest and :sha-<commit>. The commit-pinned tag is what makes a rollback a one-line change.

Evaluation

Scored on 84 QASPER questions. ALCE Citation Precision went from 0.057 to 0.8373 on the strength of one change, extract_final_answer(). The reasoning block carried no citations and was inflating the false-negative Citation Recall count. Strip it, and the cited answers score for what they are. This is the sort of defect an evaluation harness surfaces and a demo hides.

Impact

• Zero-build deployment. Users run docker compose up and pull pre-built GHCR images. There's no Python environment to create, no model download script, and no index to manage by hand.
• Runs on CPU. The Ollama backend works on commodity hardware, at roughly 2 to 5 tokens per second.
• Auditable reasoning. Every answer shows its Reasoning block before the Final Answer, so a reader can check the logic that produced it.
• Follow-up questions work. 6 turns of memory mean the user doesn't have to restate context.