Overview
This is the site you are reading. It is also the project I would point at first in a technical interview, because it is the one where I can show you numbers instead of adjectives.
The visible half is a Next.js front end. The interesting half is the chatbot in the corner: a Retrieval-Augmented Generation system that answers questions about my experience from my own resume and a written Q&A corpus, streaming responses through a FastAPI backend. Plenty of portfolios have a chatbot bolted on. What I wanted was the thing most of them skip — a way to know whether the retrieval underneath it actually works, and a gate that fails the build when it stops working.
The problem with "it seems to work"
A RAG pipeline gives you a lot of knobs and very little feedback. Mine started with the parameters you would expect: a weighted blend of semantic and keyword search, ten candidates narrowed to five by a second model. Every one of those numbers was a guess. I had no answer to "does the re-ranking stage earn its latency?" or "would keyword search alone be just as good?", and nothing would have told me if a change to how documents get split quietly broke retrieval.
That is the situation evaluation exists for, and it is not specific to this project. Any system whose output is text has the same problem: it is always producing something plausible, so "it seems to work" is the default state whether it works or not. The only way out is to decide in advance what good means, label enough ground truth to measure it, and put the measurement somewhere it can fail.
How I decided what "good" means
The golden set is a set of questions of the kind a recruiter, hiring manager, or technical interviewer actually asks — employment history, specific technologies, project details, education, logistics. Each one is labelled with the phrases a correct answer has to be grounded in.
Labelling by phrase rather than by document ID is the decision I would defend hardest. The corpus gets re-split every time I edit a source document, so any label tied to a chunk's identity is invalidated on every rebuild — the labels would rot faster than I could maintain them. Phrases survive re-chunking, which is what makes the golden set an asset rather than a chore.
The metrics are chosen for what they decide, not for what is easy to report:
- hit@5 — did any relevant passage reach the model? This is the one that gates the build, because it is what determines whether the system can answer at all.
- recall@5 — what share of the relevant passages made it. Reported, deliberately not gated: several questions have more than five relevant passages, so their recall is capped below 1.0 by construction and no honest threshold could ever be met. A metric you cannot set a fair threshold on is a metric you should not gate on.
- MRR and nDCG@5 — how far up the list the good passage landed, which matters because a model attends unevenly across a long context.
The thresholds sit below measured performance with headroom for noise. They are floors, not targets. Tuning parameters until a floor is cleared would be fitting to my own test set, which produces numbers that look better and mean less.
The questions are split into a development portion and a held-out portion, and the table below reports the held-out one. Choices about the architecture are made against the development questions; the held-out questions are never looked at while making one. It is a smaller sample and therefore a coarser number, and I would rather publish a coarse number that means what it says than a precise one that is really the score of whatever configuration won on the questions used to pick it. The split is assigned by hashing each question's ID, so I cannot move an inconvenient question across the line without it showing up in a test.
The current results
Everything in this table is generated. The harness runs in CI, writes its results to a file, and commits them; this page renders that file. The sentence underneath the table — which configuration is in production, and whether any other one is distinguishably better or worse — is derived from the numbers rather than written by me. On a few dozen held-out questions, a lead of one or two questions is noise, so a configuration is only called ahead or behind when the questions it disagrees with production on are too lopsided to be chance. The last column shows that raw disagreement: how many questions each configuration got right that production missed, and the reverse.
| CONFIGURATION | recall@5 | hit@5 | MRR | ndcg@5 | VS PRODUCTION | LATENCY · LOAD |
|---|---|---|---|---|---|---|
| Keyword onlyIN PRODUCTION | 0.572 | 0.814 | 0.707 | 0.628 | — | ~1ms~1ms |
| Meaning only, shippable | 0.496 | 0.744 | 0.606 | 0.514 | 4 won · 7 lost | ~10ms~1s |
| Both combined, shippable | 0.554 | 0.884 (best) | 0.638 | 0.547 | 4 won · 1 lost | ~10ms~1s |
| Keyword, then re-ranked from 20, shippable | 0.584 (best) | 0.861 | 0.762 (best) | 0.644 (best) | 3 won · 1 lost | ~1s~100ms |
Production runs Keyword only. Across the 43 held-out questions, no other arm is distinguishable from it on hit@5.
Configurations
- Keyword only
- Matches the words in the question against the words in the document, with no machine learning involved.
- BM25 over stemmed, stopword-filtered terms. Needs no embedding model, which is why the production image ships none. Matches RAGPipeline.retrieve.
- Meaning only, shippable
- Matches on meaning rather than wording, on a lightweight engine small enough to go into production.
- Inner-product search over normalized all-mpnet-base-v2 embeddings, computed with ONNX Runtime instead of torch.
- Both combined, shippable
- Merges the keyword ranking with the shippable meaning-based ranking into one list.
- Weighted reciprocal-rank fusion of the BM25 ranking (weight 0.3) and the `onnx-dense` ranking (weight 0.7), damped at k=1.
- Keyword, then re-ranked from 20, shippable
- Takes the top 20 keyword results and has a lightweight second model re-read them against the question to pick the best few.
- BM25 over 20 candidates → ms-marco-MiniLM-L-6-v2 cross-encoder on ONNX Runtime narrows to k. The pool is fixed at 20 so the re-ranker can recover Chunks BM25 ranks below the top 10.
That last part is deliberate, and it is the only way I could find to publish results honestly over time. If I wrote the conclusion by hand, it would be a claim about one afternoon that slowly stopped being true, and I would be the last person to notice. Generating it means the page cannot flatter the architecture I happened to build: when the simplest configuration beats the elaborate one, the page says so, on its own, until I fix it.
It also means this write-up does not need rewriting when the system changes. The numbers move, the verdict follows them, and the prose around it stays true because it never claimed to know the answer.
How well it answers
Retrieval only says whether the right text reached the model. It does not say whether the answer was right, stayed inside that text, or declined when it should have. A second harness measures that: it asks the chatbot every held-out question, splits each answer into claims, and has a separate judge model, pinned to an exact version, check each claim against the retrieved text one passage at a time. Before I trusted the judge, I checked it against sentences I had labelled by hand, and it failed the first time.
| METRIC | SCORE |
|---|---|
| FaithfulnessGATEDShare of factual sentences in an answer that the retrieved text supports. | 1.0 |
| Decline accuracyGATEDShare of questions handled correctly: answered when the documents cover them, declined when they do not. | 0.90 |
| Fact RecallShare of the facts a complete answer should contain that the answer conveyed. | 0.60 |
- RETRIEVAL
- Keyword only
- MODEL
- gpt-6-luna
- JUDGE
- jev-1.13.0
- COST PER ANSWER
- $0.00009
Faithfulness and decline accuracy gate every change that could alter an answer: the prompt, the retrieval code, the documents and the questions. Unlike retrieval, the same commit does not give the same answers twice, so I ran it three times, set the floors below the observed spread, and round each published score to a step that noise alone cannot move.
The retrieval table can only say which configuration finds the right text more often. Whether that changes what visitors are told is a question for this harness, so a separate run, triggered by hand, answers the same questions through every configuration that could ship. A configuration replaces keyword search only if its answers convey distinguishably more of the facts they should, without being distinguishably less faithful or worse at declining, and stays within about a tenth of a second per query. I decide on one set of questions and publish another, so the table below confirms the choice rather than makes it.
No comparison has been published yet. It appears here after the comparison run is next dispatched on main.
What the evaluation changed
Metrics are only worth collecting if they alter what you build. This is the running record of what the harness found and what I did about it — appended to, never rewritten.
Retiring the torch arms cut backend CI by more than half and changed no score
The four torch arms only ever duplicated their ONNX twins, but every backend pull request still installed torch, its CUDA wheels and FAISS, and the two torch re-rankers took about 65 seconds of each retrieval gate. With them gone, Backend CI (lint, tests and retrieval eval, summed) took 2m18s on a warm cache, against 5m20s before, and 2m28s on the first run after the lockfile changed, against 6m57s for the last such run (which also published results, about 70 seconds of its total). The retrieval gate fell from 3m16s to 1m29s, tests from 50 to 15 seconds and lint from 32 to 9.
What changed: The torch arms, the FAISS index and the sentence-transformers and faiss-cpu dependencies were removed from the harness, so every arm it measures could ship. The shipped arm, its floors and every remaining arm’s definition are unchanged, and the torch measurements stay in the evaluation docs as the record of the parity check.
Swapping torch for ONNX Runtime changed no retrieval score, only the speed
Run on ONNX Runtime instead of torch, semantic search and hybrid search scored identically on all four metrics for 68 of 68 development questions (158 chunks): the embeddings agreed to a cosine similarity of 0.9999999, and the re-ranker put every candidate in the same order. The runtime changed only the cost. The ONNX models load in about a second rather than a tenth of one, and the ONNX re-ranker ran 5–9× slower than torch.
What changed: Each torch Reference Arm was a duplicate row of an arm that could ship, so the scoreboard now shows only the arms that could ship. The torch arms are still measured, and their results stay in the evaluation docs.
No shippable model changed the answers enough to replace keyword search
Each arm that could ship (keyword search, ONNX semantic search, ONNX hybrid, and keyword search re-ranked by an ONNX cross-encoder) retrieved for every golden question, gpt-6-luna answered from what it retrieved, and the pinned Judge scored the answers. On the 82 development cases (152 Required Facts), every arm was indistinguishable from keyword search on Fact Recall: the most lopsided split, hybrid’s 20 facts gained to 15 lost, has a sign-test p of 0.50. They were also indistinguishable on Faithfulness and Decline accuracy. The 49 held-out cases (112 Required Facts) gave the same result. The re-ranking arm came closest there (12 gained, 4 lost, p = 0.077), and its ~1s warm Latency Band rules it out anyway. This is not a win for keyword search; with this many cases the arms cannot be told apart.
What changed: Keyword search stays the shipped arm by ADR-0009’s default, so the shipped flag, the production dependencies and the image are unchanged. Telling these arms apart needs more of the paraphrase, conceptual and visitor questions they disagree on.
The cheapest OpenAI model answered more faithfully than the Groq model it replaces
Given the exact chunks the Groq baseline answered from, gpt-6-luna and Groq differed on 10 held-out answers where only one of them wrote an unsupported sentence. Every one was Groq’s, a split a sign test rules out as chance (Faithfulness 0.762 → 0.986). gpt-6.1-sol conveyed more Required Facts (Fact Recall 0.782 vs 0.701) but was indistinguishable from Luna on Faithfulness and Decline accuracy, at about 22× the cost per answer ($0.0028 vs $0.00012). The decision was made on dev, and held-out gave the same answer.
What changed: Chose gpt-6-luna as the production generation model under the written rule (cheapest unless distinguishably worse; Fact Recall alone never justifies an upgrade), and added a replay mode so candidate models answer from stored contexts.
Groq gpt-oss-120b → gpt-6-luna, Keyword only, held-out (39 questions)Faithfulness0.762to0.986
Cutting the corpus along its sources’ structure moved no arm beyond noise
Re-chunking the corpus from ~1,000-character packs into one chunk per question and answer, project section and resume role (123 chunks to 157) was measured on the 54 development questions. Keyword search gained four questions and lost one, including “Where is David based?”, which had missed although its answer chunk contained the question. No arm’s gains and losses were lopsided enough to rule out chance (exact sign test, p ≥ 0.375). Its five remaining misses are vocabulary gaps — “study in college” against “Major: Biochemistry”, “national defense vetting” against “security clearance” — that no stemmer bridges.
What changed: The index build has one chunking entry point with one rule per source, every chunk carries the heading that says what it is about, and a source that stops parsing into its structure now fails the build instead of indexing as one block. The tokenizer was left alone: no miss that survived re-chunking is a tokenizer problem. The floors were re-checked, not edited — over the whole golden set keyword search cleared them at hit@5 0.885 and MRR 0.752, against 0.85 and 0.75.
Keyword only (development questions, all 54)hit@50.852to0.907
Query categories revealed where dense and re-ranking models beat keyword search
Expanding the Golden Set to 87 cases across direct, paraphrase, and conceptual queries exposed where keyword search succeeds and where semantic retrieval excels. While BM25 dominated direct in-vocabulary questions (hit@5 0.964 vs dense’s 0.818), out-of-vocabulary paraphrases reversed the ordering: hybrid and re-ranking arms achieved hit@5 0.938 compared to BM25’s 0.875, and re-ranking lifted MRR from 0.617 to 0.828 over the full set. Across all 87 questions, the shipped BM25 arm maintained hit@5 0.908 and MRR 0.765, comfortably clearing production regression floors.
What changed: Categorized the benchmark into direct, paraphrase, and conceptual partitions and shipped interactive Scoreboard filtering in Schema Version 3. Production remains on BM25 for its low latency, light footprint, and strong overall hit rate, but the diagnostic breakdown proves dense retrieval and cross-encoder re-ranking provide measurable recall and reciprocal-rank gains on natural, out-of-vocabulary queries.
Both combined, then re-ranked (paraphrase queries, all 16)MRR0.617to0.828
Plain keyword search beat the hybrid pipeline, so the architecture was deleted
On the 73-chunk corpus, across all 55 labelled questions, BM25 alone scored hit@5 0.982 against the shipped hybrid + cross-encoder pipeline’s 0.909, and led every other metric too. (The table above reports the held-out 22 only, so its numbers are the same finding on a smaller sample.) The tempting reaction was to retune the fusion weights, which would have been the wrong experiment: under reciprocal-rank fusion, driving the dense weight toward zero just converges on BM25’s ordering, so “find the optimal ratio” and “turn dense off” are the same question. A new arm asked it directly — BM25 candidates fed straight to the cross-encoder, no dense stage — and beat the full pipeline on hit@5, recall@5 and nDCG@5 on every split. The dense stage was contributing nothing the re-ranker could not recover. The cross-encoder then failed the same test against BM25 alone: it tied on hit@5 everywhere and lost on recall@5 and nDCG@5. Its one advantage was MRR on the development questions, and that reversed on the held-out ones — which is the comparison holding a portion back exists to make possible.
What changed: Production retrieval is now BM25 and nothing else. That took the semantic search stage, the fusion step and the cross-encoder off the serving path, and with them ~500MB of model weights baked into the container image and the faiss-cpu and sentence-transformers dependencies — 87 production packages down to 40, and no model loading on a cold start. Cheaper and better is a stronger outcome than a tuned constant, and it is not the one the architecture was built expecting. Every deleted stage is still measured as an arm on every run: this corpus is written as anticipated questions and answers, exactly the condition that flatters lexical search, so the conclusion is expected to flip as the corpus grows and the harness should be what says so. The golden set was also split into development and held-out portions, and the published table now reports the held-out 22 — an architecture chosen by a measurement must not be scored by the questions that chose it. The container image went from 17.2GB to 544MB, most of it not model weights but the CUDA libraries that torch pulls in behind the embedding model, shipped into a container with no GPU.
shipped configuration, all 55 questionshit@50.909to0.982
A textbook constant, used at the wrong scale, had disabled half the pipeline
Reciprocal-rank fusion combines ranked lists as weight / (k + rank), and the standard k of 60 assumes lists thousands of documents long. Over ten-item lists it varied the rank term by 1.15x while the arm weights varied by 2.33x, so every semantic hit outscored every keyword hit and the keyword retriever could never contribute a candidate the other had missed. Fusion was running, passing its tests, and contributing nothing.
What changed: Dropped k to 1 so rank position outweighs the arm weighting at this list length, and added fusion tests that assert a low-weighted retriever can still introduce a candidate. No unit test could have caught this: every component behaved exactly as specified, and only a metric moved.
Index-time and query-time tokenizers disagreed, silently costing keyword recall
Both the index builder and the query path tokenized with lower().split(), which never matched "engineering" against "engineer", left punctuation welded to terms, and scored stopwords as if they carried meaning.
What changed: Replaced both with one shared stopword-filtered, Snowball-stemmed tokenizer in app/rag/tokenize.py. Shared is the operative word — a mismatch between the two sides is unfixable at query time, because the terms being searched for were never written into the index.
Keyword onlyMRR0.745to0.892
Two of those entries share a pattern I found useful to learn: the code looked correct, every component behaved exactly as specified, every test passed, and nothing was going to surface the problem except a number moving. Unit tests tell you the parts do what you said; only measurement tells you the system does what you meant.
The third one is the one I would actually want to be asked about. I built a hybrid retrieval pipeline — semantic search fused with keyword search, then a cross-encoder re-ranking the candidates — because that is the architecture you build when you are demonstrating that you can build it. The evaluation said plain keyword search beat it on every metric. The tempting response was to retune the fusion weights until the numbers came back, and that would have been the wrong experiment: under the fusion formula I was using, tuning the semantic weight toward zero and switching the semantic stage off are the same experiment, so "find the optimal ratio" would have been a slower path to the answer I already had, ending in a magic constant with nothing behind it. So I ran the experiment that actually asked the question — keyword candidates fed straight to the re-ranker, no semantic stage at all — and it beat the full pipeline. Then the re-ranker failed the same test against keyword search alone.
The fix was to delete the architecture. Production retrieval is now keyword search and nothing else, which cut the backend from 87 production dependencies to 40 and the deployed container image from 17.2GB to 544MB. Most of that was not the model weights — it was torch, which the embedding library pulls in, and whose default Linux wheels ship the NVIDIA CUDA runtime into a container that has no GPU. I had been deploying several gigabytes of GPU libraries to serve keyword search, and nothing was ever going to tell me that except deciding the stage that needed them had not earned its place. Cheaper and better is a stronger result than a tuned constant, and it is not the result I wanted when I started. Every deleted stage is still measured as a configuration on every run, because this corpus is written as anticipated questions and answers — the exact condition that flatters keyword matching — and I expect that to stop being true as the corpus grows. When it does, I want the harness to be what tells me, not a rebuild from scratch.
Engineering around it
The eval is a CI gate, not a notebook. It runs on every backend change and fails the build if the shipped configuration drops below its floor, so retrieval quality cannot regress silently between commits. The other configurations are measured and published for comparison but not gated: a floor exists to stop a change degrading what a visitor is served, and nothing is served by a configuration production does not run. Because production is now keyword-only, the gate itself runs in seconds and needs no model weights at all.
The rest of the pipeline is held to the same standard. Ruff, ESLint, Prettier, and tsc gate every push. CodeQL, gitleaks, pip-audit, npm audit, and dependency review run on every pull request and weekly against unchanged code. Lighthouse CI watches performance and accessibility budgets. Every check lives in scripts/ and is shared verbatim between the Makefile and GitHub Actions, so make ci-cd reproduces the PR gates exactly — no drift between what passes locally and what passes in CI.
One gate is there because of a specific mistake. My resume is a source document for the index, and it carried my home address and phone number, which meant the chatbot would read them back to anyone who asked. The fix was three-layered: strip it at the source, redact it during the index build as a backstop, and add a regression test that fails the build if a phone number or street address ever appears in the committed index again. The redaction and the detection are deliberately asymmetric — the redactor is precise, because rewriting the corpus silently means a false positive destroys real content unnoticed, while the test guard casts a wide net, because its only power is to fail a build and a false positive there costs one glance from me.
What I am not claiming
The corpus is small, and much of it is written as anticipated questions with answers — which is exactly the condition that favours keyword matching over semantic search. Relevance is also labelled by exact phrase presence, which is close to what keyword ranking measures. Every configuration is scored against identical labels, so the comparison is internally fair, but any conclusion it supports is about this corpus rather than about retrieval in general.
That bias cuts directly against the conclusion I drew from it, which is why the retired configurations are still measured rather than deleted from the harness: the result that told me to remove them is the same result I expect to reverse as the corpus grows past the point where the questions and the documents share vocabulary.
I would rather state that plainly than let a table imply more than it earned. The full write-up of the method, the biases, and what is still unmeasured — adversarial robustness, and multi-turn conversations — is in the repository.
Skills Demonstrated
The RAG pipeline is the part that looks impressive. The evaluation suite is the part that matters, and it is the difference between an engineer who assembles components and one who can tell you whether the assembly works.
This project shows I can define what "good" means for a system whose output is text, label ground truth in a form that survives the corpus changing underneath it, hold a portion of it back so published numbers are not tuned numbers, choose metrics for what they decide rather than what is easy to report, wire the result into CI as a gate rather than a report — and build the publishing path so that the conclusions stay honest as the system evolves, including the time the measurement contradicted the architecture I had already built and the right answer was to delete it.