OAKENMAILEvaluation results
How well hybrid search finds the right message and how few tokens it spends, rendered from the latest run.
Measured on 2026-10-20 with corpus seed 0: 1,000 messages in 2,249 chunks, 48 answerable questions (12 lexical, 12 paraphrase, 12 identifier, 12 thread) and 12 that the mail does not answer. How the pieces work: Retrieval and ask.
- Mode: offline. Embedder: concept-embed (offline stand-in). Reader: extractive stand-in (not a model).
- Context budget: 3,000 tokens. Summaries in the context: first sentence (stand-in for the summary enricher).
An offline run uses stand-ins for both models, so it measures how the retrieval and context code behaves, not how any real model does. The corpus is synthetic and was built so that each retriever has a kind of question it is bad at. Do not read these numbers as what a real mailbox would score; a run with real models is a manual step (
--live).
Retrieval
Over the answerable questions. A message is relevant when it holds the planted fact.
| Retriever | Recall@5 | Recall@10 | MRR | nDCG@10 |
|---|---|---|---|---|
| Keyword only (BM25) | 0.750 | 0.750 | 0.740 | 0.742 |
| Vector only | 0.625 | 0.625 | 0.546 | 0.562 |
| Hybrid (RRF, exact-text boost, minimum match) | 1.000 | 1.000 | 0.990 | 0.992 |
| Hybrid, plain RRF | 0.646 | 0.667 | 0.605 | 0.612 |
nDCG@10 by kind of question:
| Retriever | lexical | paraphrase | identifier | thread |
|---|---|---|---|---|
| Keyword only (BM25) | 0.969 | 0.000 | 1.000 | 1.000 |
| Vector only | 0.969 | 1.000 | 0.000 | 0.279 |
| Hybrid (RRF, exact-text boost, minimum match) | 0.969 | 1.000 | 1.000 | 1.000 |
| Hybrid, plain RRF | 0.969 | 0.000 | 0.556 | 0.922 |
- lexical: the question uses the message's own words.
- paraphrase: the question uses different words for the same idea.
- identifier: the question is about a code, number or other exact string.
- thread: the answer is in a reply, so the thread has to be followed.
Answers and cost
Every question goes through ask. The baseline puts the whole body of each of the top 10 messages into the prompt, with no budget, and checks the reader's citations the same way.
| Context builder | Top 10 full bodies | |
|---|---|---|
| Answerable questions answered correctly | 48/48 | 48/48 |
| Unanswerable questions refused | 12/12 | 12/12 |
| Citation precision | 100.0% | 100.0% |
| Mean evidence tokens per question | 1,885 | 3,503 |
| Evidence tokens saved vs the baseline | 46.2% | |
| Mean share of the hits' full bodies not sent | 32.0% |
Citation precision is the share of citations that point at the planted message and quote the planted fact. A claim whose quote is not in the cited excerpt is dropped before it is counted (0 dropped in this run).
The saving is below the 60% the design aims for. The messages in this synthetic mailbox are short, so the chunk that matched is often most of the message and there is little to leave out; the context builder's gain grows with the length of the mail. The figure is reported as measured.
Reproduce
uv run python -m oaken.evals retrieval --messages 1000 --seed 0
The command rewrites docs/evals/latest.md and docs/evals/latest.json, which this page is built from. With the offline stand-ins it is deterministic.