Measure and improve search quality#

A knowledge base is useful when it finds the passage that answers a question, and you only know it does by asking it questions whose answers you know. You write an evaluation set, measure the base, compare settings on the same questions — with and without the reranker, by meaning, by words and both — look at why each passage came up, change one thing, and measure again.

What you need:

A knowledge base with documents, here handbook from Your company handbook, Ready, with a reranker.
The editor role (evaluations need knowledge-bases:evaluate), and the variables in How the recipes are written.

1. Write a set that looks like real questions#

One JSON object per line: a question, and the documents that answer it — their paths in the drive (policies/leave-policy.pdf), or <source>:<path> when two sources have the same path.

evals.jsonl

# Questions people asked in #ask-hr, October.
{"question": "How many vacation days do I have?", "expected": ["policies/leave-policy.pdf"]}
{"question": "do unused holidays roll over", "expected": ["policies/leave-policy.pdf"]}
{"question": "Who approves a trip abroad?", "expected": ["policies/travel.docx"]}
{"question": "per diem for Lisbon", "expected": ["policies/travel.docx", "policies/expenses.pdf"]}
{"question": "What do I need on my first day?", "expected": ["onboarding/first-week.docx"]}
{"question": "Can I expense a home office chair?", "expected": ["policies/expenses.pdf"]}

Use people's words, typos and all: do unused holidays roll over measures something Can unused vacation days be carried over to the following year? does not.
Mix kinds of question: ones that need meaning (holidays for vacation) and ones that need exact words (a form's code, a product name, a number).
Several expected documents when several answer: recall counts each.
20 to 50 questions tell settings apart; at most 500. Blank lines and # comments are skipped.

Keep it where the base finds it:

ConsoleIn a driveCLI

Open the base, Evaluations → Add a set (.jsonl), and choose evals.jsonl. It is kept in the base's own files at .metis/evals.jsonl, listed under Evaluation sets with its number of questions, and never indexed.

Put it in the hidden folder .metis/ of a source drive — at the drive's root, or under the base's folder of it: /srv/handbook/.metis/evals.jsonl. Evaluation sets lists it as handbook-files:.metis/evals.jsonl.

$ astra metis add handbook evals.jsonl --to .metis --no-wait

2. Measure the base as it is#

ConsoleCLIAPI

Metis → Evaluations: each base with its sets. On handbook, press Run with and without the reranker: the set is asked twice — hybrid with the reranker, then hybrid without — and both runs appear in the base's history, side by side.

$ astra metis evaluate handbook --file metis-handbook:notes/.metis/evals.jsonl -k 5

astra metis evaluate asks the way searches do by default: hybrid, with the reranker. To compare settings, use the console or the API.

$ for rerank in true false; do
    curl -fsS -X POST "$ASTRALYX_API/metis/knowledge-bases/handbook/evaluate" -H "Authorization: Bearer $ASTRALYX_TOKEN" \
      -H 'content-type: application/json' \
      -d "{\"file\": \"metis-handbook:notes/.metis/evals.jsonl\", \"k\": 5, \"rerank\": $rerank}" \
      | jq -c '{mode, rerank, recall_at_k, mrr}'
  done

The file is <source>:<path>: a set added to the base is in its own source, metis-handbook, under notes/; a set in a drive is handbook-files:.metis/evals.jsonl.

3. Read the numbers#

Number	What it says	Moves when
recall@k	Of each question's expected documents, the share found among the first `k` documents (a document counted once however many of its chunks came back), averaged over the questions	The right document is found at all
MRR	The average of 1 / the rank of the first expected document, among the first 10 (0 when it is not there)	The right document comes first

Recall tells you whether a person who reads k results finds the answer; MRR tells you whether the first result is it — what a model given only the first passages needs. Read the report's rows too: each question, the expected documents, the rank of the first one found (— when none was), and what was found instead. A question missed with the right topic found is a different problem from one missed with nothing related found.

The base keeps the last 20 evaluations' numbers and settings — never the questions — under History on its Evaluations tab, and the last one on its page and in Metis → Overview (Quality (last evaluation)).

4. Compare the retrieval modes#

The same set, asked three ways (API, or the base's Evaluations tab: Run one, choose the mode, tick or clear with the reranker, Evaluate):

$ for mode in hybrid dense keyword; do
    curl -fsS -X POST "$ASTRALYX_API/metis/knowledge-bases/handbook/evaluate" -H "Authorization: Bearer $ASTRALYX_TOKEN" \
      -H 'content-type: application/json' \
      -d "{\"file\": \"metis-handbook:notes/.metis/evals.jsonl\", \"k\": 5, \"mode\": \"$mode\", \"rerank\": false}" \
      | jq -c '{mode, recall_at_k, mrr}'
  done

Mode	Finds	Misses
`dense` (by meaning)	Paraphrases, other words for the same thing, questions in another language than the document	Exact codes, names and numbers the model blurs
`keyword` (by words, BM25)	Exact names, codes, numbers, rare words; accents ignored	Synonyms; words in another form (`travel` does not match `travelling`: no stemming)
`hybrid` (both, fused)	What either finds; a passage both find ranks first	Rarely what both miss

Hybrid with the reranker is the default. The numbers on your own set tell you whether another setting suits your documents better.

5. See why a passage came up#

Metis → Search, tick handbook, write a question the set missed, tick compare by meaning, by words, both and press Search. The table lists every passage any mode found, with its rank in each (Both, Meaning, Words; — when that mode did not find it) and Why it came up:

close in meaning (#3) and shares words (#1); the reranker scored it 0.912 — both retrievers agree;
shares words (#2); not among the closest in meaning — it names a term exactly, and says little else like the question;
close in meaning (#1); its words do not match — it says the same thing in other words.

Timings under the table gives each mode's results and how long each step took. On a base's own Search tab, the mode and rerank are settings of one search, and each result shows its scores: meaning, words, fused, reranker, with its rank in each retriever.

6. Change one thing, measure again#

What you see	Try
The expected document is found, but low (recall fine, MRR low)	A reranker, if the base has none (Settings → Reranker)
Questions with exact codes or names are missed in `dense` but found in `keyword`	Keep `hybrid`: it already has both
Paraphrased questions missed in `keyword`, found in `dense`	Keep `hybrid`; check the embedding model is serving (a warning results by keywords only means it was not)
The right document is found but the wrong part of it	Smaller chunks (`chunking.size` below its default of 1200) for documents of many short facts; larger ones for long explanations
A document is never found	Is it indexed? Documents may say no text layer or another reason
Questions in several languages	A multilingual embedding model (BGE-M3 is one)
Questions are typed with a prefix a model expects (E5, Nomic)	Leave `query_prefix` and `document_prefix` to their defaults: the base sets them from the model

Change one setting at a time in the base's Settings, wait for it to be Ready again — changing the embedding model or the chunking reads every document again — and run the same set. The history shows the runs one under the other with their settings.

Keep the set with the documents

When people ask a question the base missed, add it to the set with the document that answers it. The set grows into the base's regression test: run it after every change of model, chunking or documents.

Footprint and cost#

An evaluation is searches: each question is one search (the reranker reads up to 50 passages for each), on the base's machine and its models. It must finish within 60 s: with a reranker on a CPU, a long set may not — split it into several files. Nothing is metered beyond the cores and memory the base and its models already hold.

Troubleshooting#

Symptom	Cause	Fix
`EVAL_FILE`: … is not one of this knowledge base's sources	The file's source is wrong	`<drive>:<path>` for a drive, `metis-<base>:notes/.metis/<file>` for a set added to the base
`EVAL_FILE`: line 3: …	A line is not JSON	Fix that line; Evaluation sets shows the error too
`EVAL_CASE_INVALID`	A question without an expected document	Give each question at least one
`EVAL_TOO_LARGE`	More than 500 questions	Split the set
`504 UPSTREAM_TIMEOUT`	The evaluation took longer than 60 s	Split the set, or evaluate without the reranker
Recall 0 for every question	Expected paths do not match the documents' paths	Copy paths from Documents → Everything indexed (`policies/leave-policy.pdf`, not `/srv/handbook/policies/leave-policy.pdf`)