Skip to content

Run evaluations#

An evaluation asks a knowledge base a set of questions whose answers you know and says how well it found them: recall@k and MRR, and for each question where the expected document came. Run one after making a base, and again after every change of model, chunking or documents. For a walkthrough that compares settings and acts on the results, see Measure and improve search quality.

Before you begin#

  • A knowledge base with documents, Indexing or Ready.
  • Permission to evaluate: knowledge-bases:evaluate (workspace editors and admins, the Developer preset). Reading the sets and past runs needs knowledge-bases:read.
  • For the CLI, astra signed in with astra use <org>/<workspace>; for the API, a token in ASTRALYX_TOKEN and ASTRALYX_API=https://api.astralyx.cloud/v1.

Write a set#

A set is a JSON Lines file: one question per line, with the documents that answer it.

evals.jsonl
# Leave and travel.
{"question": "How many vacation days do I have?", "expected": ["policies/leave-policy.pdf"]}
{"question": "per diem for Lisbon", "expected": ["policies/travel.docx", "policies/expenses.pdf"]}
{"question": "Who signs off a conference trip?", "expected": ["handbook-files:policies/travel.docx"]}
Field Description
question Asked as a search is. Required.
expected One or more documents that answer it: the path in its drive (policies/travel.docx), or <source>:<path> when two sources hold the same path. A file added to the base is notes/<path>. Required.

Blank lines and lines starting with # are skipped. At most 500 questions.

Keep it where the base finds it#

Where How Named in an evaluation as
A source drive, in its hidden folder .metis/ — at the drive's root, or under the base's folder of it Write the file there, as any file of the drive <drive>:.metis/evals.jsonl (or <drive>:<folder>/.metis/evals.jsonl)
The base's own files Console: Evaluations → Add a set (.jsonl); CLI: astra metis add <base> evals.jsonl --to .metis; API: PUT …/files/.metis/evals.jsonl metis-<base>:notes/.metis/evals.jsonl
Your computer astra metis evaluate --cases evals.jsonl — (sent with the evaluation)
Written in the console Evaluations → Run one → Written here —

Files under .metis/ are never indexed: they cannot answer the questions they hold. Evaluation sets on the base's Evaluations tab — and GET …/eval-sets — lists every set found, with its number of questions, or why it cannot be read (line 3: …).

Run it#

On one base: open it, Evaluations, Run one:

  1. A set (choose it) or Written here (paste the lines).
  2. k: how many documents are judged per question (default 5).
  3. The mode: hybrid, by meaning only or by words only; with the reranker ticked or not.
  4. Evaluate.

Across bases: Metis → Evaluations lists every base with its sets. Run with and without the reranker asks a set twice — hybrid with the reranker, then without — for a base that has one (Run otherwise).

$ astra metis evaluate handbook --file handbook-files:.metis/evals.jsonl -k 5
        #1  How many vacation days do I have?
        #2  per diem for Lisbon
 not found  Who signs off a conference trip?
recall@5 0.5  MRR 0.5  (3 questions)
$ astra metis evaluate handbook --cases evals.jsonl --json
Flag Default Description
--file <source>:<path> A set in one of the base's sources.
--cases <file> A set on this computer.
-k N 5 Documents judged per question, 1 to 50.
--json off The whole report.

The CLI evaluates as searches do by default — hybrid, with the reranker when the base has one. To choose the mode or leave the reranker out, use the console or the API.

$ curl -fsS -X POST "$ASTRALYX_API/metis/knowledge-bases/handbook/evaluate" -H "Authorization: Bearer $ASTRALYX_TOKEN" \
    -H 'content-type: application/json' \
    -d '{"file": "handbook-files:.metis/evals.jsonl", "k": 5, "mode": "hybrid", "rerank": false}'
Field Default Description
cases The questions: [{"question", "expected"}].
file A set in one of the base's sources, <source>:<path>. One of cases and file.
k 5 Documents judged per question, 1 to 50.
mode hybrid hybrid, dense or keyword.
rerank true false: the fused order, without the reranker.

Read the report#

Field Console Description
recall_at_k Recall@k Per question, the share of its expected documents among the first k documents found (each document counted once, however many of its chunks came back); averaged
mrr MRR Per question, 1 / the rank of the first expected document among the first 10 (0 when not there); averaged
cases, k, mode, rerank Questions, Settings What was asked, and how
outcomes[] The table Each question: expected, rank (of the first expected document; empty when none was found), found (the documents that came instead, best first) and its own recall
warnings Above the table What the searches said: an embedding model or a reranker that did not answer, so some questions were asked another way

A warning means the numbers do not measure the settings you asked for: fix the model and run it again.

Past runs#

Each run's numbers and settings are kept on the base — never its questions or what was found: the base's Evaluations → History (the last 20, with Settings such as hybrid + reranker), the last one on the base's page and in Metis → Overview (Quality (last evaluation)), and GET …/evaluations. astra metis get <base> prints the last one.

Limits#

Limit Value
Questions per evaluation 500
k 1 to 50
Time The evaluation answers within 60 s, or fails (504 UPSTREAM_TIMEOUT): split a long set
Runs kept per base 20 (numbers and settings only)

Troubleshooting#

Code Cause Fix
EVAL_EMPTY No questions: neither cases nor file Give one
EVAL_FILE The file is not in a source of the base (… is not one of this knowledge base's sources), cannot be read, or a line is not JSON Name it <source>:<path>; fix the line it names
EVAL_CASE_INVALID A question without an expected document Give each one at least one
EVAL_TOO_LARGE More than 500 questions Split the set
TOP_K_OUT_OF_RANGE k is not between 1 and 50 Use 1 to 50
KNOWLEDGE_BASE_NOT_READY The base is not serving See Troubleshoot a knowledge base
Every question not found The expected paths are not the documents' paths Copy them from Documents → Everything indexed