Skip to content

Knowledge bases and hybrid search#

A knowledge base is three things: its sources (drives, or folders of them), an index kept on one of your workspace's machines, and one search API. This page explains how each part works, so you can choose models and settings and read what a base tells you.

Sources#

A source is a drive, or one folder of it (path). Any kind of drive works — a folder on one machine, a copy on each machine, a shared filesystem — and so does a drive from a data source (a bucket, NFS, a Hugging Face repository). A base reads up to 16 sources; two folders of one drive may be two sources as long as neither contains the other.

These files are read; others are counted by extension and left out:

Kind Extensions What is kept for citations
md .md, .markdown, .mdx Headings (# and underlined), the line each passage starts at, the title (front matter's title, else the first # heading)
txt .txt, .text, .rst, .adoc, .org, .csv, .tsv, .json, .jsonl, .log, .tex The line each passage starts at
html .html, .htm, .xhtml <title>, headings <h1>–<h6>; scripts, styles and navigation are dropped
pdf .pdf The page. Only the text layer is read: a scanned PDF without one is reported as such (no OCR)
docx .docx Headings by their Word style (in any language), the title

A file is read up to 64 MB. Text that is not UTF-8 is read as Latin-1 (older Windows files), UTF-16 by its byte-order mark. Hidden files and folders (.git, .DS_Store, .metis) are not read.

Chunks#

Each document is cut into chunks of at most 1200 characters (about 300 words; chunking.size, 200 to 4000). A chunk never spans two sections, so it keeps the headings above it, its page and its line; it ends at a paragraph, sentence or line end when one falls in its second half, and starts 200 characters (chunking.overlap) before the previous one ended, on a word — a sentence cut in two is whole in one of them.

A chunk is embedded with its document's title and headings in front (Handbook > Leave policy > Vacation), so a passage under Installation is found by a question about installing.

flowchart LR
  Q[Question] --> E[Embed the question]
  Q --> W[Its words]
  E --> V[50 closest chunks<br/>by meaning]
  W --> K[50 best chunks<br/>by BM25]
  V --> F[Reciprocal-rank fusion]
  K --> F
  F --> R[Reranker orders<br/>the best 20–50]
  R --> T[top_k results<br/>with citations]
  1. By meaning. The question is embedded by the base's embedding model and compared with every chunk (cosine similarity). This finds "how many holidays do I get" in a passage about vacation days.
  2. By words. The question's words are looked up with BM25, accents folded (configuracao finds Configuração). This finds exact names, codes and numbers — Xid 79, nvidia-smi — that meaning alone blurs.
  3. Fusion. The two lists are merged by reciprocal rank: a chunk both found ranks above one only one of them found first.
  4. Reranking. When the base has a reranker, it reads the question with each of the best fused candidates (at least 20, three times top_k, at most 50) and orders them by relevance. Without one, results come in the fused order.

Each result carries its score in each step (dense, keyword, fused, rerank) and the search says how long each step took.

When a model does not answer, the search still answers: without the embedding model, by words only; without the reranker, in the fused order — and its warnings say so. A search may also ask for one retriever only (mode: keyword needs no embedding model; mode: dense) or skip the reranker (rerank: false).

Citations#

Field What it says
source The drive
document The file's path in the drive
title The document's title, when it has one
kind md, txt, html, pdf or docx
page A PDF's page, from 1
headings The headings above the passage, outermost first
line The line the passage starts at (text and Markdown), from 1
snippet Up to 240 characters around what matched — the sentence with most of the question's words

Incremental indexing#

The index is kept on a drive of the base's own, so it survives restarts. A pass walks the sources and compares each file with what was indexed: same size and modification time, it is left alone; otherwise its content hash decides (a file copied over itself is not read again). New and changed files are read, cut and embedded; files that are gone are dropped. Searches are answered during a pass, from what is indexed so far.

A pass runs when the base starts, every rescan_seconds (60 by default), when you ask (Reindex), and when a source drive backed by a data source has a new revision — the base then reads exactly that revision. Everything is read again when you change the embedding model or the chunking, or ask for a full reindex.

A file that cannot be read (a scanned PDF, a password-protected document, one larger than 64 MB, one the embedding model refuses) is listed with why in the base's documents, and not tried again until it changes. When the embedding model does not answer, the pass stops and waits — the base shows Waiting and why — and resumes where it stopped.

Where it runs#

A knowledge base runs as ordinary work on one of the workspace's machines: one that can read its source drives and reach its models over the workspace's network. It takes 1 CPU core (and idle cores beyond) and 2 GiB of memory by default (placement); the index is held in memory, about 5 KB per chunk at 1024 dimensions. In Astraeus it shows as a replica group and a drive named metis-<name>, both managed by the base and deleted with it.

Its models are Eos deployments it calls on the workspace's network, so keep each at one replica at least: a deployment that scales to zero is not woken by the base, and the base warns you.

Nothing leaves your machines: the documents are read where they are, the index stays on your drive, and questions reach the base only. The control plane keeps the base's totals (documents, chunks, progress), never a document's name or text.

Runs of the workspace can search a base directly at http://metis-<name>:7700/v1/search, with the same body as the search API.

States#

State Meaning
Pending Waiting for a machine, or for a data source's first listing
Starting Loading its index, or a replacement taking over after a change
Indexing Reading documents; searches are answered from what is indexed
Ready Everything indexed; watching its sources
Waiting Waiting for something — its embedding model most often; the reason says which
Failed It cannot run as asked: a drive or deployment that does not exist, or a deployment that serves something else

Evaluation#

An evaluation set is a list of questions, each with the documents that answer it. Asked as searches, it gives two numbers:

  • recall@k — of each question's expected documents, the share found among the first k documents (each document counted once, however many of its chunks came back), averaged over the questions;
  • MRR (mean reciprocal rank) — the average of 1 / the rank of the first expected document, among the first ten (0 when it is not there).

Keep the set in the hidden folder .metis/ of a source drive (.metis/evals.jsonl): it is read by evaluations and never indexed. The last evaluation's numbers are kept on the base. Run it again after changing the models or the chunking to see whether the change helped.