Skip to content

Knowledge base specification#

A knowledge base is metadata and spec, in YAML or JSON. Names in it are your workspace's own (docs, bge-m3).

handbook.yaml
metadata:
  name: handbook
spec:
  sources:
    - drive: docs
      path: handbook
    - drive: runbooks
  embedding:
    deployment: bge-m3
  reranker:
    deployment: bge-reranker
  chunking:
    size: 1200
    overlap: 200
  placement:
    labels:
      pool: cpu
    memory_gb: 4
  rescan_seconds: 60

Fields#

Field Type Default Description
metadata.name string required Lowercase letters, digits and -, not starting or ending with -; at most 40 characters.
metadata.labels map Free-form labels.
spec.sources list required What it reads: 1 to 16 sources.
spec.sources[].drive string required A drive of the workspace, plain or from a data source.
spec.sources[].path string the whole drive Only the documents under this folder. No . or .. parts. Two sources of one drive may not overlap (handbook and handbook/hr).
spec.embedding.deployment string required An Eos deployment of an embedding model.
spec.embedding.query_prefix string the model's Put before each question when it is embedded. By default, what the library's model was trained with: search_query: (Nomic Embed Text), query: (E5), an instruction (Qwen3 Embedding), Represent this sentence for searching relevant passages: (BGE English, mxbai, Snowflake Arctic), task: search result \| query: (EmbeddingGemma); none for others (BGE-M3, MiniLM, Granite).
spec.embedding.document_prefix string the model's Put before each passage: search_document: (Nomic), passage: (E5), title: none \| text: (EmbeddingGemma); none for others.
spec.reranker.deployment string none An Eos deployment of a reranker. Without one, results come in the fused order.
spec.reranker.query_prefix string none Put before each question sent to the reranker.
spec.chunking.size integer 1200 Characters per chunk, at most: 200 to 4000.
spec.chunking.overlap integer 200 Characters each chunk repeats of the one before; less than half size.
spec.placement.machines list any Only on one of these machines.
spec.placement.labels map Only on machines with these labels (a pool).
spec.placement.cpu_cores integer 1 CPU cores it is placed by, 1 to 64; it uses the machine's idle cores beyond them.
spec.placement.memory_gb integer 2 Memory, GiB, 1 to 256. The index is held in memory: about 5 KB per chunk at 1024 dimensions.
spec.rescan_seconds integer 60 Seconds between looks for changes: 0 (only when asked, or on a data source's new revision) or 10 to 86 400.

Changing a knowledge base keeps its index: the embedding model or the chunking changed reads every document again; a new source's documents are read; a removed source's are dropped.

What it shows#

Field Description
status.state, status.reason Its state (below), and why in words
observed.index.documents, .chunks, .failed Documents and chunks indexed; documents that could not be read
observed.index.pass_total, .pass_done The current pass's documents, and how many are read
observed.index.phase The index's own phase: loading, scanning, indexing, ready, waiting, standby (a replacement taking over after a change; searches are answered meanwhile)
observed.index.last_indexed_at When its last pass finished
observed.index.dimensions The embeddings' dimensions
observed.machine The machine its index runs on
observed.address Its address for the workspace's runs
observed.revisions, .indexed_revisions For a drive from a data source: the revision it reads, and the revision a finished pass indexed
observed.warnings What you should know — a model that scales to zero
evaluation The last evaluation: k, cases, recall_at_k, mrr, at

States#

State Meaning
Pending Waiting for a machine that can read its drives and reach its models, or for a data source's first listing
Starting Its index is loading, or a replacement is taking over
Indexing Reading documents; searches are answered from what is indexed
Ready Everything indexed; watching its sources
Waiting Its index waits — for its embedding model most often; the reason names what
Failed It cannot run as asked: a drive or deployment that does not exist, a deployment that serves something else

Files read#

Kind Extensions
md .md, .markdown, .mdx
txt .txt, .text, .rst, .adoc, .org, .csv, .tsv, .json, .jsonl, .log, .tex
html .html, .htm, .xhtml
pdf .pdf (its text layer, page by page)
docx .docx

Other files are counted by extension (skipped). Hidden files and folders are not read. Per file: at most 64 MB; a PDF is given 120 s to read.

Limits#

Limit Value
Sources per base 16
Question 4000 characters
Results per search (top_k) 1 to 50, default 5
Candidates per retriever 50 (more when top_k is large)
Candidates the reranker orders 20 to 50
Waiting for the embedding model on a search 10 s, then keywords only
Waiting for the reranker 10 s, then the fused order
Evaluation 500 questions; must finish within 60 s; k 1 to 50
Documents listed per page 1000 (default 100)

Errors#

Code Status Meaning
INVALID_KNOWLEDGE_BASE 400 The spec is invalid, or names a drive or deployment that does not exist or serves something else; the message says which
KNOWLEDGE_BASE_NOT_FOUND 404 No such knowledge base
KNOWLEDGE_BASE_EXISTS 409 One of that name exists
KNOWLEDGE_BASE_CONFLICT 409 It changed at the same time; retry
KNOWLEDGE_BASE_NOT_READY 503 Its index is not serving yet; the message says its state and why
QUERY_EMPTY, QUERY_TOO_LONG 400 The question is empty, or longer than 4000 characters
TOP_K_OUT_OF_RANGE 400 top_k (or k) is not between 1 and 50
NO_EMBEDDING_MODEL 400 mode: dense without an embedding model
EMBEDDING_MODEL_UNAVAILABLE 503 mode: dense and the embedding model did not answer
INVALID_BODY 400 The body is not a search
EVAL_EMPTY, EVAL_FILE, EVAL_CASE_INVALID, EVAL_TOO_LARGE 400 No questions; the file could not be read or is not JSON Lines; a question without an expected document; more than 500
STANDBY 409 A reindex while a replacement is taking over; ask again in a moment