Troubleshoot a knowledge base#
A knowledge base always says what it is doing and why, in words: its state and reason, a reason for each document it could not read, and warnings on a search that could not do everything it was asked. Start there. This page lists each one, from the code that writes them, with what to do.
Where to look#
- Metis → Knowledge bases: each base's state, its reason under it, and the pass's progress.
- The base's page: its state and reason at the top, warnings below (a model that scales to zero), Documents → Everything indexed (tick to show only those that failed, each with why), Activity (its passes and its state changes, newest first).
- Metis → Overview → Documents that could not be read: across every base.
$ curl -fsS "$ASTRALYX_API/metis/knowledge-bases/handbook" -H "Authorization: Bearer $ASTRALYX_TOKEN" | jq '.status, .observed'
$ curl -fsS "$ASTRALYX_API/metis/knowledge-bases/handbook/status" -H "Authorization: Bearer $ASTRALYX_TOKEN" | jq '{phase, message, errors, skipped}'
$ curl -fsS "$ASTRALYX_API/metis/knowledge-bases/handbook/history" -H "Authorization: Bearer $ASTRALYX_TOKEN"
GET …/status is the index's own account, read from its machine:
its phase and message (why it waits, in its words), the first
errors (documents and folders it could not read), and skipped
(files left out, counted by extension).
States and reasons#
Pending#
| Reason | Cause | Fix |
|---|---|---|
| Created | Just made | Wait a few seconds |
| waiting for a machine: … | No machine can run its index service now; the rest of the reason says what is short (CPU cores, memory, a data location, a machine that can read its drives and reach its models) | Free a machine, lower placement.cpu_cores or memory_gb, widen placement.machines, or choose a data location on a machine |
| waiting for a machine to run its index service | Its service has not been placed yet | Wait; if it lasts, see the reason above |
| waiting for drive X's data source to be listed (its first revision) | A source drive from a data source has never been listed | astra astraeus drives revisions X; a failed listing shows why on the drive |
| drive X could not be read: …, drive X's revision could not be read: … | The drive's record could not be read just then | Wait; it is checked again |
| spec.embedding.deployment: deployment X could not be read: … (or reranker, extractor) | The deployment's record could not be read just then | Wait; it is checked again |
Starting#
| Reason | Meaning |
|---|---|
| its index service is starting on M | The service is starting on machine M (its image pulled the first time) |
| its index service started on M; its first totals are read within a minute | Started; its totals are not read yet |
| loading its index from its drive | Reading the index it kept, after a restart |
| a replacement is taking over its index; searches are answered meanwhile | The base changed (its spec, a model's address, a drive's new revision): a new service is taking over from the old one, which answers searches until it stops |
Indexing#
| Reason | Meaning |
|---|---|
| looking through its sources | Its first pass is finding what to read |
| indexing: N of M documents read; D documents, C chunks searchable | A pass is reading new and changed documents; what is indexed is searchable. (F could not be read: see its documents) when some failed |
Ready#
D documents, C chunks, and (F could not be read: see its documents) when some failed. Everything it can read is indexed, and it watches its sources. Failed documents are not tried again until they change.
Waiting#
The base serves searches from what it has indexed, but its index cannot go on.
| Reason | Cause | Fix |
|---|---|---|
| waiting for its embedding model, deployment X (State: why); D documents, C chunks searchable | The embedding deployment is not Ready |
Follow X in Eos → Deployments; the pass resumes where it stopped once it serves |
| its index waits: its status says why (the embedding model not answering, its drive); … | The deployment is Ready but the index cannot reach it, or cannot use its drive |
Read GET …/status → message (below) |
| A data source drive's reason, while the base serves | As in Pending |
As in Pending |
The index's own message while it waits:
| Message | Fix |
|---|---|
| the embedding model at … could not be reached (is it running?); indexing resumes when it answers (N of M documents left) | The deployment has no replica serving, or the base's machine does not reach it. Keep it at one replica at least (one that scales to zero is not woken by the base) |
| the embedding model at … did not answer within N s; … | The model is overloaded or too slow on its machine: give it more cores, or another replica |
| the embedding model failed (status): … | Read the deployment's log |
| the index on its drive could not be read: … | The base's own drive (metis-<base>) is damaged or unreadable: a Full reindex reads everything again |
| the index's drive cannot be written: … | The data location of the base's machine is full or read-only: free space |
Failed#
It cannot run as asked, and nothing runs until you change it.
| Reason | Fix |
|---|---|
| spec.sources[i].drive: there is no drive X in this workspace | Make the drive, or remove it from the base's sources |
| spec.embedding.deployment: there is no deployment X: deploy an embedding model (…) first | Deploy it (BGE-M3 runs on a CPU), or name another |
| spec.reranker.deployment: there is no deployment X: deploy a reranker (…) first | Deploy BGE Reranker v2 M3, or remove the reranker |
| spec.extractor.deployment: there is no deployment X: deploy a small chat model (…) first | Deploy Qwen3 4B, or remove the extractor |
| …: deployment X serves chat, not embedding: name a deployment of an embedding model (or not rerank, or serves embedding, not chat) | Name a deployment of the right kind |
| a replica group named metis-X exists and is not this knowledge base's: rename one of them | A replica group of yours has the base's service's name: rename the group or the base |
| the index service's run: … | The settings make a run that cannot be placed (the message says which); change placement |
The base's page also shows warnings: its embedding model, deployment X, scales to zero: the index's calls do not wake it — keep it at one replica at least (or its reranker). Clear Scale to zero when idle on that deployment.
Documents that were not indexed#
Each document that could not be read keeps its reason, in Documents
and astra metis documents <base> --failed; it is tried again only when
it changes.
| Reason | Fix |
|---|---|
| the PDF has no text layer (a scanned document?): text in images is not read | There is no OCR yet: export the document with text from its source, or run OCR on it yourself and replace the file |
| the PDF is encrypted (password-protected) | Save a copy without a password |
| not a readable PDF: … | The file is damaged or not a PDF |
| pdftotext took longer than 120 s on it | A very large or complex PDF: split it |
| PDFs are read with pdftotext (poppler-utils), which this machine's index service does not have | The index service runs without its PDF reader on that machine: report it to your administrator |
| not a DOCX (it has no word/document.xml): an older .doc, or another kind of archive? | Save it as .docx |
| not a readable DOCX: … | The file is damaged |
| it is N MB, more than the 64 MB read | Split it, or export a smaller version |
| it has no text | Empty, or only images |
| could not be read: … | A permission or disk error on the drive |
| the embedding model refused it (400): … (a chunk of up to N characters: lower chunking.size if the model's window is smaller) | A chunk is longer than the model takes: lower chunking.size (everything is read again) |
Problems with a source itself appear among the status's errors, with the
source's folder as the document:
| Reason | Fix |
|---|---|
| there is no folder P/ on drive X | The source's path does not exist in the drive: fix it |
| drive X is not mounted (no folder at …) | The drive could not be mounted on the base's machine: see the drive's page |
| the folder P leaves the drive | Remove .. from the path |
Files that are not a kind a base reads (.doc, .pptx, .xlsx, images)
are not errors: they are counted by extension in skipped. Hidden files
and folders are not read.
Added files#
Documents → Added to this base and astra metis files give each added
file a state: pending (not read yet; a pass starts at once), indexed
(with its chunks), failed (with one of the reasons above), or
evaluation-set (a .metis/*.jsonl file, never indexed).
Search warnings#
A search answers what it can and says what it could not do (When a model does not answer):
| Warning | Fix |
|---|---|
| the embedding model at … did not answer within 10 s: results by keywords only (or could not be reached, failed) | Check the embedding deployment |
| the reranker at … did not answer within 10 s: results in the fused order (or could not be reached, failed) | Check the reranker deployment |
| no chunk has a vector yet (indexed while the embedding model did not answer): results by keywords | Once the model serves, a Full reindex embeds them |
| nothing is indexed yet: the index is … | Wait for the pass |
| nothing is indexed: the sources hold no document Metis reads (Markdown, text, HTML, PDF, DOCX) | Check the sources' folders and the files' kinds |
Error codes#
| Code | Status | Cause | Fix |
|---|---|---|---|
KNOWLEDGE_BASE_NOT_READY |
503 | No index service serves the base yet; the message gives its state and why | Wait for Indexing, Ready or Waiting; for Failed, fix the reason |
KNOWLEDGE_BASE_UNREACHABLE |
503 | The base's service did not answer (its machine restarting, a network break) | Retry |
KNOWLEDGE_BASE_TIMEOUT |
503 | A request about files, notes, drafts or facts did not answer in time (60 s; 10 minutes for an upload or an extraction) | Retry; a smaller file |
UPSTREAM_TIMEOUT |
504 | A search, an evaluation or a reindex took longer than 60 s | Split an evaluation; check the models |
EMBEDDING_MODEL_UNAVAILABLE |
503 | mode: dense while the embedding model does not answer |
Use hybrid or keyword meanwhile |
NO_EMBEDDING_MODEL |
400 | mode: dense on a base without one |
Use keyword |
STANDBY |
409 | A reindex asked while a replacement takes over | Ask again in a moment |
FILE_EXISTS |
409 | A file of that path was added before | Replace it (--replace, ?overwrite=true) |
FILE_TOO_LARGE |
413 | Over 64 MB | Split it |
UNSUPPORTED_TYPE |
400 | Not a kind a base reads, or not .jsonl under .metis/ |
Export it as PDF, Word, Markdown or text |
INVALID_PATH |
400 | ./.., an empty part, a hidden name, or over 400 characters |
Rename it |
FILE_EMPTY |
400 | An empty file | — |
UPLOAD_INTERRUPTED |
400 | The upload broke off; nothing was kept | Add it again |
NOTES_UNWRITABLE |
500 | The base's drive cannot be written (its data location full) | Free space on the base's machine |
NOTES_UNAVAILABLE |
409 | The base's service started without its own files folder; its next update gives it one | Retry in a minute |
NOTE_INVALID, NOTE_TOO_LARGE |
400, 413 | No title or text, an unknown source; over 256 KB | Fix it; upload long text as a file |
DRAFT_NOT_FOUND |
404 | The proposal was saved, discarded, or a day passed | Ask again |
FACT_INVALID, FACT_SENSITIVE |
400 | A fact missing a part, too long, over 100 at once; one that looks like a secret | Fix it; never keep secrets |
NO_EXTRACTOR |
409 | A memory without an extractor | Set spec.extractor, or keep facts by hand |
EXTRACTOR_UNAVAILABLE, EXTRACTOR_REFUSED, EXTRACTOR_UNREADABLE |
503, 502, 502 | The extractor did not answer, refused, or did not answer with facts | Check the deployment; try again or use a larger model |
Every code is in the specification.