Keep a base in sync with a data source#

Your documents live in a bucket that other tools write: a docs pipeline publishes there, a team drops policies in it. You want the knowledge base to follow it without anyone pressing a button. You register the bucket as a data source, make a drive of it that is listed again every hour, and make a base of that drive. Each listing that differs is a new revision of the drive; the base reads exactly that revision, and only the files that changed in it.

flowchart LR
  B[Bucket] -- listed every hour --> R[Drive policies<br/>revision r-…]
  R -- new revision: copied to<br/>the base's machine --> K[Base policies]
  K -- reads only what changed --> I[Index]

What you need:

A bucket of yours, below <bucket>, with documents under policies/ (Markdown, text, HTML, PDF, Word), and a machine whose own identity may read it (an EC2 instance role here; other clouds and keys in Data sources).
The embedding model bge-m3 and the reranker bge-reranker, deployed (step 2 of Your company handbook).
A machine with a data location with room for a copy of the documents — the base reads its drive from that machine's disk.
The editor role, and the variables in How the recipes are written.

1. Register the bucket as a data source#

astra has no data source commands: use the console or the API.

ConsoleAPI

Resources in the workspace, the cluster, Data sources → New. Replace the starting point with:

{"metadata": {"name": "lake"}, "spec": {"kind": "S3", "s3": {"bucket": "<bucket>", "auth_type": "instance_profile"}}}

and press Create. Its State becomes Ready.

$ curl -fsS -X POST "$ASTRALYX_API/data-sources" -H "Authorization: Bearer $ASTRALYX_TOKEN" \
    -H 'content-type: application/json' \
    -d '{"metadata": {"name": "lake"}, "spec": {"kind": "S3", "s3": {"bucket": "<bucket>", "auth_type": "instance_profile"}}}' > /dev/null
$ curl -fsS "$ASTRALYX_API/data-sources/lake" -H "Authorization: Bearer $ASTRALYX_TOKEN" | jq -r .status.state
Ready

The data source holds no credential: the machine that lists it uses its own identity, and the files stay in the bucket.

2. Make a drive of its folder, listed every hour#

ConsoleCLIAPI

Drives → New drive. Name: policies. What it is: From a data source (a dataset). Data source: lake. Folder in it: policies. How runs read it: Copy. List it again every: 1h. Create drive.

$ astra astraeus drives create policies --source lake --path policies --mode copy --refresh 1h
policies created; its data source is being listed (astra astraeus drives revisions policies)

$ curl -fsS -X POST "$ASTRALYX_API/drives" -H "Authorization: Bearer $ASTRALYX_TOKEN" \
    -H 'content-type: application/json' \
    -d '{"metadata": {"name": "policies"}, "spec": {"data_source": {"name": "lake", "path": "policies", "mode": "copy", "refresh_seconds": 3600}}}' > /dev/null

Its first listing is its first revision:

$ astra astraeus drives revisions policies
REVISION              FILES       SIZE        LISTED                  CHANGES
r-0b6e2f9c11d4a873 (current)  214         88 MB       2026-10-12T09:00:00Z    first

3. Make the base#

$ astra metis create policies --drive policies --embedding bge-m3 --reranker bge-reranker --rescan-seconds 0
policies: created; its index service starts on one of the workspace's machines and reads its drives (`astra metis get policies`)

--rescan-seconds 0: the base does not look through its files on a timer. A drive from a data source changes only by a new revision, and a new revision starts a pass by itself; files you add to the base start one too. The console (Advanced → Look for changes every (seconds): 0) and the API ("rescan_seconds": 0) set the same.

Until the drive's first listing is done, the base is Pending with waiting for drive policies's data source to be listed (its first revision). Then the revision is copied to the base's machine, and the base indexes it: Starting, Indexing, Ready.

4. See which revision is indexed#

ConsoleCLI

The base's Settings → Sources: for policies, Revision mounted (what the base reads) and Revision indexed (what a finished pass has read). When they are the same, everything in that revision is searchable.

$ astra metis get policies --json | jq '{state: .status.state, mounted: .observed.revisions, indexed: .observed.indexed_revisions}'
{
  "state": "Ready",
  "mounted": {"policies": "r-0b6e2f9c11d4a873"},
  "indexed": {"policies": "r-0b6e2f9c11d4a873"}
}

5. Change a document in the bucket#

Upload a new policy and change two others in <bucket>/policies/. Within the hour — or at once:

$ astra astraeus drives refresh policies
policies: revision r-3f9a1c0d2b7e4a56 (215 files, 89 MB)
$ astra astraeus drives diff policies
r-0b6e2f9c11d4a873 → r-3f9a1c0d2b7e4a56: +1 −0 ~2
  + remote-work.md
  ~ travel.docx
  ~ expenses.pdf

What happens next, without you:

The base's index service is replaced by one that reads r-3f9a1c0d2b7e4a56. The new revision is copied to the machine first; files unchanged since the revision it holds are linked, not copied again.
While the replacement takes over, the base is Starting — a replacement is taking over its index; searches are answered meanwhile — and searches keep being answered from what was indexed.
Its first pass compares each file with what was indexed (size and modification time, then content): it reads the 3 new and changed files, leaves the 212 others alone, and drops any file that is gone.
The base is Ready; Revision mounted and Revision indexed are both r-3f9a1c0d2b7e4a56.

The base's Activity tab shows each pass — Added 1, Changed 2, Removed 0, Unchanged 212, how long it took — and its state changes. A search for the new policy finds remote-work.md.

When to read everything again#

A pass reads only what changed, except when you change the base's embedding model or its chunking: every document is then read and embedded again, by itself. A Full reindex (astra metis reindex policies --full) does the same on demand — after you changed how the model is served, say. Adding a source drive reads its documents; removing one drops them.

A plain drive instead#

A drive that is not backed by a data source (a folder on a machine, a shared filesystem) has no revisions. The base looks through it every rescan_seconds (60 by default) and reads what changed since; Reindex (astra metis reindex <base>) looks at once. Keep rescan_seconds above 0 for such a drive.

Footprint and cost#

What	Where	Size
The listing	The machine serving the data source	Paths, sizes and identities only; file names stay on your machines
The revision the base reads	The base's machine's data location, `drives/policies/<revision>`	The documents' size; the machine keeps the revisions in use and the newest whole one
The index	The base's memory and its drive `metis-policies`	About 5 KB per chunk at 1024 dimensions
Each new revision	The base's models	Only new and changed files are embedded again

A listing every hour reads the bucket's listing, not its files; only changed files are copied and embedded. The base, its models and the machine that lists are metered by the cores and memory they hold, like any work of the workspace; your cloud may charge for the bucket's listing requests and the bytes read from it.

Clean up#

Delete the base (astra metis delete policies), then the drive policies (its copies on the machines go; the bucket is never touched), then the data source lake if nothing else uses it.

Troubleshooting#

Symptom	Cause	Fix
The base stays `Pending`: waiting for drive policies's data source to be listed (its first revision)	The first listing has not finished, or failed	`astra astraeus drives revisions policies`; a failed listing shows why in the drive's events
Revision indexed stays behind Revision mounted	The pass is under way, or waiting for the embedding model	The base's state says which; `Waiting` names the model
A changed file is still found in its old version	The drive has not been listed since	`astra astraeus drives refresh policies`
`Failed`: spec.sources[0].drive: there is no drive policies in this workspace	The drive was deleted	Make it again, or change the base's sources
Nothing changes for hours	`refresh_seconds` is `0` on the drive	Set List it again every on the drive