Create and use sealed secrets#

A sealed secret holds named values that your device encrypts before they are sent; Astralyx stores only the ciphertext, and only the machines an admin approved can decrypt it. This page shows how to create, change and delete sealed secrets and how runs use them. Read Sealed secrets first for how they work and what they protect against. If your workspace has a secret manager, use a credential that references it instead.

In the console, sealed secrets are under Secrets.

Before you begin#

The workspace needs a key. It is made when an admin approves the workspace's first machine: see Approve machines for sealed secrets. Until then, the console shows No key for secrets yet and a write is refused with 409 SEALING_NOT_READY.
Approve every machine that may run the workers that use a secret. A worker placed on a machine that is not approved waits in Preparing.
You need the editor or admin role in the workspace to create, change or delete sealed secrets. Viewers see their names and key names.
Sealed secrets are kept per cluster: the console's cluster picker, and the cluster astra works in (astra use <org>/<workspace>@<cluster>), choose which.
For astra, install and sign in. For the API, set TOKEN and API as in Drives.

Create a sealed secret#

The example below creates the secret app with one key, api-token, and uses a random token so you can run it as written. Use your own value instead.

ConsoleCLIAPI

Open Secrets in the workspace and click New secret.
Enter a Name: lower-case letters, digits and -, at most 63 characters (app). Runs name the secret by it.
Under Key, enter api-token; under Value, paste the value. Show reveals what you typed. From file takes the value from a file instead (a certificate, a key file); the key defaults to the file's name.
Add a key for each further value.
Click Encrypt and create. The browser encrypts each value to the workspace key, then sends the ciphertext. The secret's page opens.

$ head -c 24 /dev/urandom | base64 | tr -d '\n' | astra secrets set app --from-stdin api-token
app: set api-token (1 key in all)
Runs use it as the credential app: its keys are files, or environment variables.

astra secrets set creates the secret or adds keys to it. It takes the values three ways, which you can combine:

Form	The value
`KEY=VALUE` (arguments)	The text after the first `=`. It stays in your shell's history: prefer the other two.
`--from-file KEY=PATH` (repeatable)	The file's bytes.
`--from-stdin KEY`	Standard input, to its end.

$ astra secrets set db username=analytics --from-file ca.pem=./ca.pem
db: set ca.pem, username (2 keys in all)
Runs use it as the credential db: its keys are files, or environment variables.

Values are encrypted on your side, to the workspace key that GET $API/sealing/self returns. seal.py below does it with Python's cryptography package (pip install cryptography), then writes the secret with PUT, which creates it when it does not exist:

seal.py

"""Seal values for an Astraeus sealed secret and write them: PUT /sealed-secrets/<name>.

    python3 seal.py <secret> KEY=VALUE [KEY=VALUE …]     (API and TOKEN in the environment)
"""
import base64, json, os, sys, urllib.request
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF

def b64(b): return base64.urlsafe_b64encode(b).rstrip(b"=").decode()
def unb64(s): return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
POINT = serialization.Encoding.X962, serialization.PublicFormat.UncompressedPoint

def seal(public_key, kid, aad, plaintext):
    recipient = ec.EllipticCurvePublicKey.from_encoded_point(ec.SECP256R1(), unb64(public_key))
    eph = ec.generate_private_key(ec.SECP256R1())
    epk = eph.public_key().public_bytes(*POINT)
    shared = eph.exchange(ec.ECDH(), recipient)
    key = HKDF(hashes.SHA256(), 32, salt=epk + recipient.public_bytes(*POINT), info=b"astraeus-sealed/v1").derive(shared)
    iv = os.urandom(12)
    return {"v": 1, "kid": kid, "epk": b64(epk), "iv": b64(iv), "ct": b64(AESGCM(key).encrypt(iv, plaintext, aad))}

def call(method, path, body=None):
    req = urllib.request.Request(os.environ["API"] + path, method=method, data=json.dumps(body).encode() if body is not None else None,
                                 headers={"Authorization": "Bearer " + os.environ["TOKEN"], "Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)

if __name__ == "__main__":
    name, pairs = sys.argv[1], [a.split("=", 1) for a in sys.argv[2:]]
    view = call("GET", "/sealing/self")
    if not view.get("key"):
        sys.exit(view["message"])
    ns, kid, pk = view["namespace"], view["key"]["kid"], view["key"]["public_key"]
    aad = lambda k: f"astraeus-sealed-secret/v1\0{ns}\0{name}\0{k}\0{kid}".encode()
    s = call("PUT", f"/sealed-secrets/{name}", {"set": {k: seal(pk, kid, aad(k), v.encode()) for k, v in pairs}})
    print(name, "keys:", ", ".join(s["data"]))

$ python3 seal.py app api-token=$(head -c 24 /dev/urandom | base64 | tr -d '\n')
app keys: api-token

The format is in Encryption format. POST $API/sealed-secrets with {"metadata": {"name": "app"}, "data": {"api-token": <envelope>}} also creates a secret, and answers 409 SEALED_SECRET_ALREADY_EXISTS if it exists.

Creating a sealed secret also creates a credential of the same name (backend Astralyx (sealed), labelled astraeus.io/sealed-secret), unless a credential of that name already exists. Runs use the secret through it.

Use it in a run#

A run names the secret's credential in secret_refs, as for any credential. Each key of the secret is a file of the credential's directory, so env_vars and mount_path name keys directly. The directory also holds value: every key and its value as one JSON object (unless the secret has a key named value).

This run, on any machine approved for the workspace's secrets, lists the files, prints the token through its environment variable, and counts its bytes:

ConsoleAPI

Open Runs and click New run.

Image busybox:1.36. Under What it runs, choose A script and enter:

ls /run/secrets/app && echo "token: $API_TOKEN" && wc -c < /run/secrets/app/api-token

Under Credentials, click + Add a credential. Credential app, As files at /run/secrets/app, As variables api-token=API_TOKEN.
Click Start run.

sealed-check.json

{
  "metadata": {"name": "sealed-check"},
  "spec": {
    "task_template": {
      "image": "busybox:1.36",
      "command": "sh",
      "args": ["-c", "ls /run/secrets/app && echo \"token: $API_TOKEN\" && wc -c < /run/secrets/app/api-token"],
      "restart_policy": "Never",
      "requested_resources": {"cpu_cores": 1, "memory_bytes": 268435456},
      "secret_refs": [
        {"external_secret_name": "app", "env_vars": {"api-token": "API_TOKEN"}, "mount_path": "/run/secrets/app"}
      ]
    }
  }
}

$ curl -sS -X POST "$API/jobs" -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" -d @sealed-check.json

Its log shows the files, the token masked, and its length:

api-token
value
token: ***
32

The machine decrypts the values in memory, writes them to the run's credential directory on the machine (0700, files 0600) and masks every value in what the run's logs show (see Log masking). When no worker on the machine needs the secret any more, its directory is deleted.

A sealed secret's credential works wherever a credential is named: runs, schedules' and replica groups' templates, data sources, and agents' connections and environment variables.

Change a sealed secret#

Values cannot be read back — not by you, not by Astralyx. You replace them.

ConsoleCLIAPI

Open Secrets and click the secret. Keys lists each key and the workspace key it is Sealed to.
Replace adds a row for a key: enter its new value. Remove marks a key for removal (Keep undoes it). Add a key adds a new one.
Click Encrypt and save.

astra secrets set adds or replaces only the keys you give:

$ astra secrets set app --from-stdin api-token < new-token.txt
app: set api-token (1 key in all)
Runs use it as the credential app: its keys are files, or environment variables.
$ astra secrets keys app
api-token                                sealed to w5e0c8a71d24b9f63

astra secrets cannot remove a single key: use the console or the API.

PUT $API/sealed-secrets/<name> with set (keys added or replaced, each an envelope) and remove (key names):

$ curl -sS -X PUT "$API/sealed-secrets/db" -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" -d '{"remove": ["username"]}'

A secret keeps at least one key: removing the last is refused with 400 INVALID_SEALED_SECRET.

A changed value reaches the machines that hold the secret within seconds; they rewrite its directory. As with every credential, a running worker keeps the environment variables it started with, and a worker that mounted the directory keeps the directory it started with: restart the run to use the new value (see Rotation and refresh).

List and delete#

ConsoleCLIAPI

Secrets lists each secret's Name, Keys, when it was Updated and By whom. To delete one, open it and click Delete.

$ astra secrets list
NAME                              KEYS  UPDATED     BY
app                                  1  2m ago      user:8f14e45f-ceea-467f-a0e6-b1c5c2e0a9d3
db                                   2  1h ago      user:8f14e45f-ceea-467f-a0e6-b1c5c2e0a9d3
$ astra secrets delete db
db deleted

astra secrets ls and astra secrets rm are the same commands. delete takes several names.

$ curl -sS "$API/sealed-secrets" -H "Authorization: Bearer $TOKEN"
$ curl -sS -X DELETE "$API/sealed-secrets/db" -H "Authorization: Bearer $TOKEN"

A read returns each key's envelope (ciphertext), never a value. DELETE answers 204 No Content.

Deleting is final

Deleting a sealed secret deletes its values and the credential made for it. Nobody can recover them: Astralyx never could read them. Runs that still use the secret wait in Preparing.

Limits#

Limit	Value
Name	Lower-case letters, digits and `-`; at most 63 characters; starts and ends with a letter or digit
Key name	Letters, digits, `.`, `_` and `-`; at most 128 characters; not `.` or `..` (it becomes a file name)
Keys per secret	1 to 64
One value	64 KiB before encryption
One secret	256 KiB of ciphertext, all keys together

Encryption format#

What astra, the console and seal.py do, for writing your own client. Every key and field is unpadded base64url; public keys are uncompressed P-256 points (65 bytes).

Read namespace, key.kid and key.public_key from GET $API/sealing/self.
For each value: make an ephemeral P-256 key pair; ECDH with the workspace public key; derive 32 bytes with HKDF-SHA256 (salt: the ephemeral public key followed by the workspace public key; info: astraeus-sealed/v1).
Encrypt with AES-256-GCM, a random 12-byte IV, and as associated data astraeus-sealed-secret/v1 \0 namespace \0 secret name \0 key \0 kid. The secret name is the one you write (app).
Send {"v": 1, "kid": <kid>, "epk": <ephemeral public key>, "iv": <iv>, "ct": <ciphertext and 16-byte tag>} as the key's value.

A value encrypted to another workspace key, or anything that is not an envelope, is refused.

Troubleshooting#

Symptom	Cause	Fix
No New secret button; No key for secrets yet. `astra`: This workspace has no key for secrets yet: approve a machine first. API: `409 SEALING_NOT_READY`	No machine is approved yet, or the first approved machine has not made the key.	An admin approves a machine. If one is approved, it makes the key when its agent is online.
`409 SEALED_SECRET_CONFLICT`: key "api-token" is sealed to workspace key w… ; the workspace key is w… now (read it again and encrypt again)	The workspace key was rotated after your client read it.	Read `GET $API/sealing/self` again and encrypt again. The console and `astra` do this once by themselves.
`409 SEALED_SECRET_CONFLICT`: the secret or the workspace key changed meanwhile	Someone changed the secret, or the key rotated, while you wrote.	Read again and retry.
`400 INVALID_SEALED_SECRET`	A key name that is not a file name, too many keys, a value too large, or a value that is not an envelope.	Read the message; see Limits.
Worker waits in `Preparing`: `waiting for external secret app (key api-token) to be synced to this node`; the credential shows `Error`: this machine does not hold workspace key w… (is it approved for this workspace's secrets?)	The worker was placed on a machine not approved for the workspace's secrets, or whose approval is still `Waiting`.	Approve the machine, or keep these runs on approved machines with a pool or machine labels.
The credential shows `Error`: this machine has no sealing key	The machine's credentials agent could not read or create its key file.	Check `/var/lib/astraeus/credentials/sealing.key` on the machine: a damaged file is never replaced.
The credential shows `Error`: key "…" does not open (tampered with, or moved from another secret)	The ciphertext was not encrypted for this secret, key and workspace key.	Set the value again from the console or `astra`.

Create and use sealed secrets#

Before you begin#

Create a sealed secret#

Use it in a run#

Change a sealed secret#

List and delete#

Limits#

Encryption format#

Troubleshooting#

Related#