The Operations screen#
Astraeus → Operations is one screen to run the cluster from: what is up, what is busy and what is wrong, over charts of the span you choose. It is made for a wall or a second monitor.
What it shows#
Now, in cards: Machines up (of all), GPUs in use (held by work, of all), Runs running and waiting, Deployments ready, and how many Alerts need action.
Alerts, worst first, each opening the machine, run or deployment:
| Alert | When |
|---|---|
| Machine … is down (or another state) | A machine not Up |
| Machine … takes no new work (cordoned) | A machine cordoned (information) |
| GPU fault (Xid) on … | An Xid error in the span |
| GPU at 85 °C on … | A GPU at 85 °C or more: its clocks are held down soon |
| Disk 92% full on … | A filesystem with less than 10% free |
| … ran out of memory | A worker killed for running out of memory in the span |
| Run … failed | A run that failed in the span |
| Deployment … is pending (or failed) | A deployment that should serve and does not, with why |
Charts, by machine unless said: GPU utilisation, GPU memory in use, GPU power, the hottest GPU's temperature, CPU in use, memory available, network received and sent; the workers' memory and CPU by workspace; GPU faults (Xid); workers out of memory.
Span, refresh and full screen#
- Span: 5 minutes, 15 minutes, 1 hour (default), 6 hours, 24 hours or 7 days.
- Refresh: off, every 5 s, 10 s, 30 s (default), 1 minute or 5 minutes.
- Live: the last 5 minutes, read every 5 seconds.
- Full screen: the screen alone, larger, for a wall; Esc leaves it.
The span and the refresh are kept in the address: a link opens the screen as it was set. A workspace bound to several clusters chooses one.