Bounded runs on a far machine

A run placed on a machine reached over ssh can start processes the server cannot see: a test suite, a build, a shell a tool opened. Without a bound, one of them can fork or allocate until the whole machine runs out of memory, and it can keep running after the run that started it has ended.

So on a far Linux machine whose login account runs a user systemd manager, every run starts in a transient scope of its own, and that scope is stopped when the run returns.

What the bound is

When the run returns, its scope is stopped and every process it left behind ends with it. The run's own exit status is kept.

⚠ The values are fixed in this release. They are not settable per run or per machine.

How many runs a machine carries at once

Separately from the per-run bound, a machine has a live-run ceiling: the number of runs it may carry at once, across all its seats. A dispatch that would exceed it is refused by name.

The default for every machine is the configuration key scheduler.host_max_live. To set one machine's ceiling, as the owner:

1. Read saphan machine list --workspace <workspace root> and find the machine.
2. Rehearse: saphan dry-run machine set-max-live --machine <id> --max <n> --added-by <owner> --workspace <workspace root>. It writes nothing.
3. Run the same line without dry-run at the owner's terminal and confirm. saphan machine list now shows max-live=<n> on that machine.
4. To return the machine to the default, repeat steps 2–3 with --max 0.

A certificate renewal keeps the ceiling. Re-admitting a retired machine does not: set it again afterwards.

⇒ Size the ceiling from memory, not from seat count. A machine carries the memory bound once per live run, so ceiling × 20 GiB should fit in what the machine has free after its own services.

Reading which bound a run had

Every run's stderr.log carries one line that starts run-bound:, written on the far machine before the run starts. It has two forms.

In the unit name, every character of the run id other than a letter, a digit, - or _ is written as _.

⚠ A macOS machine always takes the second form: it has no user systemd manager. The same is true of a Linux account that has no user session manager running.

Listing and stopping bounded runs on a machine

On the far machine, as the account the runs use:

1. List the live runs: systemctl --user list-units 'saphan-leg-*'
2. Read one run's usage: systemctl --user status saphan-leg-<run id>.scope — it prints the task count and memory in use, and the processes in it.
3. Stop one run and everything it started: systemctl --user stop saphan-leg-<run id>.scope

⚠ Stopping a scope by hand ends the run's processes, including the agent itself; the run does not finish its work.

Reading why a run ended on its bound

While the scope still exists, its cgroup counters say which bound was met:

1. Find the cgroup: systemctl --user show -p ControlGroup saphan-leg-<run id>.scope
2. Read pids.events there: a max above 0 means a fork was refused at the task bound.
3. Read memory.events there: high counts throttling at 20 GiB; oom_kill above 0 means the hard bound ended a process.

⚠ The counters go away with the scope. Once the run has returned and its scope is stopped, these files no longer exist; the machine's journal (journalctl --user -u saphan-leg-<run id>.scope) is what remains.

What this does not cover

• A run on the server's own machine (not reached over ssh) is not bounded this way.
• A process that escapes its scope on purpose — for example by asking the user manager for a scope of its own — is outside the bound.