---
title: "Nodes as services"
description: "systemd, launchd and Windows services: a node that starts at boot, comes back after a crash, updates itself and stays down once purged."
canonical_url: "https://docs.oz0.ai/fleet/services"
---
# Nodes as services

> systemd, launchd and Windows services: a node that starts at boot, comes back after a crash, updates itself and stays down once purged.

For: cluster operators.

Install a service after joining the node so it starts at boot and returns after a crash. Check the account, data path and log location for its operating system below.

A node should keep running without a terminal open. The install command the [console](https://docs.oz0.ai/getting-started/web-ui) shows installs it as a service of the machine's own service manager, and `oz0-node service install` does the same for a node you joined by hand.

<table>
<thead>
  <tr>
    <th>
      
    </th>
    
    <th>
      Linux
    </th>
    
    <th>
      macOS
    </th>
    
    <th>
      Windows
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Service manager
    </td>
    
    <td>
      systemd
    </td>
    
    <td>
      launchd
    </td>
    
    <td>
      the service control manager
    </td>
  </tr>
  
  <tr>
    <td>
      As root or Administrator
    </td>
    
    <td>
      a system unit, as the account <code>
        oz0-node
      </code>
    </td>
    
    <td>
      a launch daemon, as the user who ran <code>
        sudo
      </code>
      
       (or <code>
        --account
      </code>
      
      )
    </td>
    
    <td>
      a Windows service, as its virtual account <code>
        NT SERVICE\orchestrator-zero-node
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Its data
    </td>
    
    <td>
      <code>
        /var/lib/orchestrator-zero-node
      </code>
    </td>
    
    <td>
      <code>
        /Library/Application Support/orchestrator-zero-node
      </code>
    </td>
    
    <td>
      <code>
        C:\ProgramData\orchestrator-zero-node
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Otherwise
    </td>
    
    <td>
      a unit of your user's systemd
    </td>
    
    <td>
      a launch agent of your user
    </td>
    
    <td>
      (Windows services always belong to the machine)
    </td>
  </tr>
  
  <tr>
    <td>
      Its log
    </td>
    
    <td>
      <code>
        journalctl -u orchestrator-zero-node -f
      </code>
    </td>
    
    <td>
      <code>
        &lt;data-dir>/logs/node.log
      </code>
    </td>
    
    <td>
      <code>
        &lt;data-dir>\logs\node.log
      </code>
    </td>
  </tr>
</tbody>
</table>

A service of the machine runs a copy of the binary in `<data-dir>/bin`, which its account owns, so that the node can [replace it when it updates itself](https://docs.oz0.ai/fleet/releases). Nobody but that account (and root, SYSTEM and Administrators) can read the data directory, which holds the node's key.

## Install

Join first, then install the service with the same data directory:

```bash [Terminal]
sudo OZ0_JOIN_TOKEN='K10...' oz0-node join --data-dir /var/lib/orchestrator-zero-node --server https://edge.example.com:7443
sudo oz0-node service install
```

`service install` makes the account when it does not exist, gives it the data directory, writes the unit, enables it and starts it, and shows its state:

```text
Made the account oz0-node, which the node runs as.
Installed the node as the systemd system service orchestrator-zero-node (/etc/systemd/system/orchestrator-zero-node.service).
  Service  orchestrator-zero-node (systemd, system)
  State    active (running), pid 2814
  Runs     /var/lib/orchestrator-zero-node/bin/orchestrator-zero-node --data-dir /var/lib/orchestrator-zero-node run
  As       oz0-node
  Logs     journalctl -u orchestrator-zero-node -f
```

Without root it installs a service of your user instead. On Linux it also turns on lingering (`loginctl enable-linger`), so that the node starts at boot and keeps running after you log out; `--no-linger` leaves it off. A launch agent on macOS runs while you are logged in; for a Mac nobody logs in to, install it with `sudo`.

<table>
<thead>
  <tr>
    <th>
      Flag
    </th>
    
    <th>
      What
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        --user
      </code>
      
      , <code>
        --system
      </code>
    </td>
    
    <td>
      a service of your user or of the machine (default: the machine as root or Administrator)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --account NAME
      </code>
    </td>
    
    <td>
      the account a service of the machine runs as
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --service-name NAME
      </code>
    </td>
    
    <td>
      another name, for a second node on the machine with its own <code>
        --data-dir
      </code>
      
       (launchd label <code>
        ai.oz0.&lt;name>
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --no-start
      </code>
    </td>
    
    <td>
      install and enable it, but do not start it now
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --dry-run
      </code>
    </td>
    
    <td>
      print the unit, property list or service settings, and change nothing
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --check
      </code>
    </td>
    
    <td>
      check that the service can be installed here, and change nothing; the install commands run it before they join
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --replace-binary
      </code>
    </td>
    
    <td>
      put this binary in place of the service's own copy, even when that copy is newer
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --plugin-sandbox
      </code>
    </td>
    
    <td>
      the service's <a href="https://docs.oz0.ai/concepts/security/plugins#plugins-are-trusted-code">
        plugin sandbox
      </a>
      
      : <code>
        auto
      </code>
      
       or <code>
        off
      </code>
      
      , for a node that must not hold its plugins to the egress allowlist. Left out, the node's default
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        --plugin-sandbox-exempt
      </code>
    </td>
    
    <td>
      a plugin whose <code>
        sandbox: host
      </code>
      
       or <code>
        sandbox: own
      </code>
      
       tools the service runs outside the sandbox, <code>
        &lt;plugin>
      </code>
      
       or <code>
        &lt;plugin>@&lt;commit>
      </code>
      
      ; repeat it for more than one. Installing the service again without it takes the exemption away
    </td>
  </tr>
</tbody>
</table>

Run `service install` again to change the service, for example after moving the data directory. A service of the machine keeps its own copy of the binary when that copy is newer than the binary you run `service install` with: the node [updates that copy itself](https://docs.oz0.ai/fleet/releases), while the binary in `/usr/local/bin` stays at the version you first installed, so installing again does not put an older node back.

On Linux and macOS, `service install` asks the copy for its version, running it as the service's account; it replaces a copy of the same or an older release.

A copy whose version has no order to yours, such as a development build's commit, stays, and so does every copy on Windows, where an Administrator running a file the service's account could have written would hand it the Administrator's rights.

`--replace-binary` replaces it anyway.

If the install command cannot install the service after the node has joined, it says why and runs the node in the foreground instead, so the join token is not spent for nothing. Fix the cause, stop the node and run `service install`.

## Check and remove it

```bash [Terminal]
oz0-node service status
sudo oz0-node service uninstall
```

`service status` prints the command line the service's definition holds, so it shows which plugins a node exempts from its sandbox, whatever flags you run it with.

Without `--user` or `--system`, both look for a service of your user first and then for one of the machine, so `service status` finds the service the install command made as root without `sudo`.

Removing a service of the machine needs root (on Windows, an elevated PowerShell). `service uninstall` stops the service and removes it.

On Windows, deleting a service only marks it: Windows removes it once every program that has it open has closed it, and `service uninstall` waits up to 30 seconds for that, since an install right after it would otherwise meet a service "marked for deletion". If a program such as the Services console still has it open after that, the command says so, and Windows removes the service when that program closes it.

The data directory, with the node's identity, and the account stay: `run` or `service install` start the same node again. To take the node out of the cluster, [purge it](https://docs.oz0.ai/fleet/manage-nodes).

## What the service does

- It starts at boot and is started again five seconds after a crash.
- It updates itself. On Linux and macOS the node puts the new binary in place and runs it in the same process, so the service manager sees no restart. A Windows service cannot replace its own process: the node exits without reporting that it stopped, which Windows counts as a failure, and the service's recovery actions start the new binary five seconds later. A new binary that does not reach an edge within two minutes goes back to the previous one either way.
- It stays down once purged. A purge removes the node's identity, and the service only starts while the identity is there: systemd skips the unit (`ConditionPathExists`), launchd keeps the job running only while the file exists, and a Windows service with no identity logs why and stops cleanly. Remove the service with `service uninstall`. A Windows service's purge leaves `bin\` with the running binary in the data directory, since Windows does not delete a running program, together with the marker that stops a `join` with that directory; delete the directory by hand after `service uninstall`.
- It stops gracefully. On stop the node drains: it takes no new work and waits for what runs to end, within its stop grace (ten minutes by default, `service install --stop-grace`), then stops its runtimes and plugins.<br />

The service manager waits for the grace and a minute more before it kills what is left: systemd's `TimeoutStopSec`, launchd's `ExitTimeOut`, and on Windows the service reports its stop as pending while it drains.<br />

When Windows shuts down, it waits for the node as long as the service's preshutdown timeout, which the install sets to the grace and a minute (Windows otherwise gives a service ten seconds); a stop from the Services console stops being watched by the console after 125 seconds, while the node goes on draining.

### The systemd unit

A system unit runs as `oz0-node` with systemd's sandboxing: `NoNewPrivileges`, `ProtectSystem=strict` with only the data directory writable, `ProtectHome`, `PrivateTmp`, protected kernel modules, control groups and clock, and no SUID or realtime.

It leaves out what the node needs: user namespaces and a fresh `/proc` in them for its own [sandbox](https://docs.oz0.ai/concepts/security/sandbox-tiers#sandboxes) (bubblewrap), devices for GPUs, and writable executable memory for Node.js and Python.

`ProtectKernelTunables` and `ProtectKernelLogs` are left out because their mounts over parts of `/proc` make the kernel refuse bwrap's fresh one (`Can't mount proc on /newroot/proc`). This left every such node at tier 0 until this release.

If the node service was installed before this release, run `service install` again to rewrite its unit. Include the original `--data-dir` and `--service-name`, if any, for example `sudo oz0-node service install --data-dir /srv/oz0 --service-name oz0-b`. `oz0-node sandbox check` on the node names both. Without them, the command writes the default unit for the default data directory, not yours.

Run the binary in `/usr/local/bin`, never the copy in `<data-dir>/bin` that the node's account owns.

For container tools, add the account to the `docker` group (`sudo usermod -aG docker oz0-node`); for GPUs, to the groups that own the devices (often `video` and `render`). `--dry-run` prints the whole unit.

The node's temporary files go in `<data-dir>/tmp` (`TMPDIR`), not in the unit's private `/tmp`: the Docker daemon mounts a container tool's egress proxy socket by its path, which it looks up in the machine's own `/tmp`. A data directory whose path is longer than 60 characters keeps the machine's `/tmp` and no private one, so that the sandbox's socket paths stay within the 107 bytes Linux allows.

On Fedora, RHEL and other distributions with SELinux on, `service install` labels `<data-dir>/bin` `bin_t` (with `semanage fcontext` and `restorecon`, or `chcon` where `semanage` is not installed), since systemd does not start a service from a file labelled for `/var/lib`. This has not been tried on an SELinux machine yet; if the unit fails with `status=203/EXEC`, label the folder by hand: `sudo semanage fcontext -a -t bin_t '/var/lib/orchestrator-zero-node/bin(/.*)?' && sudo restorecon -R /var/lib/orchestrator-zero-node/bin`.

## Windows

In an elevated PowerShell (Run as Administrator), the console's *Add nodes* panel has a command for Windows:

```powershell [PowerShell]
curl.exe -fsSk https://edge.example.com:7443/v1/ca.crt -o $env:TEMP\oz0-ca.crt; if ((Get-FileHash $env:TEMP\oz0-ca.crt -Algorithm SHA256).Hash -ne '<sha256 of the CA>') { throw 'the CA certificate does not match' }; $s = curl.exe -fsS --ssl-no-revoke --cacert $env:TEMP\oz0-ca.crt https://edge.example.com:7443/install.ps1; & ([scriptblock]::Create($s -join "`n")) -Token 'K10...'
```

It works the way the Linux and macOS command does: it downloads the cluster's CA certificate, checks it against the hash in the command, and fetches `install.ps1` over TLS that trusts only that CA. The script downloads `orchestrator-zero-node.exe`, checks its digest, installs it in `C:\Program Files\Orchestrator Zero`, joins, and installs and starts the Windows service. It runs as a script block, which PowerShell's execution policy does not stop.

It needs `curl.exe`, which Windows 10 1803 and Windows Server 2019 and later include. `--ssl-no-revoke` is there because the cluster's CA publishes no revocation list for Windows' TLS to check; the certificate is still checked against the CA.

`install.ps1` takes `-Name`, `-DataDir`, `-Dir` for the binary, `-ServiceName`, `-Foreground` to run the node in the window instead (no Administrator needed), and `-NoRun` to install and join only. Windows on arm64 runs the amd64 node, as its Python does.

The service is `orchestrator-zero-node` in *Services*, set to start automatically (delayed), with recovery actions that restart it after 5, 10 and 30 seconds. It runs as its virtual account, which Windows manages and which has no password; the data directory's access list gives it, SYSTEM and Administrators full control and nobody else anything.

`C:\ProgramData` lets every user read what is in it and make folders there, so `join` makes the data directory private to SYSTEM, Administrators and you, and gives it to Administrators, before it writes the node's key in it.

If the data directory is there already and it, or anything in it, belongs to another user or is a link, `join` refuses it before it spends the token: another user may have made it to read the key. Look at who made it, remove it, and run the one-liner again.

Their definitions are tested on every platform, and the monthly platforms workflow installs a real Windows service and a launchd agent on GitHub's hosted runners (`scripts/test-windows.ps1`, `scripts/test-macos.sh`).

Its first run, on 21 October 2026, passed the launchd test on GitHub's macOS runner.

It installed the Windows service, which started, found no identity, said so in its log and stopped cleanly; its last check, that the service is gone after `service uninstall`, failed, because the test still had the service open itself.

The test now closes it first and the uninstall waits for Windows to remove the service; the second run, the same day, passed both.

## Without a service

```bash [Terminal]
oz0-node run
```

runs the node in the foreground, as before. The install command does that by itself where there is no service manager, such as in a container or on WSL without systemd, and with `--foreground`. In containers and on Kubernetes the container runtime is the service manager: see [Docker](https://docs.oz0.ai/fleet/docker) and [Kubernetes](https://docs.oz0.ai/fleet/kubernetes).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.