Remote workers¶

A worker runs node code on another machine: a GPU host, a Raspberry Pi or a batch allocation. It connects outward to the engine over an authenticated WebSocket. Keep one engine; add workers where the hardware lives.

Create a worker token¶

On the engine, use a superuser access token. In these examples:

• API_URL is the engine's API base, including /api/v1, such as https://fluksio.example.com/api/v1 or http://localhost:8080/api/v1 for a Docker container.
• TOKEN is your superuser bearer token. The login endpoint POST /api/v1/login/access-token accepts form fields username and password.

Mint a token with the labels this machine should carry:

curl -X POST "$API_URL/workers/tokens" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'

The response contains token, jti and the expiry. Save the token as FLUKSIO_WORKER_TOKEN on the worker host. It is returned once and is valid for 365 days. The signed name and labels control which nodes and secrets the engine sends to that worker.

Install and attach¶

Use Python 3.12 or later on the worker host:

pip install fluksio-worker
export FLUKSIO_WORKER_TOKEN='<token from the engine>'

fluksio-worker \
  --url wss://fluksio.example.com/api/v1/workers/attach \
  --labels gpu,cuda12 \
  --python /opt/torch-venv/bin/python

Use your engine's hostname. --python selects the environment containing the node's dependencies, such as PyTorch or a device SDK. If you install the worker in that environment, you can omit the option. An engine installation also provides the same agent as fluksio worker.

For a trusted LAN stack, the URL is ws://<host-ip>:8080/api/v1/workers/attach and requires --insecure. Outside loopback, the worker otherwise requires encrypted URLs (wss:// and https://): node secrets and the worker token travel over the connection.

Node code receives a filtered host environment. Credential-like variables are removed; pass credentials through Secrets.

Send a node to it¶

Set these fields on a node through PUT /api/v1/flows/{name}:

{
  "id": "train",
  "worker": "gpu",
  "worker_policy": "require",
  "timeout": 7200
}

This is a node fragment; keep the node's other fields and the rest of the flow when updating it. worker and worker_policy are not in the node panel yet.

Source code travels with each call and compiles on the worker, so remote-only dependencies do not need to be installed on the engine. Calls to emit, save_artifact and load_artifact work there too.

Cancelling a run stops its active node processes. A disconnected worker fails in-flight calls. While executing, the agent sends heartbeats every ten seconds; 90 seconds without contact fails the call. Heartbeats do not reset the node's own silence timeout.

Resources and artifacts¶

Workers report CPU, GPU and memory inventory, and scheduling checks both labels and available resources. CPU and memory detection uses Slurm allocations when present, otherwise host figures. GPU detection uses SLURM_GPUS_ON_NODE, FLUKSIO_WORKER_GPUS or SLURM_JOB_GPUS before counting NVIDIA device nodes. Explicit flags override detection.

Artifacts are identified by content hash. Remote workers fetch and upload over HTTP beside the WebSocket endpoint; node code uses the same artifact helpers as on the engine. Downloads are cached by digest:

Oldest files are removed when the cache exceeds its limit. Raise it if workers repeatedly use large inputs.

The Workers screen shows attached hosts, available resources and queued nodes. The API exposes the same information:

curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"

Revoke a token¶

List tokens and revoke one by its jti:

curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
  -H "Authorization: Bearer $TOKEN"

Revocation also disconnects attached workers and blocks artifact access. Replacing the OAuth signing key revokes all worker tokens. Older tokens without signed labels and a jti need to be replaced.

Upgrading workers¶

Engine and worker must speak the same protocol. A worker on another protocol is refused and exits, so a release that bumps the protocol needs pip install -U fluksio-worker on every worker host. Upgrade the engine first, then the workers.

The Workers screen shows each worker's package version and protocol. A worker refused for an older protocol stays listed as Needs upgrade until it attaches again, or until the engine restarts.

Slurm provisioning¶

The engine can submit a Slurm job that starts a worker. Put this configuration in $DATA_DIR/provisioners.json, replacing the example addresses and paths:

[{
  "type": "slurm",
  "name": "hpc",
  "login": "me@login.cluster",
  "ssh_key": "/secrets/hpc_ed25519",
  "engine_url": "wss://fluksio.example.com/api/v1/workers/attach",
  "max_idle_s": 300,
  "provision_timeout_s": 900,
  "profiles": [{
    "name": "gpu-small",
    "cpus": 8,
    "gpus": 1,
    "ram_mb": 65536,
    "labels": ["gpu"],
    "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
    "prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"]
  }]
}]

The engine uses system SSH to submit sbatch. Prepare the environment named by prerun with fluksio-worker and your node dependencies; the generated job does not install packages. Compute nodes must be able to connect to the engine. Each job receives its own worker token, valid for 24 hours.

Only one request per profile is outstanding at a time. Jobs that fail to attach within provision_timeout_s are cancelled, as are pending requests when the engine shuts down. max_idle_s releases idle workers. Either setting accepts 0 to disable its limit. Provisioning status appears under GET /api/v1/workers/resources.

See also¶

• Runs: remote execution.
• Writing node code: node functions and artifact helpers.
• Getting started: data science: resource declarations.