Remote workers¶

The engine runs where the automations are. The GPU is somewhere else, the Raspberry Pi with the relays is in a shed, and neither of them is on the same network as the other.

A worker is a process that runs the code of nodes marked for it. It dials out to the engine over one authenticated websocket, so nothing on that machine has to be reachable — and nothing has to expose the engine's state backend across hosts, which it never should.

Install and attach¶

pip install fluksio-worker

fluksio-worker \
  --url wss://api.example.com/api/v1/workers/attach \
  --token "$FLUKSIO_WORKER_TOKEN" \
  --labels gpu,cuda12 \
  --python /opt/torch-venv/bin/python

fluksio-worker is its own distribution — the agent, the node runner, and websockets. Nothing of the engine, so a GPU box does not install a database driver in order to run a training step. An engine host already has it, and fluksio worker … is the same program.

--python is the important one. It is how this machine keeps its own wheels — the CUDA build, the vendor SDK, the thing that will not install anywhere else — without the engine ever installing them or knowing about them.

Mint the token¶

On the engine, as a superuser:

curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"name": "gpu-dev"}'

Shown once, valid for a year — a worker is a machine somebody sets up and leaves running. It is signed with the same keypair agent tokens use, so rotating that key revokes every worker along with them.

Send a node to it¶

A node declares the label of the machine it needs:

{
  "id": "train",
  "device": "gpu",
  "device_policy": "require",
  "timeout": 7200
}

prefer is what makes a flow work before the GPU box exists. require is what you want once it does.

What follows from this¶

• The node's source travels with every call. Nothing has to be deployed to the worker, and changing a node's code takes effect on the next execution.
• A node bound to a device is compiled on that machine. A node importing torch is correct on the GPU box and a missing module on the engine, so checking it here would fail something that is fine.
• import fluksio inside a node is the worker's own reporter. emit, save_artifact, load_artifact — installed before your code runs, so an installed fluksio package on that box never shadows it.
• Cancelling a run kills what it is executing, there or here, and leaves other runs of the same node alone.
• If the worker disappears mid-call, the run fails in seconds with worker went away mid-call rather than waiting out its timeout.
• A worker sends a heartbeat while it executes, so a long node is distinguishable from a dead socket. Ninety seconds of silence is gone.

Artifacts across machines¶

An artifact reference names content by its hash, not a location, so it stays valid wherever the store is reachable from. A worker that shares the engine's filesystem writes to it directly; one that does not fetches and uploads over HTTP, using the artifact endpoint beside the socket it already has. Either way your node code is the same two calls.

Seeing what is attached¶

curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq

Name, labels, how many calls it will take at once, how many are in flight, when it attached, when it was last seen, its Python version, and a digest of its environment.

What a worker is not¶

It is not a second engine. Subscriptions, schedules, webhooks, the dashboards and the run queue all stay in one process — that is what keeps a message having one definition and a cron tick happening once. A worker executes node bodies.

Running two engines against one data directory is not supported. Distribute work with workers.

See also¶

• Runs: pipelines that finish
• Writing node code
• Getting started: data science