Runs: pipelines that finish¶
A live flow reacts to events. A batch flow starts with parameters, runs its graph once and returns a result. Each execution is a run, with its own state, parameters, node records, metrics and artifacts.
Use runs for experiments, reports and other jobs with a finish line. For a complete first example, start with the data science guide.
A batch flow¶
In Python, Flow(...) defaults to mode="batch". Declare inputs for
run parameters and outputs for the final result. On the canvas, choose batch
mode and define the same inputs and outputs.
A batch flow does not need activation and does not open subscriptions, schedules or webhooks. A run executes each node once after its dependencies finish. Missing parameters use their declared initial values.
Non-streaming ports cannot use interval in a batch flow: there is no timer
to deliver a delayed value. Streaming ports can use it to reduce display
updates while keeping every metric point.
Submitting¶
In Runs → New run, choose a flow, enter its inputs and press Run. Use draft runs the working copy without publishing it.
From a terminal:
Or submit through the API:
curl -X POST "$FLUKSIO/runs/flows/train" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.1, "epochs": 20}, "seed": 7}'
Here $FLUKSIO is the API base URL ending in /api/v1, and $TOKEN
is an access token. See The HTTP API for authentication.
Submission returns immediately with status queued. Read
GET /api/v1/runs/{id} for status and results. Unknown parameters or wrong
types return 422 before execution.
The API body also accepts "draft": true, "no_cache": true and an
idempotency_key. Reusing an idempotency key returns the original submission
instead of creating another run. Drafts are read when execution starts, so
edits made while a run is queued can affect it.
A seed is recorded on the run and fills a declared input named seed.
Your node code must use that input to seed its random number generator.
Sweeps¶
A sweep submits every combination of input values as one group:
These four runs share a group_id and execute concurrently within the
engine's limits. Each has its own state, so one run cannot overwrite another's
messages. Vary seeds with --param seed=1,2,3 if the flow declares that input.
The API equivalent is POST /api/v1/runs/flows/{name}/sweep with a
runs list of parameter dictionaries and optional seeds. Filter history
with GET /api/v1/runs?group=<group-id>.
Producing values before you are finished¶
Training curves and progress values leave through output ports. A Python
generator publishes them with yield; callbacks can use fluksio.emit.
Declare the port with stream: true. See
Writing node code for examples.
Every numeric emission is stored under its flow-qualified message name,
such as train.loss, with a step counter and timestamp. Retrieve it with:
GET /api/v1/runs/{id}/metrics?name=train.loss
GET /api/v1/runs/series/compare?ids=<a>,<b>&metric=train.loss
A streaming port's interval reduces updates sent to the canvas, while the
run history keeps every point. In a run, intermediate emissions do not start
downstream nodes; those wait for the node's final result.
A node's timeout measures silence. Each yield or emit resets it, so a node
can run for hours while reporting progress. No timeout is imposed by default.
Artifacts¶
Use fluksio.save_artifact for checkpoints, datasets, plots and other bytes.
Return its reference through an artifact port. Another node calls
fluksio.load_artifact to obtain a local path.
Identical bytes share one stored copy. Returned artifacts are listed on the run and can be downloaded with:
Use the exact message name shown by the list command. Emitted media frames are transient; return a frame you need to keep as a run artifact.
Stage caching¶
Before executing a Python node, the engine checks for a successful previous
execution with matching source, settings, port declarations and input values.
On a hit, it restores the outputs and artifacts and records the node as
cached.
For code-defined flows, sync also records digests of the project modules
reachable from each node. Editing those modules invalidates the corresponding
stage. Static import tracking does not follow computed import names or track
installed dependency versions; use --no-cache after changes it cannot see.
Artifact inputs are compared by content digest. Secret settings contribute their references rather than secret values. External state is not part of a cache key, so disable caching on a node that reads changing data or performs side effects:
@node(provides=Port("reading", "float"), cache=False)
def read_sensor():
return get_current_reading()
Use fluksio run ... --no-cache, fluksio sweep ... --no-cache, or
"no_cache": true in an API submission to execute every node.
Only Python nodes are cached. Built-in and connector nodes are not. The cache uses run history; outputs too large to store there are not cacheable, and missing artifact bytes make an entry a miss. An entry written before the cache key last changed shape can never match again, so the engine drops it from the cache and keeps the node's recorded outputs.
A cache hit does not replay emissions. Its metrics are read from the original run, so comparisons still show the recorded curve. Deleting that original run leaves cached results available but removes their curve. The run's page then names the deleted run each lost curve went with.
Objects that cannot be serialized¶
Live models, data loaders and compiled functions cannot cross a JSON message boundary. Keep stages that share live objects in one node, or save a checkpoint artifact and rebuild the object in the next node.
Running a node somewhere else¶
Attach a remote worker to the engine and declare the
worker's label as the node's worker. worker_policy="require" waits for
that label; "prefer" uses the local engine when none is attached.
The selected machine needs your imported packages. Code-defined flows also need your project installed or available there: generated import shims travel with the call, but your package does not. Workers connect outward to the engine; see the worker guide for tokens and resource requests.
Durability¶
Queued runs are stored in the database and picked up when the engine restarts.
Running jobs refresh a lease. If the engine dies and the lease expires, those
runs become abandoned rather than being executed again automatically.
Retry a finished run or the unfinished results of a sweep:
A retry creates a new run with the previous inputs, seed and group, linked to its parent. It uses the flow's current definition, and can reuse completed cached stages. Cancel active runs before retrying them.
Looking at what ran¶
The Runs screen filters by flow, status or sweep. Each run shows its parameters, per-node timings and logs, traceback, metrics, artifacts and result. Its inputs are listed as the flow declared them when the run was submitted, and the ones it took from the flow are marked as defaults.
Select two or more runs to compare curves and the parameters that differ. Plot against step, elapsed seconds or another metric from the same runs. Comparisons have shareable URLs.
Deleting a flow keeps its run history, and the artifacts those runs recorded stay on disk. You can still inspect those runs, but cannot retry them until the flow exists again. Tick Also delete its runs in the delete dialog to clear them with the flow, or delete history separately after active runs finish or are cancelled.
Taking it into a dataframe¶
Export metrics as a long table and results as a row per run:
fluksio export metrics --flow train -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
Both include run IDs. Result rows also include input values, final numeric
outputs, status, duration and code identifiers. Nested values use dotted paths,
such as metric.final_metrics.train_loss.
Python returns the same rows directly:
from fluksio.sdk.client import Client
client = Client()
curves = client.export_metrics(flow="train")
results = client.export_runs(flow="train", status="ok")
Pass either list to pandas.DataFrame. The
CLI reference covers selection filters,
JSON Lines and Parquet.
Bringing in runs made elsewhere¶
Import historical runs without executing them:
from fluksio.sdk.client import Client
Client().import_runs([{
"flow": "train",
"external_id": "previous-study:run-42",
"params": {"lr": 0.05, "epochs": 10},
"status": "ok",
"created_at": "2026-03-01T10:00:00Z",
"result": {"score": 0.94},
}])
The flow must exist, and parameters are checked against its inputs. Repeated
imports of the same external_id reuse the existing record. Imported runs
have no code commit or digest because the engine did not execute them.
Entries may also carry metric points and base64-encoded artifacts; see the
API schema for their fields.
A dashboard, read against runs¶
Choose Open in dashboard from a run or comparison to view those results with a dashboard's widgets. Charts draw one line per run, scalar widgets use the first run, and controls are disabled.
If the flow has no dashboard, the same menu can create one from its streaming
ports and outputs. The API equivalent is
POST /api/v1/dashboards/from-flow/{name}. The generated dashboard is
published and can be edited like any other dashboard.
To keep a chart showing recent experiments, set its source to Runs and select the latest runs of a flow, a sweep or specific run IDs.