fix(nodes): never schedule a model onto a node that cannot store it

[localai-bot]

The incident

An NVIDIA Jetson Thor worker (nvidia-thor) filled its disk completely:

/dev/nvme0n1p1  937G  937G  0  100% /

It continued to report status: healthy via the nodes API, remained a scheduling candidate, was selected to host a 70 GB video model, accepted the staging request, transferred ~17 GB, and only then failed:

staging .../whisper-large-v3/model.fp32-00001-of-00002.safetensors:
  upload to node b7bacbf4-... failed with status 500:
  writing file: /models/longcat-video-avatar-1.5/...: no space left on device

Total elapsed before the failure surfaced: 16 minutes. The node had zero bytes free when it accepted the work — it could never have held the model. The error message itself is good (it names the file and the cause, from an earlier fix). The defect is that the request was accepted at all.

What the health signal actually covered

• /readyz (WorkerReadiness + NATSReadiness) checks the NATS link — "can this worker receive work", added for Docker HEALTHCHECK probes a frontend endpoint, so every worker container is permanently unhealthy #10987.
• scripts/build/healthcheck.sh, the container HEALTHCHECK, checks liveness.
• status: healthy in the node registry is driven purely by heartbeat recency (LastHeartbeat, MarkOffline/FindStaleNodes).

None of these say anything about capacity to accept work.

Node capacity carried total_vram, available_vram, total_ram, available_ram — no disk figure existed at all, unused or otherwise. SmartRouter.scheduleNewModel compared an estimated VRAM figure against available_vram - reserved_vram and never looked at free space on the filesystem staging actually writes to.

The fix

Report it. Workers now measure the filesystem backing their models directory (--models-path) — not /, since staged weights land in the models path and that mount is very often separate — and report total_disk / available_disk on registration and on every heartbeat. Free disk moves much faster than VRAM under staging traffic (each accepted model permanently consumes space), so a registration-time-only reading would have missed a node that filled up hours later. disk.Usage().Free rather than Total - Used: the ext4 reserved-blocks pool sits between them, and writing into it is exactly what fails with ENOSPC for a non-root process.

Use it. scheduleNewModel now drops nodes that cannot store the model before picking one, ahead of the replica-slot and VRAM passes. The requirement comes from modelPayloadBytes — the same local paths stageModelFiles uploads, already computed for the size-derived load budget — plus a modest margin (5%, at least 1 GiB).

Fail clearly. When no node fits, scheduling fails immediately with the numbers behind the decision rather than picking one and discovering it 16 minutes into a transfer:

scheduling longcat-video-avatar-1.5: no node has enough free disk for the model:
need 73.5 GB free on the models filesystem, but nvidia-thor has 0 B free of 937.0 GB

The operator knob

The check is on by default, with one switch exposed two ways:

Surface	Name
CLI / env	--distributed-disk-headroom-check / LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK (default true)
Runtime setting	distributed_disk_headroom_check — POST /api/settings, and Settings → Distributed in the WebUI

Both write the same DistributedConfig.DiskHeadroomDisabled member, and the SmartRouter reads that member live on every scheduling decision via a closure over the application config rather than a value snapshotted at construction. Env/CLI sets the boot value, the runtime setting overrides it live, last write wins, and there is exactly one thing to read — no second source of truth to drift. Snapshotting at construction would have made the runtime toggle a no-op until restart, which is the failure mode worth guarding against here. The registration follows the declarative registry added in #10864 (core/config/runtime_settings_registry.go); its completeness spec fails until both the struct field and the registry row exist, and I confirmed it does.

Disabled means warn, not skip

This was the judgement call. Disabling restores selection that ignores free disk — byte for byte the pre-check behaviour — but the check still runs, and when it would have rejected every node it says so, naming the knob that suppressed it:

WARN No node has room to store this model, but the disk-headroom check is DISABLED;
     scheduling anyway — staging will most likely fail with ENOSPC
     model=longcat-video-avatar-1.5 knob=distributed-disk-headroom-check

Skipping outright was rejected because it reproduces the condition that made the original bug expensive: the cluster was doing something that could not work and said nothing about it. The escape hatch exists for setups where our size estimate is wrong (deduplicating or compressing filesystems, a backend that fetches its own weights instead of using the staged copy) — and in exactly those cases the operator needs to see what LocalAI thought was wrong. Warning only on the total-rejection case keeps it actionable rather than chatty on a heterogeneous cluster where some nodes are legitimately too small. Disabling is also logged once at startup, so a silently-disabled safety check cannot hide.

Design choices

Model size, not a fixed percentage. A percentage threshold would take a small-but-usable node out of rotation for models it could comfortably store, and on a homogeneous cluster would strand every node at once. Comparing against the actual payload means a node too small for one model stays a valid target for smaller ones. When the size cannot be determined locally (a bare HF repo id the worker fetches itself), the node only has to clear a 2 GiB floor.

Selection-side, not health-side. Low disk deliberately does not flip a node to unhealthy. The check is per model; a global health threshold is exactly the hard cutoff that could strand a tight cluster.

total_disk == 0 means unknown, not full. A pre-upgrade worker (or one whose stat failed) reports total_disk == 0 and passes through untouched, so a rolling upgrade never empties the candidate pool. A genuinely full node is distinguishable: non-zero total, zero available. Using available_disk == 0 as the unknown sentinel would have reinstated the exact bug — that is what a 100%-full node reports.

Shared-models mode skips the check entirely. LOCALAI_DISTRIBUTED_SHARED_MODELS makes stageModelFiles upload nothing: every node already mounts this models directory at this path. Demanding the full checkpoint size of free space per node would have rejected a cluster that needs no new bytes at all. This was a false positive in the first draft of the check, caught while wiring the knob.

Never wedge on a read failure. A registry error from NarrowByDiskHeadroom is logged and scheduling continues unfiltered rather than failing. Standalone/single-node operation is untouched: none of this runs outside the distributed SmartRouter path.

Scope

Registration and heartbeat are plain JSON over HTTP, so this needed no protobuf change and no ripple through the gRPC surface. The additions are purely additive fields; existing React UI, e2e, and MCP consumers read node fields additively and are unaffected. list_nodes in pkg/mcp/localaitools does not project VRAM either, so disk was left out of it; the runtime-settings registry drives REST + UI only (MCP exposes settings solely for branding), so no MCP tool was added.

Testing

TDD, Ginkgo/Gomega, red first and failing on behaviour rather than missing symbols — both for the original check and for the knob.

Original check:

[FAIL] nodesWithDiskHeadroom [It] drops a node whose models filesystem is full
[FAIL] NarrowByDiskHeadroom [It] removes a full node from the candidate set
[FAIL] NarrowByDiskHeadroom [It] fails with a capacity error naming the shortfall...
[FAIL] scheduling... [It] fails at scheduling time instead of installing a backend it cannot feed
       Expected an error, got nil

Knob (4 router specs red before the wiring, all "Expected success, but got an error"):

[FAIL] ... [It] restores the pre-check behaviour when the operator disables the knob
[FAIL] ... [It] still evaluates the check when disabled, so the operator is not left blind
[FAIL] ... [It] reads the toggle live, so a runtime change applies without a restart
[FAIL] ... [It] skips the check in shared-models mode, where nothing is staged to the worker

Config-level, verified red by removing the registry row:

[FAIL] distributed disk-headroom check setting [It] lets a runtime override beat the env value, in both directions
[FAIL] runtime settings registry [It] owns every RuntimeSettings field exactly once
       RuntimeSettings field "distributed_disk_headroom_check" must be owned by exactly one registry row

The router specs assert the incident shape directly: Route returns ErrInsufficientDisk and unloader.installCalls is empty — no backend install, therefore no staging, therefore no 17 GB transferred before the truth surfaced. The live-toggle spec flips the value after the router is constructed, so a snapshotted read would fail it.

All green after the fix:

• go build ./core/... ./pkg/... → exit 0
• go test ./core/services/nodes/ → exit 0 (260s, full suite)
• go test ./core/config/ → exit 0
• go test ./core/cli/... ./core/application/... ./pkg/xsysinfo/ ./core/services/worker/ → exit 0
• make lint → exit 0, 0 issues

Docs updated in docs/content/features/distributed-mode.md: Disk headroom section with the new Turning the check off subsection, the frontend configuration table, and the scheduling-algorithm step list. Free space is surfaced on the node detail page next to VRAM, since the incident's signature was a node that looked entirely healthy.

Pre-existing flake, not from this branch: Backend traces payload bounding in core/http/endpoints/localai fails under load ("Backend trace channel full, dropping trace"). It fails 5/5 on clean origin/master (ff299df) as well as on this branch, so it comes from #11056, not from here. Left alone.

Not verified: no access to the live cluster, so the end-to-end behaviour against a genuinely full nvidia-thor is reasoned from the code paths and covered by unit/integration specs rather than observed. The React UI was parse-checked with esbuild but not built or rendered (no node_modules in the worktree).

🤖 Generated with Claude Code

https://claude.ai/code/session_0156CbKpDh8qbAYzJZJDZ2yk