fix(galleryop): make admitted operations queryable and survive a failed op

[localai-bot]

Observed on a live 2-replica distributed cluster

Sequence A — the install endpoint reported success for work nothing could observe

1. POST /models/apply {"id":"localai@longcat-video-avatar-1.5"} → job 0bae4174, ran to ~19%, then failed with a transient read error (materialize primary model artifact: failed to read "https://huggingface.co/.../diffusion_pytorch_model-00003-of-00006.safetensors").
2. Three subsequent POST /models/apply calls each returned HTTP 200 with a fresh UUID (a2893eed, cd884437, f8876d41).
3. GET /models/jobs/<uuid> for each returned HTTP 500 could not find any status for ID.
4. GET /models/jobs listed only 0bae4174. The three later UUIDs were never in the map.

Earlier in the same session one replica also stopped answering POST /models/apply entirely (connection closed, no response) while GET /readyz kept returning 200.

Sequence B — an op orphaned by a controller restart claimed "downloading" forever

A controller replaced mid-download left the op reporting phase=downloading, processed=false, error=none, progress=13.8% while nothing was downloading (du -sb flat for 25+ minutes). Every retry for that model was rejected with already being processed by another instance (op <id>) until the op was manually cancelled. This happened three times.

Mechanism

The admission handlers mint the job UUID, hand the op to an unbuffered channel, and answer 200 immediately. The worker (GalleryService.Start) is a single goroutine consuming both channels serially, and the first status write happens inside modelHandler/backendHandler — only once the worker actually starts the work.

So an op queued behind a running install has no status by construction, which is symptoms 3 and 4 exactly. On the paths that sent directly rather than from a goroutine, the same unbuffered channel blocked the HTTP handler for the entire duration of the in-flight install — that is the replica that accepted no /models/apply while /readyz stayed green.

The worker also had no panic containment: a panic in any handler propagated out of the single consumer goroutine and killed the process, taking every queued op with it.

For B, the PostgreSQL side does recover on its own — FindDuplicate ignores rows untouched for 30 minutes and CleanStale marks them failed, on a 15-minute tick. But the reaper only ever corrected the database. The in-memory statuses map that GET /models/jobs/<id> and /api/operations actually read was never corrected, so every replica kept serving the frozen tick indefinitely, on both the originating replica and any peer that received it over NATS.

Changes

• EnqueueModelOp / EnqueueBackendOp publish a queued status before handing the op over, so a job ID is queryable from the instant it is handed out. All HTTP admission paths now go through them.
• Delivery selects on the op's context, so cancelling a still-queued op releases the delivery goroutine instead of stranding it on a send that will never be received. An op the worker never accepts becomes a terminal failure rather than a silent leak.
• The worker contains a panicking handler to the op that caused it instead of dying.
• ReapStaleOperations reconciles the in-memory status with the reap (GalleryStore.ListStale).
• The two ignored galleryStore.Create errors are logged.
• The model and backend delete endpoints now run under the same ID they hand back — they previously ran under an empty ID and returned a status URL for a job that could never have a status.

Not addressed

Operation ownership is still not tracked. gallery_operations has a FrontendID column that nothing writes, so a live op and one whose owner died are distinguished only by the 30-minute staleness timeout. Narrowing that window needs a lease/heartbeat mechanism — out of scope here, noted for follow-up.

Testing

New Ginkgo specs in core/services/galleryop/enqueue_test.go, each written failing on behaviour first:

• a model/backend op admitted while the worker cannot accept it is queryable straight away (previously nil status)
• an op abandoned before the worker accepts it becomes a terminal failure (previously stayed queued forever)
• an op is still delivered to a draining worker
• ReapStaleOperations marks the reaped op failed in memory, not just in the store
• the worker keeps consuming after a handler panics — without the fix this crashes the whole test binary with panic: boom, which is the production failure mode

go build ./core/... ./pkg/... exit 0. go test ./core/services/galleryop/ ./core/services/distributed/ ./core/http/endpoints/localai/ ./core/http/routes/ exit 0. Scoped golangci-lint on the touched packages: 0 issues. Full make lint fails on a pre-existing core/http/endpoints/ollama/helpers_internal_test.go typecheck error that reproduces on a pristine origin/master checkout and is unrelated to this change.

The transient network failure that triggered the original sequence is not reproducible locally, so the fix is verified against the mechanism established in code rather than against the live cluster.

🤖 Generated with Claude Code

https://claude.ai/code/session_0156CbKpDh8qbAYzJZJDZ2yk