fix: orchestrator reload 500 on storage timeout #183

Merged
meshflows merged 2 commits from fix/orchestrator-reload-storage-timeout into main 2026-09-02 11:41:43 +00:00
Owner

Summary

  • POST /admin/reload used a 10s urllib timeout and turned TimeoutError into HTTP 500, so Deploy Workflows failed with curl -f and no body.
  • Catch the timeout, retry, and return 503 with the reason so deploy can distinguish a transient storage hang from a hard error.
  • Log the reload HTTP status and body in deploy.yml, and retry the reload call instead of failing on the first timeout.

Test plan

  • Orchestrator reload during deploy no longer fails with an empty 500 when Storage is slow.
  • Timeout/503 path is covered by unit tests.
  • Deploy Workflows logs reload attempt, HTTP code, and body.
## Summary - POST `/admin/reload` used a 10s urllib timeout and turned `TimeoutError` into HTTP 500, so Deploy Workflows failed with `curl -f` and no body. - Catch the timeout, retry, and return 503 with the reason so deploy can distinguish a transient storage hang from a hard error. - Log the reload HTTP status and body in `deploy.yml`, and retry the reload call instead of failing on the first timeout. ## Test plan - [ ] Orchestrator reload during deploy no longer fails with an empty 500 when Storage is slow. - [ ] Timeout/503 path is covered by unit tests. - [ ] Deploy Workflows logs reload attempt, HTTP code, and body.
fix: orchestrator reload 500 on storage timeout
All checks were successful
Test & Validate / Validate Repository Structure (pull_request) Successful in 12s
Build & Push Docker Images / Validate JavaScript Syntax (pull_request) Successful in 18s
Test & Validate / Validate Workflow YAML (pull_request) Successful in 27s
Test & Validate / Validate Kubernetes Manifests (pull_request) Successful in 6s
Build & Push Docker Images / Publish Release Bundle (push) Successful in 2s
Test & Validate / Test Engine Smoke Suite (pytest) (pull_request) Successful in 58s
Build & Push Docker Images / Validate Repository Structure (pull_request) Successful in 6s
Build & Push Docker Images / Detect Changed Components (pull_request) Successful in 35s
Build & Push Docker Images / Build Forgejo Runner Image (pull_request) Has been skipped
Build & Push Docker Images / Sync compact pointer to Forgejo Wiki (pull_request) Has been skipped
Build & Push Docker Images / Validate Workflow YAML (pull_request) Successful in 14s
Build & Push Docker Images / Validate Kubernetes Manifests (pull_request) Successful in 5s
Build & Push Docker Images / Resolve Auto-Deploy Targets (push) Successful in 0s
Test & Validate / Test Summary (pull_request) Successful in 0s
Build & Push Docker Images / Test Engine Smoke Suite (pytest) (pull_request) Successful in 57s
Build & Push Docker Images / Build Wiki (MkDocs) (pull_request) Successful in 6s
Build & Push Docker Images / Trigger Deploy DEV (dev) (push) Successful in 1s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (push) Has been skipped
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (push) Has been skipped
Build & Push Docker Images / Build Summary (push) Successful in 1s
Build & Push Docker Images / Test Summary (pull_request) Successful in 2s
Build & Push Docker Images / Test Gate (pull_request) Successful in 0s
Build & Push Docker Images / Build (orchestrator) Engine Services [./engine/services/orchestrator] (orchestrator, ./engine/services/orchestrator) (pull_request) Successful in 29s
Build & Push Docker Images / Retag Product Version (pull_request) Successful in 2s
Build & Push Docker Images / Publish Release Bundle (pull_request) Successful in 2s
Build & Push Docker Images / Resolve Auto-Deploy Targets (pull_request) Successful in 1s
Build & Push Docker Images / Build Summary (pull_request) Successful in 0s
Build & Push Docker Images / Trigger Deploy DEV (dev) (pull_request) Successful in 2s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (pull_request) Has been skipped
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (pull_request) Has been skipped
de62380fcb
POST /admin/reload used a 10s urllib timeout and let TimeoutError become HTTP 500, so Deploy Workflows failed with curl -f and no body. Catch the timeout, retry, return 503 with the reason, and log the reload response in deploy.yml.

Co-authored-by: Cursor <cursoragent@cursor.com>
ci: show env and mode in Forgejo deploy run-name
All checks were successful
Build & Push Docker Images / Detect Changed Components (pull_request) Successful in 31s
Build & Push Docker Images / Build Forgejo Runner Image (pull_request) Has been skipped
Build & Push Docker Images / Sync compact pointer to Forgejo Wiki (pull_request) Has been skipped
Build & Push Docker Images / Validate Workflow YAML (pull_request) Successful in 16s
Build & Push Docker Images / Validate Kubernetes Manifests (pull_request) Successful in 10s
Build & Push Docker Images / Test Engine Smoke Suite (pytest) (pull_request) Successful in 52s
Test & Validate / Test Summary (push) Successful in 2s
Test & Validate / Test Summary (pull_request) Successful in 1s
Build & Push Docker Images / Build Wiki (MkDocs) (push) Successful in 10s
Build & Push Docker Images / Test Summary (push) Successful in 2s
Build & Push Docker Images / Test Gate (push) Successful in 0s
Build & Push Docker Images / Build Wiki (MkDocs) (pull_request) Successful in 11s
Build & Push Docker Images / Build (orchestrator) Engine Services [./engine/services/orchestrator] (orchestrator, ./engine/services/orchestrator) (push) Successful in 32s
Build & Push Docker Images / Test Summary (pull_request) Successful in 1s
Build & Push Docker Images / Test Gate (pull_request) Successful in 0s
Build & Push Docker Images / Retag Product Version (push) Successful in 2s
Build & Push Docker Images / Build (orchestrator) Engine Services [./engine/services/orchestrator] (orchestrator, ./engine/services/orchestrator) (pull_request) Successful in 32s
Build & Push Docker Images / Publish Release Bundle (push) Successful in 2s
Build & Push Docker Images / Retag Product Version (pull_request) Successful in 1s
Build & Push Docker Images / Resolve Auto-Deploy Targets (push) Successful in 2s
Build & Push Docker Images / Publish Release Bundle (pull_request) Successful in 2s
Build & Push Docker Images / Build Summary (push) Successful in 2s
Build & Push Docker Images / Trigger Deploy DEV (dev) (push) Successful in 6s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (push) Has been skipped
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (push) Has been skipped
Build & Push Docker Images / Resolve Auto-Deploy Targets (pull_request) Successful in 2s
Build & Push Docker Images / Build Summary (pull_request) Successful in 1s
Build & Push Docker Images / Trigger Deploy DEV (dev) (pull_request) Successful in 4s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (pull_request) Has been skipped
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (pull_request) Has been skipped
26dde2db97
Forgejo 16 run-name only allows forgejo/inputs/vars. Using github.* makes evaluation fail and the Actions list stays Deploy to Cluster.
Sign in to join this conversation.
No description provided.