Mesh Workflows
  • Python 65.8%
  • JavaScript 23.1%
  • HTML 4%
  • Shell 2.8%
  • CSS 2.1%
  • Other 2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Marc Fellman 9cfe7ebc72
All checks were successful
Build & Push Docker Images / Runner Preflight (push) Successful in 0s
Build & Push Docker Images / Build Forgejo Runner Image (push) Has been skipped
Dependency Security Audit / Check language version consistency (push) Successful in 3s
Dependency Security Audit / Check Go / C# vulnerabilities (push) Successful in 29s
Test & Validate / Validate Repository Structure (push) Successful in 3s
Test & Validate / Test Engine Smoke Suite (pytest) (push) Successful in 49s
Test & Validate / Validate Workflow YAML (push) Successful in 14s
Test & Validate / Validate Kubernetes Manifests (push) Successful in 5s
Build & Push Docker Images / Validate Repository Structure (push) Successful in 3s
Build & Push Docker Images / Test Engine Smoke Suite (pytest) (push) Successful in 50s
Build & Push Docker Images / Validate Workflow YAML (push) Successful in 14s
Build & Push Docker Images / Validate Kubernetes Manifests (push) Successful in 5s
Test & Validate / Test Summary (push) Successful in 0s
Dependency Security Audit / Check Dependencies (push) Successful in 4m29s
Build & Push Docker Images / Test Summary (push) Successful in 0s
Build & Push Docker Images / Test Gate (push) Successful in 0s
Build & Push Docker Images / Retag Product Version (push) Successful in 0s
Build & Push Docker Images / Detect Changed Components (push) Successful in 3s
Build & Push Docker Images / Build Wiki (MkDocs) (push) Successful in 7s
Build & Push Docker Images / Validate JavaScript Syntax (push) Successful in 4s
Build & Push Docker Images / Build (scheduler) Engine Services [./engine/services/scheduler] (scheduler, ./engine/services/scheduler) (push) Successful in 18s
Build & Push Docker Images / Build (egress-google-workspace) Engine Services [./engine/services/google-workspace] (egress-google-workspace, ./engine/services/google-workspace) (push) Successful in 14s
Build & Push Docker Images / Build (gateway) Engine Services [./engine/services/gateway] (gateway, ./engine/services/gateway) (push) Successful in 18s
Build & Push Docker Images / Build (orchestrator) Engine Services [./engine/services/orchestrator] (orchestrator, ./engine/services/orchestrator) (push) Successful in 25s
Build & Push Docker Images / Build (identity) Engine Services [./engine/services/identity] (identity, ./engine/services/identity) (push) Successful in 34s
Build & Push Docker Images / Sync compact pointer to Forgejo Wiki (push) Successful in 5s
Build & Push Docker Images / Publish Release Bundle (push) Successful in 1s
Build & Push Docker Images / Resolve Auto-Deploy Targets (push) Successful in 0s
Build & Push Docker Images / Build Summary (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy DEV (dev) (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (push) Successful in 0s
fix(security): bump PyJWT to 2.15.1 for PYSEC-2026-4141/4183
pip-audit still failed on PyJWT 2.14.0; fix versions require >=2.15.0.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-24 20:17:42 +02:00
.ci feat/edge-router-multi-env (#4) 2026-07-04 15:26:08 +00:00
.cursor chore: use images.lock.json as version provenance (#165) 2026-08-18 15:27:14 +00:00
.githooks feature/workflows-triggers-testing (#69) 2026-07-02 23:25:45 +02:00
.github feat: Operations context columns and multi-config sample dashboards (#178) 2026-09-17 16:06:55 +02:00
ArtWork Revert "Dashboard: Grouping/Editing for Policies/Schedules, and Orchestrator: Live artifact resolution for XSLT/Payloads" 2026-04-20 21:38:28 +02:00
engine fix(security): bump PyJWT to 2.15.1 for PYSEC-2026-4141/4183 2026-10-24 20:17:42 +02:00
flows feat: roll up chain milestones onto the Operations parent row (#178) 2026-09-18 16:05:48 +02:00
internal Add registry-driven service_call compose, IMAP first-class fields, and C#/Go echo demos. 2026-08-08 21:22:32 +02:00
legacy Add registry-driven service_call compose, IMAP first-class fields, and C#/Go echo demos. 2026-08-08 21:22:32 +02:00
scripts fix(security): bump PyJWT/OTel pins and govulncheck Go image 2026-10-24 15:31:33 +02:00
testing feature/workflows-triggers-testing (#69) 2026-07-02 23:25:45 +02:00
tools/echo-bench perf(orchestrator): defer NFS trace finish and trigger-event append 2026-08-12 17:39:10 +02:00
wiki Merge branch 'feat/issue-173-registry-lean' 2026-10-24 15:17:23 +02:00
.gitattributes feat/edge-router-multi-env (#4) 2026-07-04 15:26:08 +00:00
.gitignore Ignore local tmp probe and migrate scripts. 2026-08-13 08:46:13 +02:00
.gitmessage chore: standardize workflow schema and move flows runtime data to shared PV 2026-04-12 22:05:28 +02:00
CONTRIBUTING.md Stop mirroring wiki/docs into the Forgejo wiki. 2026-08-14 13:30:34 +02:00
DEPENDENCIES.md Scan Go and C# demo providers in the security-audit pipeline alongside pip-audit (#115). 2026-08-14 16:04:42 +02:00
deploy-minikube.ps1 feat: Azure Service Bus topics with peek-lock complete 2026-08-14 18:26:24 +02:00
DEPRECATIONS.md chore: standardize workflow schema and move flows runtime data to shared PV 2026-04-12 22:05:28 +02:00
LICENSE Add GPL-3.0 LICENSE 2026-04-12 14:12:51 +02:00
pyrightconfig.json refactor(workflow): migrate invocation fields to start contract 2026-06-20 14:37:37 +02:00
README.md chore/remove-versions-manifest (#166) 2026-08-18 15:39:17 +00:00
SAFE_DELETION_AND_NESTED_WORKFLOWS.md feat: nested workflow tracing, safe delete guard, step-types docs 2026-04-23 14:54:33 +02:00
VERSION chore: bump product version to 0.4.6 2026-10-24 15:25:32 +02:00

MeshFlows

MeshFlows logo

MeshFlows is an enterprise-grade workflow orchestration and integration platform built on Kubernetes. It consists of two main components:

  1. Engine (engine/) — Infrastructure & microservices layer

    • Orchestrator: YAML-driven workflow execution
    • Gateway: Public API entry point
    • Transformers: XSLT, XML↔JSON, Liquid templates
    • Storage: Runtime key-value store & document management
    • Identity: JWT authentication & authorization
    • Egress Services: HTTP, FTP, SSH, RabbitMQ integrations
    • Scheduler: CronJob-based workflow triggers
    • Dashboard: Web UI for workflow management
  2. Flows (flows/) — Standalone workflow deployment layer

    • Workflow definitions (YAML)
    • Connection configurations
    • Policies & access control
    • Scheduled triggers
    • Can be deployed independently or with the engine
  3. Wiki (wiki/) — Complete documentation (MkDocs + Material theme)

    • Backend getting-started & technical reference
    • Workflows getting-started & technical reference
    • Deployable as containerized service in Kubernetes

Quick Navigation

For Operators & Platform Engineers

For Workflow Developers

Repository Structure

meshflows/
├── engine/                  # Microservices & infrastructure
│   ├── services/            # All 8+ microservices
│   ├── deploy/k8s/          # Kubernetes manifests (Kustomize)
│   ├── config/              # Runtime configuration
│   ├── scripts/             # Deployment helper scripts
│   ├── tests/               # pytest test suite
│   └── docker-compose.yml   # Local development stack
│
├── flows/                   # Standalone workflow layer
│   ├── workflows/           # Workflow YAML definitions
│   ├── connections/         # Connection credential templates
│   ├── policies/            # Workflow execution policies
│   ├── schedules/           # Scheduled workflow triggers
│   ├── settings/            # Runtime configuration
│   ├── requests/            # HTTP test client files
│   └── deploy/k8s/          # Flows-only K8s manifests
│
├── wiki/                    # MkDocs documentation site
│   ├── mkdocs.yml           # MkDocs configuration
│   ├── docs/                # Markdown documentation
│   ├── Dockerfile           # Docker build for wiki
│   └── deploy/k8s/          # Wiki K8s manifests
│
└── README.md                # This file

System Architecture

graph TB
    subgraph Components["🏗️ MeshFlows Components"]
        direction TB
        
        subgraph Engine["🔧 Engine (Microservices)"]
            G["Gateway<br/>API Entry"]
            O["Orchestrator<br/>Workflow Engine"]
            T["Transformers<br/>Data Xform"]
            S["Storage<br/>Key-Value"]
            I["Identity<br/>Auth/JWT"]
            SCH["Scheduler<br/>CronJobs"]
            D["Dashboard<br/>Web UI"]
            E["Egress<br/>HTTP/FTP/SSH"]
            R["RabbitMQ<br/>Message Broker"]
        end
        
        subgraph Flows["📋 Flows Layer"]
            WF["Workflows<br/>YAML Definitions"]
            CONN["Connections<br/>Credentials"]
            POL["Policies<br/>Access Control"]
        end
        
        subgraph Obs["📊 Observability"]
            TEMPO["Tempo<br/>Tracing"]
            LOKI["Loki<br/>Logs"]
            PROM["Prometheus<br/>Metrics"]
        end
        
        WIKI["📚 Wiki<br/>Documentation"]
    end
    
    subgraph Infra["☸️ Infrastructure"]
        K8S["Kubernetes Cluster"]
        LH["Longhorn<br/>Storage"]
    end
    
    O -.->|executes| WF
    G -->|routes to| O
    O -->|calls| T
    O -->|stores in| S
    O -->|retrieves| CONN
    O -->|applies| POL
    G -->|auth via| I
    SCH -->|triggers| O
    E -->|integrates| O
    O -->|publishes to| R
    
    O -.->|traces to| TEMPO
    O -.->|logs to| LOKI
    O -.->|metrics to| PROM
    
    D -->|controls| O
    
    Engine -.->|deploys to| K8S
    K8S -->|uses| LH
    
    style Engine fill:#E3F2FD
    style Flows fill:#F3E5F5
    style Obs fill:#FFF3E0
    style Infra fill:#E8F5E9

Deployment Pipeline

For details on our automated CI/CD pipeline with smart build detection and testing, see CI/CD Pipeline Architecture

graph LR
    A["🔄 Push / PR"] --> B["🔍 Detect<br/>Changes"]
    B --> C["📦 Build<br/>Changed Only"]
    C --> D["✅ Test Gate"]
    D --> E{Pass?}
    
    E -->|No| F["❌ Blocked"]
    E -->|Yes| G["🚀 Deploy dev"]
    G --> H{main merge?}
    H -->|Yes| I["🚀 tst + lab<br/>parallel"]
    H -->|No| J["🟢 dev LIVE"]
    I --> K["🟢 Promoted"]
    
    style B fill:#FFE082
    style F fill:#EF5350
    style J fill:#66BB6A
    style K fill:#66BB6A

Local Pre-commit Validation

Workflow structure validation is enforced in CI and can be enforced locally before every commit.

git config core.hooksPath .githooks

The pre-commit hook runs:

python flows/scripts/validate_workflow_headers.py --changed-only

Optional: enable the commit message template for workflow changes:

git config commit.template .gitmessage

Deprecated interfaces and removal timelines are tracked in DEPRECATIONS.md.


Getting Started

Local Development (Docker Compose)

cd engine
docker-compose up -d

This brings up:

  • All 8+ microservices
  • RabbitMQ broker
  • Observability stack (Tempo, Loki, Prometheus, OTel Collector)
  • Named volumes for persistence

Kubernetes Deployment

cd engine/deploy
kubectl apply -k overlays/minikube/

For the shared Linux cluster (CI primary path):

cd engine/deploy
kubectl apply -k overlays/linux-cluster/

Production-like environments use Forgejo Actions — see Develop and Test and CI/CD Pipeline.

Wiki (Local)

cd wiki
pip install mkdocs mkdocs-material
mkdocs serve

Access at http://localhost:8000

Wiki (Kubernetes)

cd wiki
docker build -t meshflows-wiki:latest .
kubectl apply -f deploy/k8s/

Key Features

  • YAML-driven Workflows — Declarative workflow definitions with composable steps
  • Parallel Execution — Run multiple steps concurrently
  • Retry & Compensation — Built-in failure handling and transaction-like rollback
  • Transforms — XSLT 1.0, XML↔JSON, Liquid templates
  • Integrations — HTTP, FTP/FTPS, SSH/SFTP, RabbitMQ
  • Authentication — JWT, OAuth 2.0, scope-based access control
  • Observability — OpenTelemetry, Tempo tracing, Loki logs, Prometheus metrics
  • Storage — Role-based key-value store with ACLs
  • Scheduler — CronJob-based triggers or HTTP-driven invocation
  • Dashboard — Visual workflow editor, execution history, step-level inspection

Technology Stack

Layer Technology
Language Python 3 (FastAPI)
Container Docker
Orchestration Kubernetes (Kustomize)
Storage Longhorn (persistent volumes)
Message Broker RabbitMQ 3.13
Tracing Tempo
Metrics Prometheus
Logging Loki + Promtail
Observability OpenTelemetry
Testing pytest
CI/CD Forgejo Actions (.github/workflows)

Documentation

Published MkDocs site: dev-wiki.meshflows.org (source: wiki/docs/). The Forgejo repository wiki is a compact pointer to that site, not a second docs tree.


Requirements

  • Kubernetes 1.20+
  • Longhorn for persistent storage
  • Docker (for local development)
  • Python 3.11+ (for local testing)

Contributing

See CONTRIBUTING.md for hooks, branch naming, and CI expectations.

Learn more in the Flows Deployment Guide.


License

See LICENSE file.


Support & Issues

  • Documentation: See wiki/docs/
  • Bug reports & features: Use project issue tracker
  • Questions: Check our wiki documentation first