- Python 65.8%
- JavaScript 23.1%
- HTML 4%
- Shell 2.8%
- CSS 2.1%
- Other 2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
All checks were successful
Build & Push Docker Images / Runner Preflight (push) Successful in 0s
Build & Push Docker Images / Build Forgejo Runner Image (push) Has been skipped
Dependency Security Audit / Check language version consistency (push) Successful in 3s
Dependency Security Audit / Check Go / C# vulnerabilities (push) Successful in 29s
Test & Validate / Validate Repository Structure (push) Successful in 3s
Test & Validate / Test Engine Smoke Suite (pytest) (push) Successful in 49s
Test & Validate / Validate Workflow YAML (push) Successful in 14s
Test & Validate / Validate Kubernetes Manifests (push) Successful in 5s
Build & Push Docker Images / Validate Repository Structure (push) Successful in 3s
Build & Push Docker Images / Test Engine Smoke Suite (pytest) (push) Successful in 50s
Build & Push Docker Images / Validate Workflow YAML (push) Successful in 14s
Build & Push Docker Images / Validate Kubernetes Manifests (push) Successful in 5s
Test & Validate / Test Summary (push) Successful in 0s
Dependency Security Audit / Check Dependencies (push) Successful in 4m29s
Build & Push Docker Images / Test Summary (push) Successful in 0s
Build & Push Docker Images / Test Gate (push) Successful in 0s
Build & Push Docker Images / Retag Product Version (push) Successful in 0s
Build & Push Docker Images / Detect Changed Components (push) Successful in 3s
Build & Push Docker Images / Build Wiki (MkDocs) (push) Successful in 7s
Build & Push Docker Images / Validate JavaScript Syntax (push) Successful in 4s
Build & Push Docker Images / Build (scheduler) Engine Services [./engine/services/scheduler] (scheduler, ./engine/services/scheduler) (push) Successful in 18s
Build & Push Docker Images / Build (egress-google-workspace) Engine Services [./engine/services/google-workspace] (egress-google-workspace, ./engine/services/google-workspace) (push) Successful in 14s
Build & Push Docker Images / Build (gateway) Engine Services [./engine/services/gateway] (gateway, ./engine/services/gateway) (push) Successful in 18s
Build & Push Docker Images / Build (orchestrator) Engine Services [./engine/services/orchestrator] (orchestrator, ./engine/services/orchestrator) (push) Successful in 25s
Build & Push Docker Images / Build (identity) Engine Services [./engine/services/identity] (identity, ./engine/services/identity) (push) Successful in 34s
Build & Push Docker Images / Sync compact pointer to Forgejo Wiki (push) Successful in 5s
Build & Push Docker Images / Publish Release Bundle (push) Successful in 1s
Build & Push Docker Images / Resolve Auto-Deploy Targets (push) Successful in 0s
Build & Push Docker Images / Build Summary (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy DEV (dev) (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy TST/LAB (lab) (push) Successful in 0s
Build & Push Docker Images / Trigger Deploy TST/LAB (tst) (push) Successful in 0s
pip-audit still failed on PyJWT 2.14.0; fix versions require >=2.15.0. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
| .ci | ||
| .cursor | ||
| .githooks | ||
| .github | ||
| ArtWork | ||
| engine | ||
| flows | ||
| internal | ||
| legacy | ||
| scripts | ||
| testing | ||
| tools/echo-bench | ||
| wiki | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmessage | ||
| CONTRIBUTING.md | ||
| DEPENDENCIES.md | ||
| deploy-minikube.ps1 | ||
| DEPRECATIONS.md | ||
| LICENSE | ||
| pyrightconfig.json | ||
| README.md | ||
| SAFE_DELETION_AND_NESTED_WORKFLOWS.md | ||
| VERSION | ||
MeshFlows
MeshFlows is an enterprise-grade workflow orchestration and integration platform built on Kubernetes. It consists of two main components:
-
Engine (
engine/) — Infrastructure & microservices layer- Orchestrator: YAML-driven workflow execution
- Gateway: Public API entry point
- Transformers: XSLT, XML↔JSON, Liquid templates
- Storage: Runtime key-value store & document management
- Identity: JWT authentication & authorization
- Egress Services: HTTP, FTP, SSH, RabbitMQ integrations
- Scheduler: CronJob-based workflow triggers
- Dashboard: Web UI for workflow management
-
Flows (
flows/) — Standalone workflow deployment layer- Workflow definitions (YAML)
- Connection configurations
- Policies & access control
- Scheduled triggers
- Can be deployed independently or with the engine
-
Wiki (
wiki/) — Complete documentation (MkDocs + Material theme)- Backend getting-started & technical reference
- Workflows getting-started & technical reference
- Deployable as containerized service in Kubernetes
Quick Navigation
For Operators & Platform Engineers
- Engine Documentation — Architecture, deployment, operations
- Engine Howto — Prerequisites, Kubernetes setup, Longhorn, secrets
- Engine Reference — Services, APIs, authentication, security
For Workflow Developers
- Flows Documentation — Getting started with workflows
- Flows Howto — Deploying workflows, connections, policies
- Flows Reference — Workflow YAML syntax, step types, transformations
Repository Structure
meshflows/
├── engine/ # Microservices & infrastructure
│ ├── services/ # All 8+ microservices
│ ├── deploy/k8s/ # Kubernetes manifests (Kustomize)
│ ├── config/ # Runtime configuration
│ ├── scripts/ # Deployment helper scripts
│ ├── tests/ # pytest test suite
│ └── docker-compose.yml # Local development stack
│
├── flows/ # Standalone workflow layer
│ ├── workflows/ # Workflow YAML definitions
│ ├── connections/ # Connection credential templates
│ ├── policies/ # Workflow execution policies
│ ├── schedules/ # Scheduled workflow triggers
│ ├── settings/ # Runtime configuration
│ ├── requests/ # HTTP test client files
│ └── deploy/k8s/ # Flows-only K8s manifests
│
├── wiki/ # MkDocs documentation site
│ ├── mkdocs.yml # MkDocs configuration
│ ├── docs/ # Markdown documentation
│ ├── Dockerfile # Docker build for wiki
│ └── deploy/k8s/ # Wiki K8s manifests
│
└── README.md # This file
System Architecture
graph TB
subgraph Components["🏗️ MeshFlows Components"]
direction TB
subgraph Engine["🔧 Engine (Microservices)"]
G["Gateway<br/>API Entry"]
O["Orchestrator<br/>Workflow Engine"]
T["Transformers<br/>Data Xform"]
S["Storage<br/>Key-Value"]
I["Identity<br/>Auth/JWT"]
SCH["Scheduler<br/>CronJobs"]
D["Dashboard<br/>Web UI"]
E["Egress<br/>HTTP/FTP/SSH"]
R["RabbitMQ<br/>Message Broker"]
end
subgraph Flows["📋 Flows Layer"]
WF["Workflows<br/>YAML Definitions"]
CONN["Connections<br/>Credentials"]
POL["Policies<br/>Access Control"]
end
subgraph Obs["📊 Observability"]
TEMPO["Tempo<br/>Tracing"]
LOKI["Loki<br/>Logs"]
PROM["Prometheus<br/>Metrics"]
end
WIKI["📚 Wiki<br/>Documentation"]
end
subgraph Infra["☸️ Infrastructure"]
K8S["Kubernetes Cluster"]
LH["Longhorn<br/>Storage"]
end
O -.->|executes| WF
G -->|routes to| O
O -->|calls| T
O -->|stores in| S
O -->|retrieves| CONN
O -->|applies| POL
G -->|auth via| I
SCH -->|triggers| O
E -->|integrates| O
O -->|publishes to| R
O -.->|traces to| TEMPO
O -.->|logs to| LOKI
O -.->|metrics to| PROM
D -->|controls| O
Engine -.->|deploys to| K8S
K8S -->|uses| LH
style Engine fill:#E3F2FD
style Flows fill:#F3E5F5
style Obs fill:#FFF3E0
style Infra fill:#E8F5E9
Deployment Pipeline
For details on our automated CI/CD pipeline with smart build detection and testing, see CI/CD Pipeline Architecture
graph LR
A["🔄 Push / PR"] --> B["🔍 Detect<br/>Changes"]
B --> C["📦 Build<br/>Changed Only"]
C --> D["✅ Test Gate"]
D --> E{Pass?}
E -->|No| F["❌ Blocked"]
E -->|Yes| G["🚀 Deploy dev"]
G --> H{main merge?}
H -->|Yes| I["🚀 tst + lab<br/>parallel"]
H -->|No| J["🟢 dev LIVE"]
I --> K["🟢 Promoted"]
style B fill:#FFE082
style F fill:#EF5350
style J fill:#66BB6A
style K fill:#66BB6A
Local Pre-commit Validation
Workflow structure validation is enforced in CI and can be enforced locally before every commit.
git config core.hooksPath .githooks
The pre-commit hook runs:
python flows/scripts/validate_workflow_headers.py --changed-only
Optional: enable the commit message template for workflow changes:
git config commit.template .gitmessage
Deprecated interfaces and removal timelines are tracked in DEPRECATIONS.md.
Getting Started
Local Development (Docker Compose)
cd engine
docker-compose up -d
This brings up:
- All 8+ microservices
- RabbitMQ broker
- Observability stack (Tempo, Loki, Prometheus, OTel Collector)
- Named volumes for persistence
Kubernetes Deployment
cd engine/deploy
kubectl apply -k overlays/minikube/
For the shared Linux cluster (CI primary path):
cd engine/deploy
kubectl apply -k overlays/linux-cluster/
Production-like environments use Forgejo Actions — see Develop and Test and CI/CD Pipeline.
Wiki (Local)
cd wiki
pip install mkdocs mkdocs-material
mkdocs serve
Access at http://localhost:8000
Wiki (Kubernetes)
cd wiki
docker build -t meshflows-wiki:latest .
kubectl apply -f deploy/k8s/
Key Features
- YAML-driven Workflows — Declarative workflow definitions with composable steps
- Parallel Execution — Run multiple steps concurrently
- Retry & Compensation — Built-in failure handling and transaction-like rollback
- Transforms — XSLT 1.0, XML↔JSON, Liquid templates
- Integrations — HTTP, FTP/FTPS, SSH/SFTP, RabbitMQ
- Authentication — JWT, OAuth 2.0, scope-based access control
- Observability — OpenTelemetry, Tempo tracing, Loki logs, Prometheus metrics
- Storage — Role-based key-value store with ACLs
- Scheduler — CronJob-based triggers or HTTP-driven invocation
- Dashboard — Visual workflow editor, execution history, step-level inspection
Technology Stack
| Layer | Technology |
|---|---|
| Language | Python 3 (FastAPI) |
| Container | Docker |
| Orchestration | Kubernetes (Kustomize) |
| Storage | Longhorn (persistent volumes) |
| Message Broker | RabbitMQ 3.13 |
| Tracing | Tempo |
| Metrics | Prometheus |
| Logging | Loki + Promtail |
| Observability | OpenTelemetry |
| Testing | pytest |
| CI/CD | Forgejo Actions (.github/workflows) |
Documentation
Published MkDocs site: dev-wiki.meshflows.org (source: wiki/docs/). The Forgejo repository wiki is a compact pointer to that site, not a second docs tree.
- Engine How-to Guide — Step-by-step deployment and operations
- Engine Reference — Architecture, services, API documentation
- Workflows How-to Guide — Deploying and managing workflows
- Workflows Reference — Workflow YAML syntax, step types, examples
- Security & Hardening — Enterprise security checklist
- Develop and Test — End-to-end developer workflow
Requirements
- Kubernetes 1.20+
- Longhorn for persistent storage
- Docker (for local development)
- Python 3.11+ (for local testing)
Contributing
See CONTRIBUTING.md for hooks, branch naming, and CI expectations.
Learn more in the Flows Deployment Guide.
License
See LICENSE file.
Support & Issues
- Documentation: See
wiki/docs/ - Bug reports & features: Use project issue tracker
- Questions: Check our wiki documentation first