MeshFlows

MeshFlows is an enterprise-grade workflow orchestration and integration platform built on Kubernetes. It consists of two main components:

1. Engine (engine/) — Infrastructure & microservices layer

   • Orchestrator: YAML-driven workflow execution
   • Gateway: Public API entry point
   • Transformers: XSLT, XML↔JSON, Liquid templates
   • Storage: Runtime key-value store & document management
   • Identity: JWT authentication & authorization
   • Egress Services: HTTP, FTP, SSH, RabbitMQ integrations
   • Scheduler: CronJob-based workflow triggers
   • Dashboard: Web UI for workflow management

2. Flows (flows/) — Standalone workflow deployment layer

   • Workflow definitions (YAML)
   • Connection configurations
   • Policies & access control
   • Scheduled triggers
   • Can be deployed independently or with the engine

3. Wiki (wiki/) — Complete documentation (MkDocs + Material theme)

   • Backend getting-started & technical reference
   • Workflows getting-started & technical reference
   • Deployable as containerized service in Kubernetes

---

Quick Navigation

For Operators & Platform Engineers

• Engine Documentation — Architecture, deployment, operations
• Engine Howto — Prerequisites, Kubernetes setup, Longhorn, secrets
• Engine Reference — Services, APIs, authentication, security

For Workflow Developers

• Flows Documentation — Getting started with workflows
• Flows Howto — Deploying workflows, connections, policies
• Flows Reference — Workflow YAML syntax, step types, transformations

Repository Structure

meshflows/
├── engine/                  # Microservices & infrastructure
│   ├── services/            # All 8+ microservices
│   ├── deploy/k8s/          # Kubernetes manifests (Kustomize)
│   ├── config/              # Runtime configuration
│   ├── scripts/             # Deployment helper scripts
│   ├── tests/               # pytest test suite
│   └── docker-compose.yml   # Local development stack
│
├── flows/                   # Standalone workflow layer
│   ├── workflows/           # Workflow YAML definitions
│   ├── connections/         # Connection credential templates
│   ├── policies/            # Workflow execution policies
│   ├── schedules/           # Scheduled workflow triggers
│   ├── settings/            # Runtime configuration
│   ├── requests/            # HTTP test client files
│   └── deploy/k8s/          # Flows-only K8s manifests
│
├── wiki/                    # MkDocs documentation site
│   ├── mkdocs.yml           # MkDocs configuration
│   ├── docs/                # Markdown documentation
│   ├── Dockerfile           # Docker build for wiki
│   └── deploy/k8s/          # Wiki K8s manifests
│
└── README.md                # This file

System Architecture

graph TB
    subgraph Components["🏗️ MeshFlows Components"]
        direction TB
        
        subgraph Engine["🔧 Engine (Microservices)"]
            G["Gateway<br/>API Entry"]
            O["Orchestrator<br/>Workflow Engine"]
            T["Transformers<br/>Data Xform"]
            S["Storage<br/>Key-Value"]
            I["Identity<br/>Auth/JWT"]
            SCH["Scheduler<br/>CronJobs"]
            D["Dashboard<br/>Web UI"]
            E["Egress<br/>HTTP/FTP/SSH"]
            R["RabbitMQ<br/>Message Broker"]
        end
        
        subgraph Flows["📋 Flows Layer"]
            WF["Workflows<br/>YAML Definitions"]
            CONN["Connections<br/>Credentials"]
            POL["Policies<br/>Access Control"]
        end
        
        subgraph Obs["📊 Observability"]
            JAEGER["Tempo<br/>Tracing"]
            LOKI["Loki<br/>Logs"]
            TEMPO["Tempo<br/>Metrics"]
        end
        
        WIKI["📚 Wiki<br/>Documentation"]
    end
    
    subgraph Infra["☸️ Infrastructure"]
        K8S["Kubernetes Cluster"]
        LH["Longhorn<br/>Storage"]
    end
    
    O -.->|executes| WF
    G -->|routes to| O
    O -->|calls| T
    O -->|stores in| S
    O -->|retrieves| CONN
    O -->|applies| POL
    G -->|auth via| I
    SCH -->|triggers| O
    E -->|integrates| O
    O -->|publishes to| R
    
    O -.->|traces to| JAEGER
    O -.->|logs to| LOKI
    O -.->|metrics to| TEMPO
    
    D -->|controls| O
    
    Engine -.->|deploys to| K8S
    K8S -->|uses| LH
    
    style Engine fill:#E3F2FD
    style Flows fill:#F3E5F5
    style Obs fill:#FFF3E0
    style Infra fill:#E8F5E9

Deployment Pipeline

For details on our automated CI/CD pipeline with smart build detection and testing, see CI/CD Pipeline Architecture

graph LR
    A["🔄 Push<br/>main/develop"] --> B["🔍 Detect<br/>Changes"]
    B --> C["📦 Build<br/>Changed Only"]
    C --> D["✅ Test"]
    D --> E{All Pass?}
    
    E -->|No| F["❌ Blocked"]
    E -->|Yes| G["🎯 Verify Gate"]
    
    G --> H["🚀 Deploy<br/>to Lab"]
    H --> I["🟢 LIVE"]
    
    style B fill:#FFE082
    style F fill:#EF5350
    style I fill:#66BB6A

Local Pre-commit Validation

Workflow structure validation is enforced in CI and can be enforced locally before every commit.

git config core.hooksPath .githooks

The pre-commit hook runs:

python flows/scripts/validate_workflow_headers.py --changed-only

Optional: enable the commit message template for workflow changes:

git config commit.template .gitmessage

Deprecated interfaces and removal timelines are tracked in DEPRECATIONS.md.

---

Getting Started

Local Development (Docker Compose)

cd engine
docker-compose up -d

This brings up:

• All 8+ microservices
• RabbitMQ broker
• Observability stack (Tempo, Loki, Tempo, OTel Collector)
• Named volumes for persistence

Kubernetes Deployment

cd engine/deploy
kubectl apply -k overlays/minikube/

Or for production Linux cluster:

kubectl apply -k overlays/linux-cluster/

Wiki (Local)

cd wiki
pip install mkdocs mkdocs-material
mkdocs serve

Access at http://localhost:8000

Wiki (Kubernetes)

cd wiki
docker build -t meshflows-wiki:latest .
kubectl apply -f deploy/k8s/

---

Key Features

• YAML-driven Workflows — Declarative workflow definitions with composable steps
• Parallel Execution — Run multiple steps concurrently
• Retry & Compensation — Built-in failure handling and transaction-like rollback
• Transforms — XSLT 1.0, XML↔JSON, Liquid templates
• Integrations — HTTP, FTP/FTPS, SSH/SFTP, RabbitMQ
• Authentication — JWT, OAuth 2.0, scope-based access control
• Observability — OpenTelemetry, Tempo tracing, Loki logs, Tempo backend
• Storage — Role-based key-value store with ACLs
• Scheduler — CronJob-based triggers or HTTP-driven invocation
• Dashboard — Visual workflow editor, execution history, step-level inspection

---

Technology Stack

Layer	Technology
Language	Python 3 (FastAPI)
Container	Docker
Orchestration	Kubernetes (Kustomize)
Storage	Longhorn (persistent volumes)
Message Broker	RabbitMQ 3.13
Tracing	Tempo + Tempo
Logging	Loki + Promtail
Observability	OpenTelemetry
Testing	pytest
CI/CD	GitLab CI (Kaniko builds)

---

Documentation

• Engine How-to Guide — Step-by-step deployment and operations
• Engine Reference — Architecture, services, API documentation
• Workflows How-to Guide — Deploying and managing workflows
• Workflows Reference — Workflow YAML syntax, step types, examples
• Security & Hardening — Enterprise security checklist

---

Requirements

• Kubernetes 1.20+
• Longhorn for persistent storage
• Docker (for local development)
• Python 3.11+ (for local testing)

---

Contributing

Workflows and connections are version controlled in flows/. Push to trigger CI/CD pipeline for validation and automated deployment.

Learn more in the Flows Deployment Guide.

---

License

See LICENSE file.

---

Support & Issues

• Documentation: See wiki/docs/
• Bug reports & features: Use project issue tracker
• Questions: Check our wiki documentation first