This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Backend (Engine) — Microservices Infrastructure

The Engine is the core infrastructure and microservices layer of MeshFlows.

What is the Engine?

The Engine provides all the operational infrastructure needed to:

• Execute workflows — YAML-driven workflow orchestration
• Handle requests — Gateway for public API access and alias route mapping
• Transform data — XSLT, XML↔JSON, Liquid templates
• Store state — Runtime key-value store & document management
• Authenticate users — JWT tokens, OAuth 2.0, identity management
• Schedule tasks — CronJob-based workflow triggers
• Manage integrations — HTTP, FTP, SSH, RabbitMQ connections
• Monitor everything — Observability stack (Tempo, Loki, Tempo)

Services at a Glance

Service	Purpose
Gateway	Public API entry point (port 8080)
Orchestrator	Workflow execution engine (port 8083)
Transformers	Data transformations (port 8081)
Storage	Key-value store with ACLs (port 8086)
Identity	User management & JWT (port 8088)
Scheduler	CronJob triggers (port 8089)
Dashboard	Web UI for workflows (port 8090)
Egress Services	HTTP, FTP, SSH, RabbitMQ (ports 8082-8087)
RabbitMQ	Message broker (ports 5672, 15672)

Start Here

New Platform Setup

1. Prerequisites
2. Kubernetes Deployment
3. Secrets Management
4. Operations

Local Development

1. Local Development
2. Longhorn Storage Setup
3. Flows Overview (Forgejo mirror: Flows Overview)

API and Runtime Internals

1. Architecture
2. Services Overview
3. Gateway API
4. Orchestrator API
5. Orchestrator Messaging Blueprint

I’m an Operator — What Do I Need to Know?

First Time Setup?

1. Prerequisites — Get your environment ready
2. Kubernetes Deployment — Deploy to your cluster
3. Operations & Monitoring — Run and manage the system

Local Development?

1. Local Development (Docker Compose)
2. Secrets Management

Going Deeper?

• Architecture Overview — System design
• Security & Hardening — Production hardening
• Observability Setup — Monitoring & tracing

Deployment Options

Docker Compose (Development/Testing)

cd engine
docker-compose up -d

All services + RabbitMQ + observability stack in one command.

Kubernetes (Production)

cd engine/deploy
kubectl apply -k overlays/linux-cluster/

Full production-ready deployment with persistence, networking, and monitoring.

Minikube (Learning)

cd engine/deploy
kubectl apply -k overlays/minikube/

Simplified deployment for learning and testing on a local cluster.

Key Features

• YAML Workflows — Declarative workflow definitions
• Parallel Execution — Concurrent step execution
• Retry & Compensation — Automatic failure recovery
• Data Transformations — XSLT 1.0, XML↔JSON, Liquid
• Integrations — HTTP, FTP, SSH, RabbitMQ
• Edge Route Aliases — Public endpoint-to-workflow mapping in Gateway
• OAuth 2.0 — Inbound and outbound authentication
• Observability — OpenTelemetry + Tempo + Loki
• Storage ACLs — Role-based access control
• Scheduler — CronJob or HTTP-driven triggers
• Dashboard — Visual workflow editor

Technology Stack

• Language: Python 3.11+ (FastAPI)
• Container: Docker
• Orchestration: Kubernetes (Kustomize)
• Storage: Longhorn persistent volumes
• Message Broker: RabbitMQ
• Observability: OpenTelemetry + Tempo + Loki + Tempo
• Testing: pytest
• CI/CD: Forgejo Actions + container image build pipeline

Common Tasks

Deploy to Production

→ Kubernetes Deployment How-to

Configure Secrets

→ Secrets Management How-to

Setup Longhorn Storage

→ Longhorn Storage How-to

Monitor with Tempo/Loki

→ Observability Reference

Secure Your Deployment

→ Security & Hardening Reference

Understand Our CI/CD Pipeline

→ CI/CD Pipeline Architecture — How builds, tests, and deployments work

Understand OAuth

→ Authentication Reference

Next Steps

1. Operators/DevOps: Start with Prerequisites
2. Developers: Check out Flows (Workflows) documentation (Forgejo mirror: Flows (Workflows))
3. Learning: Try Local Development