Architecture Overview¶

Machineuse is a distributed container management system designed for browser automation at scale.

System Architecture¶

┌─────────────────────────────────────────────────────────────────┐
│                         Clients                                  │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐            │
│  │   CLI   │  │   SDK   │  │REST API │  │  Web UI │            │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘            │
└───────┼────────────┼────────────┼────────────┼──────────────────┘
        │            │            │            │
        └────────────┴─────┬──────┴────────────┘
                           │
┌──────────────────────────┴──────────────────────────────────────┐
│                      Control Plane                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │  API Server  │  │   Scheduler  │  │  Metadata    │          │
│  │   (FastAPI)  │  │              │  │  (Postgres)  │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                           │                                      │
│                    NNG Messaging                                 │
└───────────────────────────┬─────────────────────────────────────┘
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
        ▼                   ▼                   ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│   Worker 1    │   │   Worker 2    │   │   Worker N    │
│ ┌───────────┐ │   │ ┌───────────┐ │   │ ┌───────────┐ │
│ │  Agent    │ │   │ │  Agent    │ │   │ │  Agent    │ │
│ ├───────────┤ │   │ ├───────────┤ │   │ ├───────────┤ │
│ │ Container │ │   │ │ Container │ │   │ │ Container │ │
│ │  Runtime  │ │   │ │  Runtime  │ │   │ │  Runtime  │ │
│ ├───────────┤ │   │ ├───────────┤ │   │ ├───────────┤ │
│ │ SQLite DB │ │   │ │ SQLite DB │ │   │ │ SQLite DB │ │
│ └───────────┘ │   │ └───────────┘ │   │ └───────────┘ │
└───────────────┘   └───────────────┘   └───────────────┘

Core Components¶

Control Plane¶

The control plane is the central coordinator:

API Server: FastAPI-based REST API for client interactions
Scheduler: Intelligent instance placement based on node capabilities
Metadata Store: PostgreSQL for cluster state and instance metadata

Worker Nodes¶

Worker nodes execute containers:

Agent: Handles control plane communication and local orchestration
Container Runtime: systemd-nspawn for isolated browser environments
Local Storage: SQLite for local state, DuckDB for metrics

Communication Layer¶

NNG (Nanomsg Next Generation): High-performance messaging without external brokers
Patterns: REQ/REP for commands, PUB/SUB for events, PUSH/PULL for tasks

Instance Lifecycle¶

                    ┌─────────┐
                    │ Creating│
                    └────┬────┘
                         │
                         ▼
┌────────┐         ┌─────────┐         ┌─────────┐
│ Failed │◄────────│ Running │────────►│  Idle   │
└────────┘         └────┬────┘         └────┬────┘
                        │                   │
                        │                   ▼
                        │              ┌─────────┐
                        │              │ Dormant │
                        │              └────┬────┘
                        │                   │
                        ▼                   │
                   ┌─────────┐              │
                   │ Deleted │◄─────────────┘
                   └─────────┘

States¶

State	Description
Creating	Instance being provisioned
Running	Active and accessible
Idle	No activity detected
Dormant	Snapshotted, resources released
Failed	Error during operation
Deleted	Instance removed

Storage Architecture¶

Three-Tier Storage¶

┌─────────────────────────────────────────────────────────┐
│                    Storage Tiers                         │
├─────────────────┬─────────────────┬─────────────────────┤
│   Operational   │    Analytics    │    File System      │
│   (SQL)         │    (DuckDB)     │                     │
├─────────────────┼─────────────────┼─────────────────────┤
│ • Instances     │ • Metrics       │ • Containers        │
│ • Nodes         │ • Usage stats   │ • Snapshots         │
│ • Config        │ • Performance   │ • Logs              │
│ • Sessions      │ • Trends        │ • Base images       │
└─────────────────┴─────────────────┴─────────────────────┘

Storage Backends¶

SQLite: Local storage for workers and single-node deployments
PostgreSQL: Distributed storage for control plane
DuckDB: Time-series analytics and metrics

Scheduling¶

Weighted Scoring Algorithm¶

The scheduler evaluates nodes using weighted scores:

Score = (CPU_weight × CPU_available) +
        (Memory_weight × Memory_available) +
        (Disk_weight × Disk_available) +
        (Network_weight × Network_capacity)

Default weights:

Factor	Weight
CPU	0.30
Memory	0.40
Disk	0.20
Network	0.10

Placement Considerations¶

Resource availability: Sufficient CPU, memory, disk
Node affinity: User preferences for specific nodes
Anti-affinity: Spreading instances across nodes
Load balancing: Even distribution of workload

Container Runtime¶

systemd-nspawn¶

Machineuse uses systemd-nspawn for container isolation:

Process isolation: Separate PID namespace
Network isolation: Virtual ethernet pairs
Filesystem isolation: Overlayfs with base images
Resource limits: CPU, memory, and I/O constraints

Container Lifecycle¶

# Create from base image
systemd-nspawn --directory=/var/lib/machines/instance-xyz ...

# Resource limits
systemd-nspawn --property=CPUQuota=200% \
               --property=MemoryMax=4G ...

# Snapshot for dormancy
machinectl snapshot instance-xyz

Snapshot System¶

Dormancy Flow¶

Idle Detection: Monitor for activity timeout
State Capture: Capture process and filesystem state
Compression: Gzip compress snapshot
Storage: Store in /var/lib/machineuse/snapshots/
Cleanup: Release container resources

Revival Flow¶

Load Snapshot: Decompress from storage
Restore State: Apply filesystem snapshot
Start Container: Boot with restored state
Verify: Health check confirmation

High Availability¶

Control Plane HA¶

Multiple control plane instances
Shared PostgreSQL backend
Leader election for scheduler

Worker Resilience¶

Heartbeat monitoring
Automatic failover
Instance migration on node failure

Security Model¶

Instance Isolation¶

Separate network namespaces
Dedicated resource quotas
No shared filesystem access

Access Control¶

Instance-level authentication tokens
API key authentication
Role-based access (future)

Network Security¶

Internal NNG communication
TLS support for inter-node traffic
Firewall rules for isolation