Architecture Overview
Technical deep-dive into RevenProx internals.
System Architecture
┌─────────────────────┐
│ Load Balancer │
└──────────┬──────────┘
│
┌──────────────────────────┼──────────────────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ RevenProx │◄──────────►│ RevenProx │◄──────────►│ RevenProx │
│ Instance 1 │ NNG │ Instance 2 │ NNG │ Instance 3 │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└──────────────────────────┼──────────────────────────┘
│
┌──────────▼──────────┐
│ Auth Webhook │
│ (JWT Verification)│
└─────────────────────┘
Component Overview
HTTP Server
Handles incoming SSE connections:
- TCP listener with configurable bind address
- Thread pool for connection handling
- Request parsing and validation
- SSE stream management
Connection Pool
Manages active client connections:
- Connection lifecycle management
- Topic subscription tracking
- Message queuing per connection
- Idle connection cleanup
JWT Verifier
Handles authentication:
- Webhook-based token verification
- Response caching with TTL
- Circuit breaker for fault tolerance
- Rate limiting
Topic Router
Routes messages to subscribers:
- Topic-to-connection mapping
- Efficient subscriber lookup
- Message broadcasting
Distributed State
Synchronizes state across instances:
- NNG-based peer communication
- Gossip protocol for updates
- Bloom filters for membership
- Merkle trees for consistency
Data Flow
Connection Establishment
1. Client Request
│
▼
2. HTTP Server accepts connection
│
▼
3. Parse request, extract topic and JWT
│
▼
4. JWT Verifier checks token
│
├─ Cache hit → Return cached result
│
└─ Cache miss → Call webhook
│
▼
5. Create connection in pool
│
▼
6. Subscribe to topic
│
▼
7. Notify peers (distributed state)
│
▼
8. Begin SSE streaming
Message Delivery
1. Message arrives for topic
│
▼
2. Topic Router looks up subscribers
│
▼
3. For each subscriber:
│
├─ Queue message in connection buffer
│
└─ Signal writer thread
│
▼
4. Writer sends SSE event to client
Distributed Subscription
1. Client subscribes on Proxy A
│
▼
2. Proxy A records local subscription
│
▼
3. Proxy A broadcasts via NNG
│
▼
4. Peers (B, C) receive notification
│
▼
5. Peers update Bloom filters
│
▼
6. Message for topic arrives at Proxy B
│
▼
7. Proxy B forwards to Proxy A
│
▼
8. Proxy A delivers to client
Memory Layout
Per-Connection Memory
Connection (~2KB)
├── ID (8 bytes)
├── Topic UUID (16 bytes)
├── JWT Hash (32 bytes)
├── Timestamps (24 bytes)
├── State/Flags (8 bytes)
├── Message Queue Pointer (8 bytes)
└── Metadata (~1.9KB)
├── User Agent
├── Remote Address
└── Counters
Message Queue
Ring Buffer (per connection)
├── Head pointer (8 bytes)
├── Tail pointer (8 bytes)
├── Capacity (8 bytes)
└── Items array (N × sizeof(Message))
└── Message (~200 bytes avg)
├── Event type
├── ID
└── Data pointer
Bloom Filter
Bloom Filter (subscription lookup)
├── Bit array (capacity × 1.44 × ln(1/fpr) / 8 bytes)
├── Hash seeds (K × 8 bytes)
└── K = ln(2) × m/n optimal hashes
Thread Model
Main Thread
├── Configuration loading
├── Component initialization
└── Signal handling
HTTP Accept Thread
└── Accept loop → dispatch to pool
Thread Pool (N workers)
├── Worker 1: Handle connections
├── Worker 2: Handle connections
├── ...
└── Worker N: Handle connections
Cleanup Thread
└── Periodic idle connection cleanup
NNG Threads (per peer)
├── Publisher thread
└── Subscriber thread
Gossip Thread
└── Periodic state synchronization
Thread Safety
Critical sections protected by:
- Mutexes for connection pool operations
- Atomic operations for counters
- Lock-free queues where possible
- Read-write locks for routing tables
Network Protocol
SSE Wire Format
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no
event: message
id: 1703894400000-001
data: {"type":"notification"}
: keepalive
event: message
id: 1703894400001-002
data: {"type":"update"}
NNG Message Format
┌─────────────────────────────────────┐
│ Message Header (16 bytes) │
├─────────────────────────────────────┤
│ Type (1 byte) │
│ Flags (1 byte) │
│ Sequence (8 bytes) │
│ Payload Length (4 bytes) │
│ Reserved (2 bytes) │
├─────────────────────────────────────┤
│ Payload (variable) │
└─────────────────────────────────────┘
Message types:
- 0x01: Subscription add
- 0x02: Subscription remove
- 0x03: Event broadcast
- 0x04: Heartbeat
- 0x05: State sync request
- 0x06: State sync response
Consistency Model
Eventual Consistency
Subscriptions converge across all nodes:
t=0: Client subscribes on Node A
t=1: Node A has subscription
t=2: Gossip propagates to Node B
t=3: Node B has subscription
...
t=N: All nodes consistent
Vector Clocks
Track causality for conflict resolution:
Merkle Tree Sync
Efficient state comparison:
Nodes exchange root hashes; differences trigger subtree sync.
Fault Tolerance
Circuit Breaker States
┌─────────┐ ┌─────────┐ ┌───────────┐
│ Closed │──fail──►│ Open │──time──►│ Half-Open │
│(normal) │ │(failing)│ │ (testing) │
└────▲────┘ └─────────┘ └─────┬─────┘
│ │
└──────────────success───────────────────┘
Peer Failure Handling
1. Heartbeat timeout detected
2. Mark peer as suspected
3. Continue gossip to other peers
4. After peer_timeout_sec, mark as failed
5. Remove peer from routing
6. Reconnection attempts continue
7. On reconnect, full state sync
Performance Characteristics
Complexity
| Operation | Time Complexity |
|---|---|
| Connection add | O(1) |
| Connection remove | O(1) |
| Topic subscribe | O(1) |
| Message broadcast | O(N) subscribers |
| Bloom filter check | O(K) hashes |
| Merkle sync | O(log N) |
Latency Budget
Total message latency breakdown:
Receive message: ~0.1ms
Topic lookup: ~0.01ms
Queue per subscriber: ~0.01ms × N
Network write: ~0.1ms
─────────────────────────────────
Total (1 subscriber): ~0.22ms
Total (100 subscribers): ~1.2ms
Throughput Limits
Per instance (modern server): - Connections: 100,000+ - Messages/sec: 100,000+ - Bandwidth: Limited by NIC