Troubleshooting
Common issues and solutions for RevenProx.
Connection Issues
Clients Can't Connect
Symptoms: - Connection refused errors - Timeout on connection
Diagnosis:
# Check if proxy is running
systemctl status revenprox
# Check if port is listening
ss -tlnp | grep 8080
# Test local connection
curl -v http://localhost:8080/health
Solutions:
1. Verify bind_address in configuration
2. Check firewall rules
3. Ensure proxy service is running
4. Verify no port conflicts
503 Service Unavailable
Symptoms: - Clients receive 503 errors - "Too many connections" message
Diagnosis:
# Check connection count
curl http://localhost:8080/health | jq .connections
# Check system limits
ulimit -n
cat /proc/$(pgrep sse-proxy)/limits | grep "open files"
Solutions:
1. Increase max_connections in config
2. Increase system file descriptor limits
3. Scale horizontally with more instances
4. Check for connection leaks
Connections Dropping
Symptoms: - Clients disconnect unexpectedly - Frequent reconnections
Diagnosis:
# Check for timeout issues
grep -i timeout /var/log/revenprox/proxy.log
# Check keepalive settings
grep keepalive /etc/revenprox/proxy.toml
Solutions:
1. Increase connection_timeout_sec
2. Reduce keepalive_interval_sec
3. Check load balancer timeout settings
4. Verify network stability
Authentication Issues
401 Unauthorized
Symptoms: - All requests return 401 - Valid tokens rejected
Diagnosis:
# Check webhook connectivity
curl -X POST http://auth-server/verify \
-H "Content-Type: application/json" \
-d '{"jwt":"test","topic":"test"}'
# Check circuit breaker state
curl http://localhost:8080/health | jq .circuit_breaker
Solutions:
1. Verify webhook URL is correct and reachable
2. Check webhook response format
3. Reset circuit breaker (restart proxy)
4. Increase timeout_ms for slow webhooks
Circuit Breaker Open
Symptoms: - All auth requests fail immediately - Logs show "circuit breaker open"
Diagnosis:
# Check webhook health
curl -I http://auth-server/health
# Check recent failures
grep "circuit breaker" /var/log/revenprox/proxy.log | tail -20
Solutions:
1. Fix the authentication webhook
2. Wait for circuit_breaker_timeout_sec
3. Restart proxy to reset circuit breaker
4. Increase circuit_breaker_threshold
Token Caching Issues
Symptoms: - Revoked tokens still work - Token changes not reflected
Solutions:
1. Reduce cache_ttl_sec for faster revocation
2. Clear cache by restarting proxy
3. Implement token revocation webhook
Performance Issues
High CPU Usage
Symptoms: - CPU consistently above threshold - Slow response times
Diagnosis:
Solutions:
1. Reduce thread_pool_size if overprovisioned
2. Enable connection limits
3. Optimize message payload sizes
4. Scale horizontally
High Memory Usage
Symptoms: - Memory grows continuously - OOM killer terminates process
Diagnosis:
# Check memory usage
ps aux | grep sse-proxy
# Check for memory leaks
grep -i "memory" /var/log/revenprox/proxy.log
Solutions:
1. Reduce max_message_queue_size
2. Reduce cache_max_size
3. Lower max_connections
4. Check for connection leaks
Message Latency
Symptoms: - High message delivery times - Clients receive stale data
Diagnosis:
Solutions:
1. Increase thread_pool_size
2. Reduce message payload sizes
3. Enable message batching
4. Scale horizontally
Distributed State Issues
Peers Not Connecting
Symptoms:
- peers_connected metric is 0
- Subscriptions not syncing
Diagnosis:
Solutions:
1. Verify peer_addresses configuration
2. Check firewall allows NNG port
3. Ensure all peers use same NNG version
4. Check for network partitions
Subscription Sync Lag
Symptoms: - Events not reaching all clients - Delay in subscription visibility
Diagnosis:
# Check sync lag
curl http://localhost:8080/metrics | grep sync_lag
# Check gossip activity
grep gossip /var/log/revenprox/proxy.log | tail -20
Solutions:
1. Reduce gossip_interval_sec
2. Check network latency between peers
3. Increase message_batch_size
4. Verify peer health
Split Brain
Symptoms: - Inconsistent subscriptions - Events delivered to wrong clients
Diagnosis:
# Compare subscription counts across peers
for host in proxy1 proxy2 proxy3; do
echo "$host: $(curl -s http://$host:8080/metrics | grep topics_active)"
done
Solutions:
1. Check for network partitions
2. Restart affected proxies
3. Verify clock synchronization (NTP)
4. Review peer_timeout_sec setting
Rate Limiting Issues
429 Too Many Requests
Symptoms: - Legitimate clients being rate limited - Burst traffic rejected
Diagnosis:
# Check rate limit hits
curl http://localhost:8080/metrics | grep rate_limit
# Check client IPs
grep "rate limit" /var/log/revenprox/proxy.log
Solutions:
1. Increase rate_limit_per_ip
2. Increase rate_limit_window_sec
3. Whitelist trusted IPs at load balancer
4. Implement client-side backoff
Startup Issues
Proxy Won't Start
Symptoms: - Service fails to start - Immediate exit on startup
Diagnosis:
# Check service status
systemctl status revenprox
# Check logs
journalctl -u revenprox -n 100
# Validate config
./sse-proxy --config /etc/revenprox/proxy.toml --validate
Solutions: 1. Fix configuration errors 2. Check file permissions 3. Ensure dependencies (libnng) installed 4. Verify port not in use
Configuration Errors
Common errors:
- Fix: Ensure address format is correct (e.g.,0.0.0.0:8080)
- Fix: Either set webhook_url or set require_authentication = false
- Fix: Set a valid percentage value
Log Analysis
Finding Errors
# Recent errors
grep -i error /var/log/revenprox/proxy.log | tail -50
# Authentication failures
grep -i "auth\|jwt\|unauthorized" /var/log/revenprox/proxy.log
# Connection issues
grep -i "connect\|disconnect\|timeout" /var/log/revenprox/proxy.log
Debug Logging
Enable debug logs temporarily:
Warning
Debug logging is verbose and impacts performance. Use only for troubleshooting.
Getting Help
Information to Collect
When reporting issues, include:
- RevenProx version (
./sse-proxy --version) - Configuration file (redact secrets)
- Relevant log entries
- System information (
uname -a, memory, CPU) - Steps to reproduce
Community Support
- GitHub Issues: Report bugs and feature requests
- Documentation: Check all relevant sections