Troubleshooting

This guide outlines common problems affecting non-validator nodes, with practical diagnostics and resolution steps. Rapid troubleshooting is critical for maintaining RPC availability and reliable access to stablecoin transaction data.

​

Synchronisation Issues

​

Node Lags Behind

If your node falls behind the network head:

• Check system load: CPU, memory, and I/O must be sufficient for Plasma’s high-frequency block production.
• Verify consensus endpoint latency: High latency impacts block ingestion speed.
• Monitor database performance: Slow queries or lock contention can bottleneck sync.

Diagnostics:

• block_height vs network head
• State application time per block
• Latency to each consensus endpoint

​

Sync Stalls Completely

If syncing halts:

• Check disk space: Full disks halt database writes.
• Verify endpoint connectivity: Ensure DNS resolution, firewall rules, and routing are not blocking consensus traffic.
• Check container resource limits: Insufficient memory or CPU can crash sync processes.

Plasma-specific checks:

• Endpoint reachability
• Valid JWT tokens / allowlist status
• Non-validator node version compatibility

​

Poor Peer Connectivity

Limited or unstable connections to consensus endpoints reduce data freshness for RPC consumers. Check:

• IP restrictions or routing policies
• Endpoint connection count and uptime
• Failover behavior across multiple endpoints

Metrics to monitor:

• Active connections
• Disconnect rate
• Block arrival lag

​

Network Connectivity

​

Port and Firewall Access

Ensure required ports are open for both:

• Consensus communication
• RPC serving to applications

Common blockers include corporate firewalls, cloud security groups, or misconfigured iptables rules. Verify:

• Port reachability from outside and inside container
• Allowed outbound traffic for consensus sync

​

DNS Failures

Unresolved consensus domains break sync.

• Confirm DNS resolution for all Plasma service domains
• Monitor resolver latency and reliability
• Add fallback resolvers as needed

​

Proxy and NAT Issues

Complex networking environments (VPNs, proxies, NATs) may interfere with:

• Consensus sync
• RPC inbound access

Validate:

• Proxy authentication rules
• NAT port forwarding for inbound RPC
• Routing paths to Plasma infrastructure

​

Configuration Errors

​

Invalid Non-Validator Config

Startup errors are often caused by:

• Incorrect or missing consensus endpoints
• Malformed URLs or JWT tokens
• Deprecated or unsupported flags

Check logs for:

• Configuration parse errors
• Chain ID mismatches
• Unknown flags or options

​

Docker Issues

Misconfigured containers can prevent normal operation. Validate:

• Volume mounts for database persistence
• Network access inside container
• Resource limits (CPU, memory)
• Image version compatibility

​

Environment Variable Conflicts

Conflicting or misordered env vars may override intended settings. Verify:

• Consensus endpoint URLs and tokens
• Network and chain ID
• Feature flags (e.g. custom gas support)

Document environment variable usage to avoid drift between deployments.

​

Resource Limits

​

Disk Space

Monitor disk usage growth due to:

• Blockchain data
• Logs and temporary files

Best practices:

• Implement log rotation
• Schedule cleanup jobs
• Monitor usage alerts

​

File Descriptor

Low ulimits can break:

• Consensus connections
• RPC sessions

Increase file descriptor limits via ulimit, systemd, or container runtime configs.

​

Diagnostics

​

Log Analysis

Use logs to trace:

• Sync progress
• RPC errors
• Consensus connectivity
• Resource-related crashes

Best practices:

• Consensus synchronisation logs: Track connectivity and sync progress with Plasma.
• Transaction processing logs: Monitor transaction handling and errors.
• RPC request logs: Analyse payment application request patterns and performance.
• Resource utilisation logs: Identify bottlenecks affecting payment processing.

​

Performance Profiling

Profile non-validator workloads to identify bottlenecks in:

• RPC performance
• DB query latency
• Consensus state application

Focus areas:

• Balance check throughput
• Transfer confirmation latency
• Resource usage under load

Maintain baselines and track changes across upgrades or config changes. Systematic troubleshooting and observability are essential to reliable non-validator node operation. Proper diagnostics, logging, and configuration hygiene ensure stable performance and uptime for RPC services supporting stablecoin infrastructure.