Skip to main content

Covalent Troubleshooting Guide

This guide helps you diagnose and resolve common issues with your Covalent Refiner node.

Quick Diagnostics​

Before diving into specific issues, run this diagnostic script:

quick-diagnostics.sh
#!/bin/bash
echo "=== Covalent Refiner Diagnostics ==="
echo ""

# Check if Docker services are running
echo "1. Docker Services Status:"
docker-compose ps 2>/dev/null || echo "❌ Docker Compose not running"
echo ""

# Check disk space
echo "2. Disk Space:"
df -h | grep -E "^/dev|Filesystem"
echo ""

# Check memory
echo "3. Memory Usage:"
free -h
echo ""

# Check running containers
echo "4. Running Containers:"
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"
echo ""

# Check recent errors in rudder logs
echo "5. Recent Rudder Errors:"
docker logs rudder 2>&1 | grep -i error | tail -10
echo ""

# Check EVM server status
echo "6. EVM Server Status:"
curl -s http://localhost:3002/health 2>/dev/null || echo "❌ EVM Server not responding"
echo ""

# Check IPFS Pinner status
echo "7. IPFS Pinner Status:"
curl -s http://localhost:3001/health 2>/dev/null || echo "❌ IPFS Pinner not responding"

Common Issues and Solutions​

1. Docker Issues​

Symptoms:

  • Docker containers exit immediately
  • docker-compose up fails

Solution:

# 1. Check container logs
docker logs rudder
docker logs evm-server
docker logs ipfs-pinner

# 2. Check environment variables
cat .env
# Ensure all required variables are set:
# - BLOCK_RESULT_OPERATOR_PRIVATE_KEY
# - NODE_ETHEREUM_MAINNET
# - WEB3_JWT

# 3. Restart containers
docker-compose down
docker-compose up -d

# 4. If still failing, rebuild
docker-compose down --volumes
docker-compose pull
docker-compose up -d

2. Rudder Pipeline Issues​

Symptoms:

  • Block specimens not being processed
  • Pipeline errors in logs

Solution:

# 1. Check Rudder logs
docker logs rudder -f

# 2. Verify environment variables
echo $BLOCK_RESULT_OPERATOR_PRIVATE_KEY | wc -c  # Should be 64 characters
echo $NODE_ETHEREUM_MAINNET  # Should be valid Ethereum RPC URL

# 3. Check operator status on dashboard
# Visit: https://www.covalenthq.com/staking/#/operator-dashboard/

# 4. Ensure sufficient CQT staking (35k testnet CQT)

# 5. Restart Rudder if needed
docker-compose restart rudder

3. EVM Server Issues​

Symptoms:

  • Cannot connect to port 3002
  • EVM execution failures

Solution:

# 1. Check EVM server status
docker logs evm-server

# 2. Test connectivity
curl -s http://localhost:3002/health

# 3. Restart EVM server
docker-compose restart evm-server

# 4. Check system resources
docker stats evm-server

# 5. If persistent issues, rebuild
docker-compose stop evm-server
docker-compose rm evm-server
docker-compose up -d evm-server

4. IPFS Pinner Issues​

Symptoms:

  • Cannot upload/download from IPFS
  • Port 3001 not responding

Solution:

# 1. Check IPFS pinner logs
docker logs ipfs-pinner

# 2. Verify WEB3_JWT token
# Ensure valid web3.storage API token is set

# 3. Test IPFS connectivity
curl -s http://localhost:3001/health

# 4. Check web3.storage quota
# Login to web3.storage dashboard to check usage

# 5. Restart IPFS pinner
docker-compose restart ipfs-pinner

5. Network and Connectivity Issues​

Symptoms:

  • Cannot connect to Ethereum RPC
  • Network timeout errors

Solution:

# 1. Test RPC endpoint
curl -s $NODE_ETHEREUM_MAINNET -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# 2. Check firewall settings
sudo ufw status

# 3. Try alternative RPC endpoints
# Update NODE_ETHEREUM_MAINNET in .env with backup RPC

# 4. Check DNS resolution
nslookup $(echo $NODE_ETHEREUM_MAINNET | cut -d'/' -f3)

# 5. Test with different provider (Alchemy, Infura, etc.)

6. Performance Issues​

Symptoms:

  • Very slow block specimen processing
  • High latency in pipeline

Solution:

# 1. Check system resources
htop
iotop -o

# 2. Optimize Docker settings
# Add to docker-compose.yml:
# services:
#   rudder:
#     cpus: '2.0'
#     mem_limit: 4g

# 3. Use SSD storage for better I/O
# Mount data directories on SSD

# 4. Increase worker threads (if supported)
# Check Rudder configuration options

# 5. Monitor metrics
curl -s http://localhost:9568/metrics | grep rudder

Performance Optimization​

System Tuning​

# Increase file descriptors
echo "fs.file-max = 65535" | sudo tee -a /etc/sysctl.conf

# Docker optimization
echo '{"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}}' | sudo tee /etc/docker/daemon.json

# Network optimization
echo "net.core.rmem_default = 262144" | sudo tee -a /etc/sysctl.conf
echo "net.core.rmem_max = 16777216" | sudo tee -a /etc/sysctl.conf

sudo sysctl -p
sudo systemctl restart docker

Monitoring Setup​

# Monitor resource usage
watch -n 5 'docker stats --no-stream'

# Monitor logs
docker-compose logs -f --tail=100

# Check metrics
curl -s http://localhost:9568/metrics | grep -E "(rudder|ipfs|evm)"

# Monitor disk usage
watch -n 30 'df -h'

Emergency Procedures​

Complete Reset​

# WARNING: This will delete all local data
docker-compose down --volumes
docker system prune -a
docker-compose up -d

Backup and Restore​

# Create backup
docker-compose down
tar -czf covalent-backup-$(date +%Y%m%d).tar.gz .env docker-compose.yml

# Restore from backup
tar -xzf covalent-backup-YYYYMMDD.tar.gz
docker-compose up -d

Data Migration​

# If moving to new server
# 1. Stop services
docker-compose down

# 2. Backup data and config
tar -czf covalent-migration.tar.gz .env docker-compose.yml

# 3. Transfer to new server
scp covalent-migration.tar.gz user@newserver:~/

# 4. Restore on new server
tar -xzf covalent-migration.tar.gz
docker-compose up -d

Getting Help​

If you're still experiencing issues:

  1. Check the logs: docker-compose logs -f
  2. Visit the documentation: Covalent Docs
  3. Join the Discord: Covalent Discord
  4. Check operator dashboard: Operator Dashboard

When asking for help, always include:

  • Your OS and version
  • Docker version: docker --version
  • Docker Compose version: docker-compose --version
  • Container logs: docker-compose logs
  • Environment variables (remove private keys): cat .env
  • System resources: docker stats