Covalent Refiner Upgrade Guide

This guide covers upgrading your Covalent Refiner components to newer versions safely and efficiently.

Before You Start

Pre-Upgrade Checklist

# 1. Check current versions
docker images | grep covalent
docker-compose ps

# 2. Backup your configuration
cp .env .env.backup
cp docker-compose.yml docker-compose.yml.backup

# 3. Check system resources
df -h  # Need at least 5GB free space
free -h  # Check available RAM

# 4. Verify current operator status
# Check your operator dashboard at:
# https://www.covalenthq.com/staking/#/operator-dashboard/

Important Notes

warning

Always backup your configuration before upgrading. Some upgrades may require environment variable updates.

info

Check the Covalent Network announcements for upgrade timing and special instructions.

Upgrade Methods

Method 1: Docker Compose Upgrade (Recommended)

Standard Upgrade
Component-Specific

# 1. Stop current services
docker-compose down

# 2. Pull latest images
docker-compose pull

# 3. Start with new versions
docker-compose up -d

# 4. Check logs for issues
docker-compose logs -f

# 5. Verify services are healthy
docker-compose ps
curl -s http://localhost:3001/health  # IPFS Pinner
curl -s http://localhost:3002/health  # EVM Server

# Upgrade specific components only

# Upgrade Rudder only
docker-compose stop rudder
docker-compose pull rudder
docker-compose up -d rudder

# Upgrade EVM Server only
docker-compose stop evm-server
docker-compose pull evm-server  
docker-compose up -d evm-server

# Upgrade IPFS Pinner only
docker-compose stop ipfs-pinner
docker-compose pull ipfs-pinner
docker-compose up -d ipfs-pinner

Method 2: Manual Docker Upgrade

# Stop containers
docker stop rudder evm-server ipfs-pinner

# Remove old containers (keeps volumes)
docker rm rudder evm-server ipfs-pinner

# Pull latest images
docker pull us-docker.pkg.dev/covalent-project/network/rudder:stable
docker pull us-docker.pkg.dev/covalent-project/network/evm-server:stable
docker pull us-docker.pkg.dev/covalent-project/network/ipfs-pinner:stable

# Restart with docker-compose
docker-compose up -d

Component-Specific Upgrades

Rudder Upgrade

# 1. Check current Rudder version
docker exec rudder mix version

# 2. Stop Rudder
docker-compose stop rudder

# 3. Pull latest Rudder image
docker pull us-docker.pkg.dev/covalent-project/network/rudder:stable

# 4. Start Rudder
docker-compose up -d rudder

# 5. Monitor startup
docker logs rudder -f

# 6. Verify metrics
curl -s http://localhost:9568/metrics | grep rudder

EVM Server Upgrade

# 1. Stop EVM Server
docker-compose stop evm-server

# 2. Pull latest image
docker pull us-docker.pkg.dev/covalent-project/network/evm-server:stable

# 3. Start EVM Server
docker-compose up -d evm-server

# 4. Test functionality
curl -X POST http://localhost:3002 \
-H "Content-Type: application/json" \
-d '{"method":"evm_version","params":[],"id":1}'

IPFS Pinner Upgrade

# 1. Stop IPFS Pinner
docker-compose stop ipfs-pinner

# 2. Pull latest image
docker pull us-docker.pkg.dev/covalent-project/network/ipfs-pinner:stable

# 3. Start IPFS Pinner
docker-compose up -d ipfs-pinner

# 4. Test upload capability
curl -X POST http://localhost:3001/api/v0/add \
-F file=@test.txt

Configuration Updates

Environment Variables

Check for new environment variables in upgrades:

.env
# Core Configuration (Always Required)
BLOCK_RESULT_OPERATOR_PRIVATE_KEY=your_64_char_private_key
NODE_ETHEREUM_MAINNET=https://eth-mainnet.alchemyapi.io/v2/YOUR_KEY
WEB3_JWT=your_web3_storage_jwt_token

# Network Configuration
NETWORK=moonbase-alpha  # or mainnet when available

# Service URLs (Internal Docker Network)
IPFS_PINNER_URL=http://ipfs-pinner:3001
EVM_SERVER_URL=http://evm-server:3002

# New in Recent Versions (check upgrade notes)
PROOF_CHAIN_TIMEOUT=300
BATCH_SIZE=100
RETRY_ATTEMPTS=3

# Monitoring (Optional)
ENABLE_METRICS=true
METRICS_PORT=9568

# Performance Tuning (Optional)
WORKER_THREADS=4
MAX_MEMORY=4096

Docker Compose Updates

docker-compose.yml
version: '3.8'

services:
evm-server:
  image: us-docker.pkg.dev/covalent-project/network/evm-server:stable
  container_name: evm-server
  restart: unless-stopped
  ports:
    - "3002:3002"
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3002/health"]
    interval: 30s
    timeout: 10s
    retries: 3
  networks:
    - cqt-net

ipfs-pinner:
  image: us-docker.pkg.dev/covalent-project/network/ipfs-pinner:stable  
  container_name: ipfs-pinner
  restart: unless-stopped
  ports:
    - "3001:3001"
  environment:
    - WEB3_JWT=${WEB3_JWT}
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3001/health"]
    interval: 30s
    timeout: 10s
    retries: 3
  networks:
    - cqt-net

rudder:
  image: us-docker.pkg.dev/covalent-project/network/rudder:stable
  container_name: rudder
  restart: unless-stopped
  depends_on:
    evm-server:
      condition: service_healthy
    ipfs-pinner:
      condition: service_healthy
  environment:
    - BLOCK_RESULT_OPERATOR_PRIVATE_KEY=${BLOCK_RESULT_OPERATOR_PRIVATE_KEY}
    - NODE_ETHEREUM_MAINNET=${NODE_ETHEREUM_MAINNET}
    - IPFS_PINNER_URL=${IPFS_PINNER_URL}
    - EVM_SERVER_URL=${EVM_SERVER_URL}
    - WEB3_JWT=${WEB3_JWT}
  ports:
    - "9568:9568"  # Metrics port
  networks:
    - cqt-net

# Optional: Autoheal container
autoheal:
  image: willfarrell/autoheal
  container_name: autoheal
  restart: unless-stopped
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - AUTOHEAL_CONTAINER_LABEL=autoheal
    - AUTOHEAL_INTERVAL=10

networks:
cqt-net:
  driver: bridge

Version-Specific Upgrades

Upgrading to Mainnet

When mainnet becomes available:

# 1. Update environment for mainnet
export NETWORK=mainnet
export NODE_ETHEREUM_MAINNET=your_mainnet_ethereum_rpc

# 2. Update contract addresses (will be announced)
export PROOF_CHAIN_CONTRACT=0x_new_mainnet_address

# 3. Ensure sufficient ETH for gas fees on mainnet
# Check your operator wallet balance

# 4. Update to mainnet-compatible versions
docker-compose pull
docker-compose up -d

# 5. Monitor transition carefully
docker-compose logs -f rudder

Upgrading to v2.0 (Hypothetical Major Version)

# Major version upgrades may require data migration
# 1. Stop all services
docker-compose down

# 2. Backup current data
mkdir -p backups
docker run --rm -v cqt_rudder_data:/data -v $(pwd)/backups:/backup \
alpine tar czf /backup/rudder-data-$(date +%Y%m%d).tar.gz -C /data .

# 3. Update configuration for v2.0
# New required variables (example):
export RUDDER_VERSION=2.0
export NEW_FEATURE_ENABLED=true

# 4. Pull v2.0 images
docker-compose pull

# 5. Run migration script (if provided)
docker run --rm -v cqt_rudder_data:/data \
us-docker.pkg.dev/covalent-project/network/migration:v2.0

# 6. Start services
docker-compose up -d

Post-Upgrade Verification

Health Checks

# 1. Check all containers are running
docker-compose ps

# 2. Verify service health
curl -s http://localhost:3001/health | jq .  # IPFS Pinner
curl -s http://localhost:3002/health | jq .  # EVM Server

# 3. Check Rudder pipeline status
docker logs rudder | tail -20

# 4. Verify metrics endpoint
curl -s http://localhost:9568/metrics | grep rudder_pipeline

# 5. Check operator dashboard
# Visit: https://www.covalenthq.com/staking/#/operator-dashboard/

# 6. Monitor for block specimen processing
docker logs rudder | grep -i "found.*bsps"

Performance Validation

# Monitor resource usage
docker stats --no-stream

# Check processing metrics
watch -n 30 'curl -s http://localhost:9568/metrics | grep -E "(pipeline_success|upload_success)"'

# Monitor error rates
docker logs rudder | grep -i error | tail -10

# Check disk usage growth
df -h
du -sh /var/lib/docker/volumes/

Rollback Procedure

# If upgrade causes issues, rollback:

# 1. Stop current services
docker-compose down

# 2. Restore previous configuration
cp .env.backup .env
cp docker-compose.yml.backup docker-compose.yml

# 3. Use previous image versions
# Edit docker-compose.yml to specify previous tags
# Example: change :stable to :v1.2.3

# 4. Start services
docker-compose up -d

# 5. Verify rollback successful
docker-compose ps
curl -s http://localhost:3001/health

Automation Scripts

Automated Upgrade Script

upgrade-refiner.sh
#!/bin/bash
set -e

echo "🔄 Starting Covalent Refiner Upgrade..."

# Configuration
BACKUP_DIR="./backups"
mkdir -p "$BACKUP_DIR"

# Create backup
echo "💾 Creating backup..."
cp .env "$BACKUP_DIR/.env.$(date +%Y%m%d-%H%M%S)"
cp docker-compose.yml "$BACKUP_DIR/docker-compose.yml.$(date +%Y%m%d-%H%M%S)"

# Stop services
echo "⏹️  Stopping services..."
docker-compose down

# Pull latest images
echo "⬇️  Pulling latest images..."
docker-compose pull

# Start services
echo "▶️  Starting services..."
docker-compose up -d

# Wait for startup
echo "⏳ Waiting for services to start..."
sleep 60

# Verify upgrade
echo "✅ Verifying upgrade..."
if curl -s http://localhost:3001/health > /dev/null && \
 curl -s http://localhost:3002/health > /dev/null; then
  echo "🎉 Upgrade successful!"
  docker-compose ps
else
  echo "❌ Upgrade failed! Check logs:"
  docker-compose logs --tail=50
  exit 1
fi

echo "📊 Current resource usage:"
docker stats --no-stream

Troubleshooting Upgrades

Common Issues

Container Restart Loop
Processing Stopped

# Check container logs
docker logs rudder
docker logs evm-server  
docker logs ipfs-pinner

# Common causes:
# 1. Invalid environment variables
env | grep -E "(BLOCK_RESULT|NODE_ETHEREUM|WEB3_JWT)"

# 2. Network connectivity issues
curl -s $NODE_ETHEREUM_MAINNET

# 3. Resource constraints
docker stats
df -h

# Check if operator is still enabled
# Visit operator dashboard

# Verify staking status (need 35k+ CQT staked)

# Check for network issues
ping google.com
curl -s https://moonbase.moonscan.io/

# Restart Rudder if needed
docker-compose restart rudder

Best Practices

Backup First: Always backup configuration before upgrading
Gradual Updates: Update one component at a time if issues occur
Monitor Health: Watch logs and metrics after upgrade
Resource Planning: Upgrades may increase resource requirements
Network Status: Ensure good connectivity to Ethereum mainnet
Operator Status: Keep CQT staking above minimum requirements

Before You Start​

Pre-Upgrade Checklist​

Important Notes​

Upgrade Methods​

Method 1: Docker Compose Upgrade (Recommended)​

Method 2: Manual Docker Upgrade​

Component-Specific Upgrades​

Rudder Upgrade​

EVM Server Upgrade​

IPFS Pinner Upgrade​

Configuration Updates​

Environment Variables​

Docker Compose Updates​

Version-Specific Upgrades​

Upgrading to Mainnet​

Upgrading to v2.0 (Hypothetical Major Version)​

Post-Upgrade Verification​

Health Checks​

Performance Validation​

Rollback Procedure​

Automation Scripts​

Automated Upgrade Script​

Troubleshooting Upgrades​

Common Issues​

Best Practices​

Resources​