Skip to main content

How to Run a Prover Node

Background​

Prover nodes are a critical part of the Aztec network's infrastructure. They generate cryptographic proofs that attest to the correctness of public transactions, ultimately producing a single rollup proof that is submitted to Ethereum.

Operating a prover node requires a solid grasp of blockchain protocols, cryptographic systems, DevOps best practices, and high-performance hardware. It's a resource-intensive role typically undertaken by experienced engineers or specialized teams due to its technical and operational complexity.

Prerequisites​

Before following this guide, make sure you:

  • Have the aztec tool installed
  • Have sufficient hardware resources for proving operations
  • Your confidence level is expected to be around "I'd be able to run a Prover without this guide"

Understanding Prover Architecture​

The Aztec prover involves three key components: the Prover Node, the Proving Broker, and the Proving Agent.

Prover Node​

The Prover Node is responsible for polling the L1 for unproven epochs and initiating the proof process. When an epoch is ready to be proven, the prover node creates proving jobs and distributes them to the broker. The Prover Node is also responsible for submitting the final rollup proof to the rollup contract.

Resources: Up to 8 cores, 16GB RAM, ~1TB disk for storing state.

Proving Broker​

Manages a queue of proving jobs, distributing them to available agents and forwarding results back to the node.

Resources: Up to 4 cores, 16GB RAM, ~1GB disk.

Proving Agents​

Executes the actual proof jobs. Agents are stateless, fetch work from the broker, and return the results.

Resources: Each agent may use up to 16 cores and 128GB RAM.

Setting Up Your Prover​

Using Docker Compose​

name: aztec-prover
services:
  prover-node:
    image: aztecprotocol/aztec:latest # Always refer to the docs to check that you're using the correct image.
    command:
      - node
      - --no-warnings
      - /usr/src/yarn-project/aztec/dest/bin/index.js
      - start
      - --prover-node
      - --archiver
      - --network
      - alpha-testnet
    depends_on:
      broker:
        condition: service_started
        required: true
    environment:
      # PROVER_COORDINATION_NODE_URL: "http://:8080" # this can point to your own validator - using this replaces the need for the prover node to be on the P2P network and uses your validator as a sentry node of some sort.
      # P2P_ENABLED: "false" # Switch to false if you provide a PROVER_COORDINATION_NODE_URL
      DATA_DIRECTORY: /data
      DATA_STORE_MAP_SIZE_KB: "134217728"
      ETHEREUM_HOSTS: # EL RPC endpoint
      L1_CONSENSUS_HOST_URLS: # CL RPC endpoint
      LOG_LEVEL: info
      PROVER_BROKER_HOST: http://broker:8080
      PROVER_PUBLISHER_PRIVATE_KEY: # The node needs to publish proofs to L1. Replace with your private key
    ports:
      - "8080:8080"
      - "40400:40400"
      - "40400:40400/udp"
    volumes:
      - /home/my-node/node:/data # Local directory

  agent:
    image: aztecprotocol/aztec:latest # Always refer to the docs to check that you're using the correct image.
    command:
      - node
      - --no-warnings
      - /usr/src/yarn-project/aztec/dest/bin/index.js
      - start
      - --prover-agent
      - --network
      - alpha-testnet
    environment:
      PROVER_AGENT_COUNT: "1"
      PROVER_AGENT_POLL_INTERVAL_MS: "10000" # Just to reduce the log spamming if you're using debug logging.
      PROVER_BROKER_HOST: http://broker:8080
      PROVER_ID: # this should be the address corresponding to the PROVER_PUBLISHER_PRIVATE_KEY you set on the node.
    pull_policy: always
    restart: unless-stopped

  broker:
    image: aztecprotocol/aztec:latest # Always refer to the docs to check that you're using the correct image.
    command:
      - node
      - --no-warnings
      - /usr/src/yarn-project/aztec/dest/bin/index.js
      - start
      - --prover-broker
      - --network
      - alpha-testnet
    environment:
      DATA_DIRECTORY: /data
      ETHEREUM_HOSTS: # Your EL RPC endpoint
      LOG_LEVEL: info
    volumes:
      - /home/my-node/node:/data # Local directory

Configuration​

Environment Variables​

Key environment variables for configuration:

  • ETHEREUM_HOSTS: Ethereum Layer 1 RPC endpoint
  • L1_CONSENSUS_HOST_URLS: Consensus Layer RPC endpoint
  • PROVER_PUBLISHER_PRIVATE_KEY: Private key for publishing proofs to L1
  • PROVER_BROKER_HOST: URL of the proving broker
  • PROVER_ID: Unique identifier for the prover
  • DATA_DIRECTORY: Directory for storing prover data
  • LOG_LEVEL: Logging level (info, debug, warn, error)

Hardware Optimization​

CPU Optimization​

# Set CPU governor to performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Increase worker processes based on CPU cores
export PROVER_AGENT_COUNT=4  # Adjust based on available cores

Memory Configuration​

# Increase system limits for high memory usage
sudo tee -a /etc/sysctl.conf << EOF
vm.max_map_count = 655300
fs.file-max = 2097152
vm.swappiness = 1
EOF

sudo sysctl -p

Monitoring​

Performance Metrics​

Monitor your prover performance using these commands:

# Check prover node status
docker logs aztec-prover-prover-node-1

# Monitor proving agent activity
docker logs aztec-prover-agent-1

# Check broker queue status
docker logs aztec-prover-broker-1

# Monitor system resources
htop

Key Metrics to Monitor​

  • Proof completion rate: Number of proofs completed per hour
  • CPU utilization: Should be high during proving operations
  • Memory usage: Monitor for memory leaks or excessive usage
  • Network connectivity: Ensure stable connection to Ethereum L1
  • Disk I/O: Monitor for storage bottlenecks

Health Checks​

# Check if services are running
docker ps

# Verify network connectivity
curl -s http://localhost:8080/health || echo "Prover node not responding"

# Check Ethereum connection
docker exec aztec-prover-prover-node-1 wget -q --spider $ETHEREUM_HOSTS || echo "L1 connection failed"

Starting the Prover​

1. Save Docker Compose Configuration​

Save the above Docker Compose configuration to a file called docker-compose.yml.

2. Configure Environment Variables​

# Set your Ethereum RPC endpoints
export ETHEREUM_HOSTS="https://your-ethereum-rpc-endpoint"
export L1_CONSENSUS_HOST_URLS="https://your-consensus-rpc-endpoint"

# Set your prover private key (keep this secure!)
export PROVER_PUBLISHER_PRIVATE_KEY="your-private-key-here"

# Set your prover ID (derived from the private key)
export PROVER_ID="your-prover-address"

3. Start the Prover​

# Start all prover components
docker-compose up -d

# Check if all services started successfully
docker-compose ps

# View logs
docker-compose logs -f

4. Scaling Proving Agents​

To run multiple proving agents (recommended for high-performance systems):

# Scale the agent service to run 4 instances
docker-compose up -d --scale agent=4

Troubleshooting​

Common Issues​

  1. Prover Node Won't Start

    # Check Docker logs for errors
    docker-compose logs prover-node

    # Verify environment variables are set
    echo $ETHEREUM_HOSTS
    echo $PROVER_PUBLISHER_PRIVATE_KEY

    # Check Ethereum connectivity
    curl -s $ETHEREUM_HOSTS -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

  2. High Memory Usage

    # Monitor memory usage
    docker stats

    # Reduce number of proving agents if needed
    docker-compose up -d --scale agent=2

    # Check for memory leaks in logs
    docker-compose logs | grep -i "memory\|oom"

  3. Proving Jobs Not Processing

    # Check broker connectivity
    docker exec aztec-prover-agent-1 curl -s http://broker:8080/health

    # Verify agent registration
    docker-compose logs agent | grep -i "registered\|connected"

    # Check for network issues
    docker-compose logs | grep -i "connection\|network\|timeout"

Performance Issues​

  1. Slow Proof Generation

    • Ensure CPU governor is set to "performance"
    • Check for CPU thermal throttling
    • Verify sufficient RAM is available
    • Consider scaling up proving agents

  2. Connection Issues

    • Verify firewall rules allow required ports
    • Check network stability to Ethereum L1
    • Ensure Docker containers can communicate
    • Verify DNS resolution works properly

Best Practices​

1. Security​

  • Private Key Management: Store your PROVER_PUBLISHER_PRIVATE_KEY securely
  • Network Security: Use firewall rules to restrict access to necessary ports only
  • Regular Updates: Keep Docker images updated to the latest version
  • Monitoring: Set up alerting for service failures or performance issues

2. Resource Management​

  • Memory Allocation: Ensure sufficient RAM for all proving agents (128GB per agent)
  • CPU Allocation: Use dedicated CPU cores for proving operations
  • Storage: Use fast NVMe SSDs for better I/O performance
  • Network: Ensure stable, low-latency connection to Ethereum L1

3. Operational Procedures​

# Regular maintenance script
cat > $HOME/prover-maintenance.sh << EOF
#!/bin/bash
# Update Docker images
docker-compose pull

# Restart services with new images
docker-compose up -d

# Clean up old images
docker image prune -f

# Check service health
docker-compose ps
EOF

chmod +x $HOME/prover-maintenance.sh

4. Backup and Recovery​

  • Configuration Backup: Regularly backup your docker-compose.yml and environment variables
  • Data Backup: Backup the prover data directory periodically
  • Recovery Testing: Test your backup and recovery procedures regularly
  • Documentation: Document your setup and configuration for team members

Resources​