Skip to main content

Aztec Troubleshooting Guide

This guide helps you diagnose and resolve common issues with your Aztec node.

Quick Diagnostics​

Before diving into specific issues, run this diagnostic script:

quick-diagnostics.sh
#!/bin/bash
echo "=== Aztec Node Diagnostics ==="
echo ""

# Check if service is running
echo "1. Service Status:"
systemctl is-active aztec-node && echo "✅ Service is running" || echo "❌ Service is not running"
echo ""

# Check disk space
echo "2. Disk Space:"
df -h | grep -E "^/dev|Filesystem"
echo ""

# Check memory
echo "3. Memory Usage:"
free -h
echo ""

# Check RPC connectivity
echo "4. RPC Status:"
curl -s localhost:8545 -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq . 2>/dev/null || echo "❌ Cannot connect to RPC"
echo ""

# Check sync status
echo "5. Sync Status:"
curl -s localhost:8545 -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' | jq . 2>/dev/null || echo "❌ Cannot get sync status"
echo ""

# Check recent errors
echo "6. Recent Errors (last 20):"
journalctl -u aztec-node --no-pager | grep -i error | tail -20

Common Issues and Solutions​

1. Node Won't Start​

Symptoms:

  • Error: listen tcp :8545: bind: address already in use

Solution:

# Find process using the port
sudo lsof -i :8545

# Kill the process (replace PID with actual process ID)
sudo kill -9 PID

# Or change the port in config
sed -i 's/:8545/:8546/g' ~/.aztec/config.toml

2. Sync Issues​

Symptoms:

  • Block height not increasing
  • Node appears frozen

Solution:

# 1. Check current block and network block
CURRENT=$(curl -s localhost:8545 -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq -r .result)
echo "Current block: $CURRENT"

# 2. Check if node is syncing
curl -s localhost:8545 -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' | jq .

# 3. Restart the node
sudo systemctl restart aztec-node

# 4. Check logs for errors
journalctl -u aztec-node -f

3. High Resource Usage​

Symptoms:

  • CPU constantly at 100%
  • Node becomes unresponsive

Solution:

# 1. Check what's consuming CPU
top -p $(pgrep aztec-node)

# 2. Reduce proving threads if running prover
sed -i 's/proving-threads = 8/proving-threads = 4/g' ~/.aztec/config.toml

# 3. Disable unnecessary features
# Edit ~/.aztec/config.toml:
# metrics-enabled = false
# rpc-cors-origins = []

# 4. Restart with resource limits
sudo systemctl edit aztec-node
# Add:
# [Service]
# CPUQuota=80%
# MemoryLimit=8G

4. Network Issues​

Symptoms:

  • Peer count is 0
  • Cannot sync with network

Solution:

# 1. Check firewall settings
sudo ufw status
sudo ufw allow 30303/tcp

# 2. Add bootstrap peers
# Edit ~/.aztec/config.toml:
# bootstrap-peers = [
#   "peer1@ip1:port1",
#   "peer2@ip2:port2"
# ]

# 3. Check network connectivity
ping google.com
nslookup aztec.network

# 4. Restart networking
sudo systemctl restart aztec-node

5. Prover Issues​

Symptoms:

  • proof generation failed errors
  • Low proof generation rate

Solution:

# 1. Check hardware requirements
lscpu | grep -E "Model name|CPU\(s\):"
free -h

# 2. Increase timeout
sed -i 's/proof-timeout = 300/proof-timeout = 600/g' ~/.aztec/config.toml

# 3. Reduce proving load
sed -i 's/proving-threads = 8/proving-threads = 4/g' ~/.aztec/config.toml

# 4. Check GPU support (if available)
nvidia-smi
# Ensure CUDA drivers are installed

Performance Optimization​

System Tuning​

# Increase file descriptors
echo "fs.file-max = 65535" | sudo tee -a /etc/sysctl.conf
echo "$USER soft nofile 65535" | sudo tee -a /etc/security/limits.conf
echo "$USER hard nofile 65535" | sudo tee -a /etc/security/limits.conf

# Optimize network settings
echo "net.core.rmem_default = 262144" | sudo tee -a /etc/sysctl.conf
echo "net.core.rmem_max = 16777216" | sudo tee -a /etc/sysctl.conf
echo "net.core.wmem_default = 262144" | sudo tee -a /etc/sysctl.conf
echo "net.core.wmem_max = 16777216" | sudo tee -a /etc/sysctl.conf

# Apply changes
sudo sysctl -p

Database Optimization​

# Regular database maintenance
sudo systemctl stop aztec-node
aztec-node db compact
sudo systemctl start aztec-node

# Adjust database cache
sed -i 's/db-cache = 256/db-cache = 512/g' ~/.aztec/config.toml

Emergency Procedures​

Complete Reset​

# WARNING: This will delete all local data
sudo systemctl stop aztec-node
rm -rf ~/.aztec/data
aztec-node init --network testnet
sudo systemctl start aztec-node

Backup and Restore​

# Create backup
sudo systemctl stop aztec-node
tar -czf aztec-backup-$(date +%Y%m%d).tar.gz ~/.aztec
sudo systemctl start aztec-node

# Restore from backup
sudo systemctl stop aztec-node
rm -rf ~/.aztec
tar -xzf aztec-backup-YYYYMMDD.tar.gz -C ~/
sudo systemctl start aztec-node

FAQs & Common Issues/Resolutions​

Here is a list of common issues node operators may face. If you don't find your issue here, please check the Aztec Discord server, namely the # operator | faq channel.

"Error getting slot number"​

Issue: Error messages related to slot number retrieval.

Resolution:

  • If it is regarding a beacon call, it has failed the beacon RPC call
  • If it is regarding the execution endpoint, then it is likely just reporting and not critical
# Check your beacon chain RPC endpoint
curl -s $L1_CONSENSUS_HOST_URLS/eth/v1/beacon/headers/head

# Verify execution endpoint connectivity  
curl -s $ETHEREUM_HOSTS -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Update Aztec Alpha-Testnet Version​

Issue: Need to update to the latest Aztec version.

Resolution:

# Update to latest alpha-testnet version
aztec-up alpha-testnet

# Restart your node
docker-compose restart

# Or if using systemd
sudo systemctl restart aztec-node

"rpc rate", "quota limit"​

Issue: Rate limiting errors from RPC providers.

Resolution:

  • Register with your RPC URL provider to get a token that may permit more requests
  • Consider using multiple RPC endpoints for load balancing
  • Switch to a higher-tier plan with your RPC provider
# Example: Using authenticated endpoint
export ETHEREUM_HOSTS="https://eth-mainnet.g.alchemy.com/v2/YOUR_API_KEY"

# Or use multiple endpoints
export ETHEREUM_HOSTS="https://rpc1.example.com,https://rpc2.example.com"

"No blob bodies found", "Unable to get blob sidecar, Gateway Time-out (504)"​

Issue: Blob sidecar retrieval failures.

Resolution:

  • Check L1_CONSENSUS_HOST_URLS (for the beacon chain)
  • If you see this regularly, it's likely also a rate/limit issue
# Test beacon chain connectivity
curl -s $L1_CONSENSUS_HOST_URLS/eth/v1/beacon/blob_sidecars/latest

# Check if rate limiting is the issue
curl -I $L1_CONSENSUS_HOST_URLS/eth/v1/beacon/headers/head

"Insufficient L1 funds"​

Issue: Not enough funds for L1 transactions.

Resolution:

  • EOA needs Sepolia ETH for testnet operations
  • Use a faucet to get testnet funds
# Check your balance
cast balance $YOUR_ADDRESS --rpc-url https://sepolia.infura.io/v3/YOUR_KEY

# Get testnet funds from faucets:
# - https://sepoliafaucet.com/
# - https://faucets.chain.link/sepolia

"CodeError: stream reset"​

Issue: Occasional stream reset errors in logs.

Resolution:

  • This error can be seen occasionally in logs
  • Action: Ignore - this is not critical and doesn't require intervention

"SYNC_BLOCK failed"​

Issue: Block synchronization failures with world state mismatch.

Error: ERROR: world-state:database Call SYNC_BLOCK failed: Error: Can't synch block: block state does not match world state

Resolution:

# 1. Stop Aztec
docker-compose down
# Or: sudo systemctl stop aztec-node

# 2. Delete current snapshot
rm -rf ~/.aztec/0.87.8/data/archiver

# 3. Update to latest version
aztec-up -v latest

# 4. Start Aztec
docker-compose up -d
# Or: sudo systemctl start aztec-node

Additional Common Issues​

"Failed to connect to P2P network"​

Resolution:

# Check if P2P ports are open
sudo ufw allow 40400/tcp
sudo ufw allow 40400/udp

# Test P2P connectivity
telnet your-public-ip 40400

"Database corruption detected"​

Resolution:

# Stop the node
docker-compose down

# Backup current data (optional)
cp -r ~/.aztec/data ~/.aztec/data.backup

# Remove corrupted database
rm -rf ~/.aztec/data/archiver

# Restart and resync
docker-compose up -d

"Prover agent not connecting to broker"​

Resolution:

# Check broker connectivity
docker exec aztec-prover-agent-1 curl -s http://broker:8080/health

# Verify environment variables
docker exec aztec-prover-agent-1 env | grep PROVER

# Check network connectivity between containers
docker network ls
docker network inspect aztec-prover_default

Getting Help​

If you're still experiencing issues:

  1. Check the logs: journalctl -u aztec-node -f or docker-compose logs -f
  2. Visit the documentation: Aztec Docs
  3. Join the Discord: Aztec Discord - Check # operator | faq channel
  4. Search GitHub issues: Aztec Issues

When asking for help, always include:

  • Your OS and version
  • Node version: aztec-node --version or Docker image tag
  • Error logs from the last 50 lines: journalctl -u aztec-node -n 50 or docker-compose logs --tail=50
  • Your config file (remove any private keys)
  • Network you're running (mainnet/testnet)