Stader Troubleshooting Guide
This guide helps you diagnose and resolve common issues with your Stader node.
Quick Diagnostics
Before diving into specific issues, run this diagnostic script:
quick-diagnostics.sh
#!/bin/bash
echo "=== Stader Node Diagnostics ==="
echo ""
# Check if service is running
echo "1. Service Status:"
systemctl is-active staderd && echo "✅ Service is running" || echo "❌ Service is not running"
echo ""
# Check disk space
echo "2. Disk Space:"
df -h | grep -E "^/dev|Filesystem"
echo ""
# Check memory
echo "3. Memory Usage:"
free -h
echo ""
# Check peers
echo "4. Connected Peers:"
curl -s localhost:26657/net_info | jq .result.n_peers 2>/dev/null || echo "❌ Cannot connect to RPC"
echo ""
# Check sync status
echo "5. Sync Status:"
curl -s localhost:26657/status | jq .result.sync_info.catching_up 2>/dev/null || echo "❌ Cannot get sync status"
echo ""
# Check recent errors
echo "6. Recent Errors (last 20):"
journalctl -u staderd --no-pager | grep -i error | tail -20
Common Issues and Solutions
1. Node Won't Start
- Port Already in Use
- Permission Denied
- Missing Configuration
Symptoms:
- Error:
listen tcp :26656: bind: address already in use
Solution:
# Find process using the port
sudo lsof -i :26656
# Kill the process (replace PID with actual process ID)
sudo kill -9 PID
# Or change the port in config
sed -i 's/:26656/:26756/g' ~/.stader/config/config.toml
Symptoms:
- Error:
permission denied
- Cannot write to data directory
Solution:
# Fix ownership
sudo chown -R $USER:$USER ~/.stader
# Fix permissions
chmod -R 755 ~/.stader
# If using systemd, ensure correct user in service file
sudo sed -i "s/User=.*/User=$USER/" /etc/systemd/system/staderd.service
sudo systemctl daemon-reload
Symptoms:
- Error:
config file not found
no such file or directory
Solution:
# Reinitialize the node
staderd init my-node --chain-id staderchain-1
# Download chain configuration
wget -O ~/.stader/config/genesis.json https://raw.githubusercontent.com/stader-space/networks/main/mainnet/genesis.json
# Restore config from backup if available
cp ~/.stader/config/config.toml.backup ~/.stader/config/config.toml
2. Sync Issues
- Stuck Syncing
- Very Slow Sync
Symptoms:
- Block height not increasing
catching_up: true
for extended period
Solution:
# 1. Check if you have peers
curl -s localhost:26657/net_info | jq .result.n_peers
# 2. If no peers, add seeds/peers
SEEDS="6a3f7c6a23fd43e5f68c39e73e61129754e32bb1@seed-1.staderchain.staderspace.com:26656"
PEERS="$(curl -s https://raw.githubusercontent.com/stader-space/networks/main/mainnet/peers.txt | tr '
' ',')"
sed -i "s/seeds = \"\"/seeds = \"$SEEDS\"/g" ~/.stader/config/config.toml
sed -i "s/persistent_peers = \"\"/persistent_peers = \"$PEERS\"/g" ~/.stader/config/config.toml
# 3. Restart the node
sudo systemctl restart staderd
# 4. If still stuck, try state sync or snapshot restore
Symptoms:
- Syncing but extremely slow
- High CPU/disk usage
Solution:
# Option 1: Use state sync
staderd tendermint unsafe-reset-all
# Configure state sync in ~/.stader/config/config.toml
# [statesync]
# enable = true
# rpc_servers = "https://rpc.staderchain.staderspace.com:443,https://rpc2.staderchain.staderspace.com:443"
# Option 2: Download snapshot (if available)
sudo systemctl stop staderd
cd ~/.stader
wget -O snapshot.tar.lz4 https://snapshots.stader.space/staderchain-1_latest.tar.lz4
lz4 -c -d snapshot.tar.lz4 | tar -x -C ~/.stader
sudo systemctl start staderd
# Option 3: Increase resources
sed -i 's/max_num_inbound_peers = 40/max_num_inbound_peers = 100/g' ~/.stader/config/config.toml
sed -i 's/max_num_outbound_peers = 10/max_num_outbound_peers = 50/g' ~/.stader/config/config.toml
3. Validator Issues
- Validator Jailed
- Double Signing Prevention
Symptoms:
- Node shows as jailed
- Missing blocks, downtime
Solution:
# 1. Check validator status
staderd query staking validator $(staderd keys show wallet --bech val -a)
# 2. Check why jailed
staderd query slashing signing-info $(staderd tendermint show-validator)
# 3. Ensure node is synced and healthy
curl -s localhost:26657/status | jq .result.sync_info
# 4. Wait for unjailing period (usually 10 minutes after fixing issues)
# Then unjail:
staderd tx slashing unjail --from wallet --chain-id staderchain-1 --gas-prices 0.025usd --gas auto --gas-adjustment 1.5
# 5. Monitor signing status
watch -n 5 'staderd query slashing signing-info $(staderd tendermint show-validator)'
Symptoms:
- Risk of double signing
- Running multiple validators
Solution:
# 1. IMMEDIATELY stop all duplicate validators
sudo systemctl stop staderd
# 2. Ensure only one validator is running
ps aux | grep staderd
# 3. Check priv_validator_key.json is unique
cat ~/.stader/config/priv_validator_key.json
# 4. Implement proper key management
# - Use remote signers (TMKMS) for production
# - Never copy validator keys to multiple machines
# 5. Start only the primary validator
sudo systemctl start staderd
4. High Resource Usage
- High CPU Usage
- High Memory Usage
Symptoms:
- CPU constantly at 100%
- Node becomes unresponsive
Solution:
# 1. Check what's consuming CPU
top -p $(pgrep staderd)
# 2. Reduce mempool size
sed -i 's/size = 5000/size = 1000/g' ~/.stader/config/config.toml
# 3. Disable unnecessary features
# In ~/.stader/config/config.toml:
# index_all_keys = false
# prometheus = false
# 4. Enable pruning
# In ~/.stader/config/app.toml:
# pruning = "default"
# pruning-keep-recent = "100"
# pruning-interval = "10"
# 5. Restart with limits
sudo systemctl edit staderd
# Add:
# [Service]
# CPUQuota=80%
Symptoms:
- RAM usage continuously growing
- Out of memory errors
Solution:
# 1. Check memory usage
free -h
ps aux | grep staderd
# 2. Reduce cache sizes
# In ~/.stader/config/app.toml:
# cache = 10000
# 3. Enable aggressive pruning
# pruning = "custom"
# pruning-keep-recent = "100"
# pruning-keep-every = "0"
# pruning-interval = "10"
# 4. Add memory limits
sudo systemctl edit staderd
# Add:
# [Service]
# MemoryLimit=4G
# MemoryAccounting=yes
5. Network Issues
- No Peers Connected
Symptoms:
- Peer count is 0
- Cannot sync with network
Solution:
# 1. Check firewall settings
sudo ufw status
sudo ufw allow 26656/tcp
# 2. Add seed nodes
SEEDS="6a3f7c6a23fd43e5f68c39e73e61129754e32bb1@seed-1.staderchain.staderspace.com:26656"
sed -i "s/seeds = \"\"/seeds = \"$SEEDS\"/g" ~/.stader/config/config.toml
# 3. Clear peer store
sudo systemctl stop staderd
rm -rf ~/.stader/data/peerstore.db
sudo systemctl start staderd
# 4. Check network connectivity
ping seed-1.staderchain.staderspace.com
6. Staking Issues
- Delegation Failed
Symptoms:
- Cannot delegate tokens
- Transaction failures
Solution:
# 1. Check account balance
staderd query bank balances $(staderd keys show wallet -a)
# 2. Check gas settings
staderd tx staking delegate $(staderd keys show wallet --bech val -a) 1000000usd \
--from wallet --chain-id staderchain-1 \
--gas-prices 0.025usd --gas auto --gas-adjustment 1.5
# 3. Check validator status
staderd query staking validator $(staderd keys show wallet --bech val -a)
# 4. Ensure proper denomination (usd for Stader)
Performance Optimization
System Tuning
# Increase file descriptors
echo "fs.file-max = 65535" | sudo tee -a /etc/sysctl.conf
ulimit -n 65535
# Network optimization
echo "net.core.rmem_default = 262144" | sudo tee -a /etc/sysctl.conf
echo "net.core.rmem_max = 16777216" | sudo tee -a /etc/sysctl.conf
# Apply changes
sudo sysctl -p
Database Optimization
# Use faster database backend
# In ~/.stader/config/config.toml:
# db_backend = "pebbledb"
# Enable pruning to save disk space
# In ~/.stader/config/app.toml:
# pruning = "default"
# pruning-keep-recent = "100"
# pruning-interval = "10"
Emergency Procedures
Complete Reset
# WARNING: This will delete all local data
sudo systemctl stop staderd
staderd tendermint unsafe-reset-all
sudo systemctl start staderd
Validator Key Backup
# CRITICAL: Backup validator keys
cp ~/.stader/config/priv_validator_key.json ~/stader-validator-key-backup.json
cp ~/.stader/config/node_key.json ~/stader-node-key-backup.json
# Store these files securely and never share them
Backup and Restore
# Create backup
sudo systemctl stop staderd
tar -czf stader-backup-$(date +%Y%m%d).tar.gz ~/.stader
sudo systemctl start staderd
# Restore from backup
sudo systemctl stop staderd
rm -rf ~/.stader
tar -xzf stader-backup-YYYYMMDD.tar.gz -C ~/
sudo systemctl start staderd
Getting Help
If you're still experiencing issues:
- Check the logs:
journalctl -u staderd -f
- Visit the documentation: Stader Docs
- Join the Discord: Stader Discord
- Search GitHub issues: Stader Issues
When asking for help, always include:
- Your OS and version
- Node version:
staderd version
- Error logs from the last 50 lines:
journalctl -u staderd -n 50
- Your node status:
curl -s localhost:26657/status | jq .result.sync_info
- Validator status (if applicable):
staderd query staking validator $(staderd keys show wallet --bech val -a)