Complete Guide: Hardening Linux Servers for Production

🚨 Introduction: Critical Environment Variables

Before starting, configure these environment variables to customize your deployment:

# Create variables file in /etc/server-config.env
sudo tee /etc/server-config.env > /dev/null << EOF
# =============================================================================
# SERVER CONFIGURATION - ENVIRONMENT VARIABLES
# =============================================================================

# SERVER IDENTIFICATION
export SERVER_NAME="prod-web-01"
export ENVIRONMENT="production"
export DOMAIN="yourdomain.com"

# NETWORK CONFIGURATION
export SSH_PORT="5822"
export SERVER_TIMEZONE="America/Argentina/Buenos_Aires"

# USERS AND ACCESS
export DEPLOY_USER="deploy"
export ADMIN_EMAIL="admin@yourdomain.com"

# DATABASE
export DB_NAME="app_production"
export DB_USER="app_user"
export DB_HOST="localhost"
export DB_PORT="5432"

# MONITORING AND LOGGING
export PROMETHEUS_PORT="9090"
export GRAFANA_PORT="3000"
export LOG_RETENTION_DAYS="30"

# BACKUP AND RETENTION
export BACKUP_DIR="/backups"
export BACKUP_RETENTION_DAYS="7"
export AWS_S3_BUCKET="backups-yourdomain"  # Optional for cloud

# SECURITY
export FAIL2BAN_BANTIME="86400"
export FAIL2BAN_MAXRETRY="3"
export UFW_DEFAULT_DENY="true"
EOF

# Load variables into current session
source /etc/server-config.env

🔧 1. Initial System Configuration

1.1 System Update Using Variables

# Load configuration
source /etc/server-config.env

# Set timezone using variable
sudo timedatectl set-timezone $SERVER_TIMEZONE

# Update system
sudo apt update && sudo apt upgrade -y

# Verify configuration
echo "=== Server Configuration ==="
echo "Server: $SERVER_NAME"
echo "Environment: $ENVIRONMENT"
echo "Timezone: $SERVER_TIMEZONE"
echo "Deploy User: $DEPLOY_USER"

1.2 User Creation with Variables

# Create deploy user from variable
sudo adduser $DEPLOY_USER
sudo usermod -aG sudo $DEPLOY_USER

# Configure secure SSH directory
sudo mkdir -p /home/$DEPLOY_USER/.ssh
sudo chmod 700 /home/$DEPLOY_USER/.ssh

🛡️ 2. Security Hardening with Flexible Configuration

2.1 Dynamic SSH Configuration

# Backup original configuration
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

# Apply dynamic configuration
sudo tee -a /etc/ssh/sshd_config > /dev/null << EOF
# =============================================================================
# AUTOMATED SSH CONFIGURATION - $(date)
# =============================================================================

# Port from environment variable
Port $SSH_PORT

# Basic security
Protocol 2
PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no

# Limit access
AllowUsers $DEPLOY_USER
MaxAuthTries 3
MaxSessions 5

# Modern cryptography
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
KexAlgorithms curve25519-sha256@libssh.org
MACs hmac-sha2-512-etm@openssh.com
EOF

# Reload SSH service
sudo systemctl reload ssh

2.2 Firewall Configuration with Variables

# Configure UFW dynamically
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Rules from variables
sudo ufw allow $SSH_PORT/tcp comment 'Custom SSH'
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'

# Rate limiting using variables
sudo ufw limit $SSH_PORT/tcp

# Enable firewall
sudo ufw enable

📊 3. Monitoring and Metrics with Configurable Variables

3.1 Node Exporter with Dynamic Configuration

# Create user for monitoring
sudo useradd --no-create-home --shell /bin/false node_exporter

# Download and configure Node Exporter
cd /tmp
NODE_EXPORTER_VERSION="1.6.1"
wget https://github.com/prometheus/node_exporter/releases/download/v$NODE_EXPORTER_VERSION/node_exporter-$NODE_EXPORTER_VERSION.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

# Create systemd service with variables
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << EOF
[Unit]
Description=Node Exporter $NODE_EXPORTER_VERSION
Documentation=https://prometheus.io/docs/guides/node-exporter/
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:$PROMETHEUS_PORT \
  --collector.systemd \
  --collector.logind \
  --collector.processes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

🔍 4. Environment Variables for Applications

4.1 Application Configuration File

# Create application-specific variables
sudo tee /etc/app-environment.env > /dev/null << EOF
# =============================================================================
# APPLICATION CONFIGURATION - DO NOT COMMIT TO REPOSITORY
# =============================================================================

# Database
export DATABASE_URL="postgresql://$DB_USER:@$DB_HOST:$DB_PORT/$DB_NAME"
export DB_POOL_SIZE="20"
export DB_TIMEOUT="30"

# Redis Cache
export REDIS_URL="redis://localhost:6379"
export REDIS_PASSWORD=""

# API Keys and Secrets
export JWT_SECRET="$(openssl rand -hex 32)"
export ENCRYPTION_KEY="$(openssl rand -hex 16)"

# External Services
export SMTP_HOST="smtp.gmail.com"
export SMTP_PORT="587"
export SMTP_USER="noreply@$DOMAIN"

# Application Features
export LOG_LEVEL="info"
export NODE_ENV="production"
export PORT="3000"
EOF

# Protect configuration file
sudo chmod 600 /etc/app-environment.env
sudo chown $DEPLOY_USER:$DEPLOY_USER /etc/app-environment.env

🚀 5. Strategic Next Steps

Phase 1: Immediate (First Week)

# 1. Implement automated backup system
sudo tee /usr/local/bin/backup-script.sh > /dev/null << 'EOF'
#!/bin/bash
source /etc/server-config.env

BACKUP_FILE="backup-$(date +%Y%m%d-%H%M%S).tar.gz"
LOG_FILE="/var/log/backups.log"

echo "$(date): Starting backup" >> $LOG_FILE

# Database backup
sudo -u postgres pg_dumpall | gzip > $BACKUP_DIR/db-$BACKUP_FILE

# Configuration backup
tar -czf $BACKUP_DIR/config-$BACKUP_FILE /etc

# Rotation
find $BACKUP_DIR -name "*.tar.gz" -mtime +$BACKUP_RETENTION_DAYS -delete

echo "$(date): Backup completed: $BACKUP_FILE" >> $LOG_FILE
EOF

sudo chmod +x /usr/local/bin/backup-script.sh

Phase 2: Short Term (2–4 Weeks)

Phase 3: Medium Term (1–3 Months)

# docker-compose.prod.yml - Example of future orchestration
version: '3.8'
services:
  app:
    image: myapp:${APP_VERSION}
    env_file: /etc/app-environment.env
    ports:
      - "3000:3000"
    depends_on:
      - postgres
      - redis
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl

📋 Verification Checklist with Variables

Automated Verification Script

#!/bin/bash
# verify-deployment.sh

# Load configuration
source /etc/server-config.env

echo "=== DEPLOYMENT VERIFICATION ==="
echo "Server: $SERVER_NAME"
echo "Environment: $ENVIRONMENT"
echo ""

checks=(
    "Environment variables:test -f /etc/server-config.env && echo '✅' || echo '❌'"
    "Deploy user:id $DEPLOY_USER >/dev/null 2>&1 && echo '✅' || echo '❌'"
    "SSH port:sshd -T | grep -q 'port $SSH_PORT' && echo '✅' || echo '❌'"
    "Firewall:sudo ufw status | grep -q 'Status: active' && echo '✅' || echo '❌'"
    "Node Exporter:curl -s http://localhost:$PROMETHEUS_PORT/metrics >/dev/null && echo '✅' || echo '❌'"
    "Backups:test -f /usr/local/bin/backup-script.sh && echo '✅' || echo '❌'"
    "Logging:sudo systemctl is-active rsyslog | grep -q 'active' && echo '✅' || echo '❌'"
    "NTP:timedatectl status | grep -q 'synchronized: yes' && echo '✅' || echo '❌'"
)

for check in "${checks[@]}"; do
    IFS=':' read -r description command <<< "$check"
    result=$(eval "$command")
    printf "%-25s %s\n" "$description" "$result"
done

echo ""
echo "=== RECOMMENDED NEXT STEPS ==="
echo "1. Configure cloud backups: $AWS_S3_BUCKET"
echo "2. Implement advanced monitoring on port: $GRAFANA_PORT"
echo "3. Configure alerts for: $ADMIN_EMAIL"
echo "4. Schedule weekly security audit"

🔮 Server Evolution Strategy

Target Architecture at 6 Months

Technologies to Evaluate

📞 Incident Response Plan

Emergency Checklist

#!/bin/bash
# emergency-checklist.sh

source /etc/server-config.env

echo "🚨 EMERGENCY CHECKLIST - $SERVER_NAME"
echo "Time: $(date)"
echo ""

echo "1. CHECK SERVER STATUS"
uptime
free -h
df -h

echo ""
echo "2. CHECK CRITICAL SERVICES"
systemctl is-active nginx postgresql redis

echo ""
echo "3. REVIEW RECENT LOGS"
journalctl --since "10 minutes ago" -p err

echo ""
echo "4. ACTIVE CONNECTIONS"
ss -tulpn | grep -E "($SSH_PORT|80|443|5432)"

echo ""
echo "5. EMERGENCY CONTACTS"
echo "Admin: $ADMIN_EMAIL"
echo "Backup Contact: backup-admin@$DOMAIN"

✅ Conclusion and Continuous Maintenance

Summary of Critical Variables

# Command to verify current configuration
echo "=== CONFIGURATION SUMMARY ==="
echo "🔧 Server: $SERVER_NAME"
echo "🌍 Domain: $DOMAIN"
echo "🔐 SSH Port: $SSH_PORT"
echo "📊 Monitoring: $PROMETHEUS_PORT"
echo "💾 Backups: $BACKUP_DIR"
echo "🛡️ Log Retention: $LOG_RETENTION_DAYS days"

Recommended Scheduled Maintenance

# /etc/cron.d/server-maintenance
0 2 * * * root /usr/local/bin/backup-script.sh
0 3 * * 1 root apt update && apt upgrade -y
0 4 * * * root /usr/local/bin/security-scan.sh
0 5 * * * root find /var/log -name "*.log" -mtime +$LOG_RETENTION_DAYS -delete

Immediate Next Steps

1. Run the verification script.
2. Configure monitoring alerts.
3. Schedule the first security audit.
4. Document recovery procedures.

Ready for the next level? Start with Phase 1 and adapt the variables and configurations to your specific use case.