Server Monitoring and Alerting: Setting Up Prometheus and Grafana

Monitoring is the backbone of reliable infrastructure. Without proper observability, you're flying blind. Let's build a production-ready monitoring stack from scratch.

Architecture Components

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Linux      │────▶│  Node       │────▶│ Prometheus  │
│  Server     │     │  Exporter   │     │   Server    │
└─────────────┘     └─────────────┘     └─────────────┘
                                                   │
                                                   ▼
                                            ┌─────────────┐
                                            │   Grafana   │
                                            │   Dashboard │
                                            └─────────────┘
                                                   │
                                                   ▼
                                            ┌─────────────┐
                                            │ Alertmanager│
                                            └─────────────┘

Installation Steps

1. Install Node Exporters on Target Servers

wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

Create systemd service:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

2. Configure Prometheus

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['server1:9100', 'server2:9100', 'server3:9100']

3. Set Up Alertmanager Rules

groups:
  - name: server_alerts
    rules:
      - alert: InstanceDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"

Key Metrics to Track

System Resources

• CPU usage (user/system/iowait)
• Memory utilization and swap
• Disk I/O and space
• Network throughput and errors

Application Metrics (Custom)

• Request rate and latency
• Error rates
• Saturation (queue depth, concurrent connections)

Grafana Dashboard Templates

Import community dashboards:

• Node Exporter Full (1860)
• Linux Server Overview (11077)
• Prometheus 2.0 Stats (3662)

Or build your own with these key panels:

• CPU/Memory/Network graphs (time series)
• Status overview (single stat with thresholds)
• Top processes table
• Disk usage breakdown

Alert Routing

Configure Alertmanager with:

• Routing trees by severity
• Inhibition rules (suppress duplicate alerts)
• Multiple receivers (Slack, email, PagerDuty)
• Grouping to avoid alert storms

Conclusion

A well-configured monitoring system pays for itself the moment something goes wrong. Set it up before you need it.