Prometheus HA 2020(3)AlertManager Cluster

Prometheus HA 2020(3)AlertManager Cluster

Install AlertManager
> wget https://github.com/prometheus/alertmanager/releases/download/v0.19.0/alertmanager-0.19.0.linux-amd64.tar.gz
> tar zxvf alertmanager-0.19.0.linux-amd64.tar.gz
> mv alertmanager-0.19.0.linux-amd64 ~/tool/alertmanager-0.19.0
> sudo ln -s /home/carl/tool/alertmanager-0.19.0 /opt/alertmanager-0.19.0
> sudo ln -s /opt/alertmanager-0.19.0 /opt/alertmanager
> cd /opt/alertmanager
> mkdir data

Webhook Example
https://github.com/prometheus/alertmanager/tree/master/examples/webhook

HA Setting example
https://github.com/prometheus/alertmanager/tree/master/examples/ha

Or I can try from here
https://requestbin.com/r/en618j5nslf7s
https://en618j5nslf7s.x.pipedream.net/

Check the alert manager configuration
> cat alertmanager.yml
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'https://en618j5nslf7s.x.pipedream.net/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Start the alert manager as follow:
> ./alertmanager  --web.listen-address="ubuntu-worker1:9092" --cluster.listen-address="ubuntu-worker1:8001" --config.file="/opt/alertmanager/alertmanager.yml" --log.level=debug

> ./alertmanager  --web.listen-address="ubuntu-worker2:9092" --cluster.listen-address="ubuntu-worker2:8001" --cluster.peer="ubuntu-worker1:8001" --config.file="/opt/alertmanager/alertmanager.yml" --log.level=debug

It is not working. I guess I can not use the name, I need use IP address
> ./alertmanager  --web.listen-address=“192.168.56.102:9092" --cluster.listen-address=“192.168.56.102:8001" --config.file="/opt/alertmanager/alertmanager.yml" --log.level=debug

> ./alertmanager  --web.listen-address=“192.168.56.103:9092" --cluster.listen-address=“192.168.56.103:8001" --cluster.peer=“192.168.56.102:8001" --config.file="/opt/alertmanager/alertmanager.yml" --log.level=debug

I think it is working fine now.
level=debug ts=2020-02-23T17:31:33.251Z caller=cluster.go:306 component=cluster memberlist="2020/02/23 17:31:33 [DEBUG] memberlist: Stream connection from=192.168.56.103:55446\n"
level=debug ts=2020-02-23T17:31:33.253Z caller=delegate.go:230 component=cluster received=NotifyJoin node=01E1SHNBFSS448B1S8GZT7EVV4 addr=192.168.56.103:8001

Write the simple Script to send alerts
> cat send_alert.sh
alerts1='[
  {
    "labels": {
       "alertname": "DiskRunningFull",
       "dev": "sda1",
       "instance": "example1"
     },
     "annotations": {
        "info": "The disk sda1 is running full",
        "summary": "please check the instance example1"
      }
  }
]'
curl -XPOST -d"$alerts1" http://ubuntu-worker1:9092/api/v1/alerts
curl -XPOST -d"$alerts1" http://ubuntu-worker2:9092/api/v1/alerts

> chmod a+x send_alert.sh

Run it
> ./send_alert.sh

It will only receive one alarm in the requestbin.com since it is clustered.

References:
https://zhuanlan.zhihu.com/p/87712350