https://www.gitbook.com/book/songjiayang/prometheus/details （Prometheus 实战）
https://github.com/1046102779/prometheus （Prometheus 非官方中文手册）
http://www.bubuko.com/infodetail-2004088.html （基于prometheus监控k8s集群）
http://www.cnblogs.com/sfnz/p/6566951.html （安装prometheus+grafana监控mysql redis kubernetes等，非docker安装）
http://blog.csdn.net/wenwst/article/details/76624019 (Kubernetes 1.6 部署prometheus和grafana数据持久）)
https://github.com/jason-riddle/monitor-k8s-with-prom （Kubernetes 上prometheus监控相关）
https://github.com/kayrus/prometheus-kubernetes （prometheus-kubernetes）
https://github.com/prometheus/node_exporter （prometheus/node_exporter）
http://dockone.io/article/2579 （ Prometheus在Kubernetes下的监控实践）
http://www.ywnds.com/?p=9656 ( 使用Prometheus+Grafana监控MySQL实践)
https://github.com/prometheus/prometheus/releases （prometheus 下载列表）
https://github.com/prometheus/node_exporter/releases/ （node_exporter下载列表）
https://laily.net/article/Prometheus%20%E5%88%9D%E4%BD%93%E9%AA%8C%281%29%20-%20%E5%AE%89%E8%A3%85 (Prometheus 初体验(1) - 安装)
http://blog.csdn.net/u010871982/article/details/77838592?locationNum=2&fps=1 (prometheus简单入门)
https://www.robustperception.io/scaling-and-federating-prometheus/ (prometheus federate)
http://dbaplus.cn/news-72-1462-1.html (360基于Prometheus的在线服务监控实践)
1、prometheus安装
[root@localhost prometheus]# wget https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gz
[root@localhost prometheus]# mkdir /opt/prometheus
[root@localhost prometheus]# tar -zxvf prometheus-1.7.1.linux-amd64.tar.gz -C /opt/prometheus --strip-components=1
[root@localhost prometheus]# cd /opt/prometheus/
[root@localhost prometheus]# cp prometheus.yml prometheus.yml.back
[root@localhost prometheus]# vim prometheus.yml #注意 yaml 文件不允许有 tab 符，一律得使用空格
# 全局配置
global:
scrape_interval: 15s #默认 15秒到目标处抓取数据
# 这个标签是在本机上每一条时间序列上都会默认产生的，主要可以用于 联合查询、远程存储、Alertmanger时使用。
external_labels:
monitor: 'codelab-monitor'
# 这里就表示抓取对象的配置
# 设置抓取自身数据
scrape_configs:
# job name 这个配置是表示在这个配置内的时间序例，每一条都会自动添加上这个{job_name:"prometheus"}的标签。
- job_name: 'prometheus'
# 重写了全局抓取间隔时间，由15秒重写成5秒。
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
启动：
nohup ./prometheus --config.file=prometheus.yml &
或
nohup /opt/ prometheus-1.7.1.linux-amd64/prometheus &
这时 浏览器中页面访问http://localhost:9090/ ，可以看到Prometheus的graph页面。
http://www.cnblogs.com/vovlie/p/Prometheus_install.html （参考）
可直接加载Prometheus配置而不停止服务方式让配置生效，在调试过程中，每次修改配置后执行该操作让配置生效更方便：
# curl -X POST http://localhost:9090/-/reload
# netstat -antl|grep 9090 #查看是否启动成功！
如果我们要采用进程方式管理它，则需要创建脚本：
可以创建一个用户名来启动：
[root@localhost config]# useradd prometheus
[root@localhost ~]# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
Deion=prometheus
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus \ #prometheus安装目录
-config.file=/usr/local/prometheus/prometheus.yml \ #prometheus安装目录下的prometheus.yml
-storage.local.path=/home/prometheusdata
Restart=on-failure
[Install]
WantedBy=multi-user.target
说明： -storage.local.path=/home/prometheusdata 指定的存储目录必须要让创建的prometheus用户有权限
保存退出后，此时可以用命令启动 systemctl start prometheus
# systemctl enable Prometheus.service
# systemctl restart Prometheus.service
2、Grafana 安装
[root@localhost prometheus]# wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.5.0-1.x86_64.rpm
[root@localhost prometheus]# yum install initscripts fontconfig -y
[root@localhost prometheus]# rpm -Uvh grafana-4.5.0-1.x86_64.rpm
warning: grafana-4.5.0-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
error: Failed dependencies:
urw-fonts is needed by grafana-4.5.0-1.x86_64
安装发现报错;所以采用如下命令重新安装：
[root@localhost prometheus]# yum localinstall grafana-4.5.0-1.x86_64.rpm
[root@localhost prometheus]# service grafana-server start #启动服务
Starting grafana-server (via systemctl): [ OK ]
[root@localhost prometheus]# netstat -anp|grep 3000
查看到3000 端口已经OK；
页面http://localhost:3000 ，默认账号、密码admin/admin
http://docs.grafana.org/installation/rpm/ （gragana 官方文档）
可以将Grafana设置为系统服务
#mkdir-p/var/run/grafana
#chowngrafana.grafana/var/run/grafana
#vim/etc/sysconfig/grafana-server,
添加:PID_FILE_DIR=/var/run/grafan
#vim/etc/systemd/system/grafana.service
[Unit]
Description=GrafanaServices
Documentation=https://github.com/grafana/grafana
After=network.target
[Service]
EnvironmentFile=/etc/sysconfig/grafana-server
User=grafana
Group=grafana
Type=simple
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/sbin/grafana-server\
--config=${CONF_FILE} \
--pidfile=${PID_FILE_DIR}/grafana-server.pid \
cfg:default.paths.logs=${LOG_DIR} \
cfg:default.paths.data=${DATA_DIR} \
cfg:default.paths.plugins=${PLUGINS_DIR}
LimitNOFILE=10000
TimeoutStopSec=20UMask=0027
[Install]
WantedBy=multi-user.target
#以上配置文件中的变量${CONF_FILE}读取的是/etc/sysconfig/grafana-server中的内容
#配置文件变更后必须先reload
# systemctl daemon-reload
# systemctl restart grafana.service
# systemctl enable grafana.service
Prometheus 和 Grafana 的对接如下：
https://prometheus.io/docs/visualization/grafana/ （prometheus和grafana对接文档）
替换grafana的dashboards
Grafana 并没有太多的配置好的图表模板，除了 Percona 开源的一些外，很多需要自行配置。
[root@localhost prometheus]# yum install git -y
[root@localhost prometheus]# git clone https://github.com/percona/grafana-dashboards.git
Cloning into 'grafana-dashboards'...
remote: Counting objects: 1308, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 1308 (delta 32), reused 40 (delta 21), pack-reused 1256
Receiving objects: 100% (1308/1308), 6.39 MiB | 1.67 MiB/s, done.
Resolving deltas: 100% (982/982), done.
[root@localhost prometheus]# cp -r grafana-dashboards/dashboards /var/lib/grafana/
[root@localhost prometheus]# vim /etc/grafana/grafana.ini
修改如下：
[dashboards.json] enabled = true path = /var/lib/grafana/dashboards
[root@localhost prometheus]# service grafana-server restart
或用如下命令重启：
[root@localhost prometheus]# systemctl restart grafana-server
3、node_exporter 安装
[root@localhost prometheus]# wget https://github.com/prometheus/node_exporter/releases/download/v0.14.0/node_exporter-0.14.0.linux-amd64.tar.gz
[root@localhost prometheus]# tar -zxvf node_exporter-0.14.0.linux-amd64.tar.gz
[root@localhost local]# mv /home/prometheus/node_exporter-0.14.0.linux-amd64 ./node_exporter-0.14.0
[root@localhost local]# cd node_exporter-0.14.0/
[root@localhost node_exporter-0.14.0]# nohup ./node_exporter &
查看进程是否OK
[root@localhost node_exporter-0.14.0]# ps -ef|grep node_exporter
root 24760 24106 0 14:39 pts/1 00:00:00 ./node_exporter
root 24766 24106 0 14:39 pts/1 00:00:00 grep --color=auto node_exporter
node_exporter 也可做成服务进程启动，
[root@localhost ~]# vim /etc/systemd/system/node_exporter.service
提供的node exporter 的 systemd 脚本如下：
[Unit]
Deion=node_exporter
Description=Prometheus node exporter
After=local-fs.target network-online.target network.target
Wants=local-fs.target network-online.target network.target
[Service]
Type=simple
User=prometheus #用户prometheus
ExecStart=/usr/local/prometheus/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
# systemctl enable node_export.service
# systemctl restart node_export.service
4、alertManager 安装
http://blog.csdn.net/y_xiao_/article/details/50818451
Prometheus Alertmanager报警组件
http://www.jianshu.com/p/239b145e2acc (Prometheus Alertmanager报警组件)
Alertmanager报警模块
https://github.com/prometheus/alertmanager ）（alertmanager gighub）
Alert template:
https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/ （自定义的alertmanager 模板）
Sending alert notifications to multiple destinations
https://www.robustperception.io/sending-alert-notifications-to-multiple-destinations/ (发送提醒到多目的地)
Alert tree:
https://prometheus.io/webtools/alerting/routing-tree-editor/ (Routing tree editor)
[root@localhost prometheus]# wget https://github.com/prometheus/alertmanager/releases/download/v0.9.1/alertmanager-0.9.1.linux-amd64.tar.gz
[root@localhost prometheus]# tar -zxvf alertmanager-0.9.1.linux-amd64.tar.gz
[root@localhost prometheus]# mv alertmanager-0.9.1.linux-amd64 /opt/alertmanager
[root@localhost prometheus]# cd /opt/alertmanager
[root@localhost prometheus]# nohup ./alertmanager -config.file=simple.yml &
重启prometheus 服务：
# ./prometheus -config.file=prometheus.yml -alertmanager.url http://localhost:9093
也可以通过加载配置文件方式而不重启Alertmanager服务：
# curl -XPOST http://localhost:9093/-/reload
# 设置Alertmanager 系统服务
# vim /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager.
Documentation=https://github.com/prometheus/alertmanager
After=network.target
[Service]
EnvironmentFile=-/etc/alertmanager/template
User=root
ExecStart=/opt/alertmanager/alertmanager \
-config.file=/opt/alertmanager/simple.yml \
-storage.path=/home/alertmanager \
$ALERTMANAGER_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
最后执行：
# systemctl enable alertmanager.service
# systemctl restrart alertmanager.service
访问Alertmanager页面：http://ip:9093/#/alerts
配置 Alertmanager
报警分两部分，报警条件规则文件默认放在Prometheus安装目录下，文件名为 alert.rules。具体通知内容，例如邮件地址和通知人员设置在Alertmanager安装目录下的simply.yml文件，以下是一些基础和常用配置，阈值和时间根据自己需求进行修改。
#alert.rules:
ALERT node_down
IF up == 0 AND job="node"
FOR 5m
ANNOTATIONS {
summary = "Node is down",
description = "Node has been unreachable for more than 5 minutes.",
severity = "warning"
}
ALERT snmp_down
IF up == 0 AND job="snmp"
FOR 5m ANNOTATIONS {
summary = "SNMP is down",
description = "SNMP has been unreachable for more than 5 minutes.",
severity = "warning"
}
ALERT fs_at_80_percent
IF hrStorageUsed{hrStorageDescr=~"/.+"} / hrStorageSize >= 0.8
FOR 15m
ANNOTATIONS {
summary = "File system {{$labels.hrStorageDescr}} is at 80%",
description = "{{$labels.hrStorageDescr}} has been at 80% for more than 15 Minutes.",
severity = "warning"
}
ALERT fs_at_90_percent
IF hrStorageUsed{hrStorageDescr=~"/.+"} / hrStorageSize >= 0.9
FOR 15m
ANNOTATIONS {
summary = "File system {{$labels.hrStorageDescr}} is at 90%",
description = "{{$labels.hrStorageDescr}} has been at 90% for more than 15 Minutes.",
severity = "average"
}
ALERT disk_load_mostly_random_reads
IF rate(diskIOReads{diskIODevice=~"sd[a-z]+"}[5m]) > 20 AND
rate(diskIONReadX{diskIODevice=~"sd[a-z]+"}[5m]) / rate(diskIOReads{diskIODevice=~"sd[a-z]+"}[5m]) < 10000
FOR 15m
ANNOTATIONS { summary = "Disk {{$labels.diskIODevice}} reads are mostly random.",
description = "{{$labels.diskIODevice}} reads have been mostly random for the past 15 Minutes.",
severity = "info"
}
ALERT disk_load_mostly_random_writes
IF rate(diskIOWrites{diskIODevice=~"sd[a-z]+"}[5m]) > 20 AND
rate(diskIONWrittenX{diskIODevice=~"sd[a-z]+"}[5m]) / rate(diskIOWrites{diskIODevice=~"sd[a-z]+"}[5m]) < 10000
FOR 15m
ANNOTATIONS {
summary = "Disk {{$labels.diskIODevice}} writes are mostly random.",
description = "{{$labels.diskIODevice}} writes have been mostly random for the past 15 Minutes.",
severity = "info"
}
ALERT disk_load_high
IF diskIOLA1{diskIODevice=~"s|vd[a-z]+"} > 30
FOR 15m
ANNOTATIONS {
summary = "Disk {{$labels.diskIODevice}} is at 30%",
description = "{{$labels.diskIODevice}} Load has exceeded 30% over the past 15 Minutes.",
severity = "warning"
}
ALERT cpu_load_high
IF ssCpuIdle < 70
FOR 15m
ANNOTATIONS {
summary = "CPU is at 30%",
description = "CPU Load has constantly exceeded 30% over the past 15 Minutes.",
severity = "warning"
}
ALERT linux_load_high
IF laLoad1 > 50
FOR 15m
ANNOTATIONS {
summary = "Linux Load is at 40",
description = "Linux Load has constantly exceeded 40 over the past 15 Minutes.",
severity = "average"
}
ALERT if_operstatus_changed
IF delta(ifOperStatus[15m]) != 0
ANNOTATIONS {
summary = "Port {{$labels.ifDescr}} changed status",
description = "Port {{$labels.ifDescr}} went up or down in the past 15 Minutes",
severity = "info"
}
ALERT if_traffic_at_30_percent
IF ifSpeed > 10000000 AND
ifOperStatus == 1 AND
rate(ifInOctets[5m]) > ifSpeed * 0.3
FOR 15m
ANNOTATIONS {
summary = "Port {{$labels.ifDescr}} is at 30%",
description = "Port {{$labels.ifDescr}} has had at least 30% traffic over the past 15 Minutes.",
severity = "warning"
}
ALERT if_traffic_at_70_percent
IF ifSpeed > 10000000 AND
ifOperStatus == 1 AND rate(ifInOctets[5m]) > ifSpeed * 0.7
FOR 15m
ANNOTATIONS {
summary = "Port {{$labels.ifDescr}} is at 70%",
description = "Port {{$labels.ifDescr}} has had at least 70% traffic over the past 15 Minutes.",
severity = "average"
}
# CPU告警
ALERT cpu_overload
IF node_load1 >= 0.8
FOR 3m
LABELS { severity = "all" }
ANNOTATIONS {
summary = "Instance {{ $labels.instance }} cpu_load1 over 80% for 3 minutes",
description = "{{ $labels.instance }} of job {{ $labels.job }} cpu_load1 over 80% for 3 minutes.",
}
# 内存告警
ALERT memory_overload
IF (node_memory_MemTotal-node_memory_MemFree)/node_memory_MemTotal >= 0.8
FOR 3m
LABELS { severity = "all" }
ANNOTATIONS {
summary = "Instance {{ $labels.instance }} memory_load over 80% for 3 minutes",
description = "{{ $labels.instance }} of job {{ $labels.job }} memory_load over 80% for 3 minutes.",
}
---------------------------------------------------
# simply.yml
主要分三部分,Global部分设置发送邮件服务器信息，route设置规则和报警时间间隔等，receivers设置接收人。
global:
#设置发送邮件的地址和smtp信息
smtp_smarthost:'smtp.abc.com'
smtp_from:'prometheus@abc.com'
smtp_auth_username:'prometheus'
smtp_auth_password:'abcd’
route:receiver:'team-X-mails'group_by:['alertname']group_wait:30s
group_interval:5m
repeat_interval:6h
inhibit_rules:
-source_match:
severity:'critical'
target_match:
severity:'warning'
#Applyinhibitionifthealertnameisthesame.
equal:['alertname']
receivers:
-name:'team-X-mails'
email_configs:
-to:'support@abc.com'
send_resolved:true
#设置完毕后需要重新加载配置文件
5、cadvisor 安装配置
docker run -d --restart=always --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/dev/disk/:/dev/disk:ro --publish=8090:8080 --detach=true --name=cadvisor google/cadvisor:latest
在浏览器中：http://ip:8090 就可以访问了
# 监控cAdvisor报警条件：
# vim containers.rules
ALERT cAdvisor_down
IF absent(container_memory_usage_bytes{name="cadvisor"})
FOR 1m
LABELS { severity = "critical" }
ANNOTATIONS {
summary= "cAdvisor containers down",
description= "cAdvisor container is down for more than 1 minutes."
}
ALERT cAdvisor_high_cpu
IF sum(rate(container_cpu_usage_seconds_total{name="cadvisor"}[1m])) / count(node_cpu{mode="system"}) * 100 > 10
FOR 5m
LABELS { severity = "warning" }
ANNOTATIONS {
summary= "cAdvisor high CPU usage",
description= "cAdvisor CPU usage is {{ humanize $value}}%."
}
ALERT cAdvisor_high_memory
IF sum(container_memory_usage_bytes{name="cadvisor"}) > 1200000000 FOR 5m
LABELS { severity = "warning" }
ANNOTATIONS {
summary = "cAdvisor high memory usage",
description = "cAdvisor memory consumption is at {{ humanize $value}}.",
}
相关推荐
在监控Docker容器时，Prometheus的主要优势包括其灵活性、强大的查询语言PromQL以及丰富的生态系统，如Alertmanager用于警报管理，Grafana用于可视化。以下是一些关键知识点： 1. **配置Prometheus Server**： - ...
现在网上有很多关于监控平台，不过大部分开源平台不够开放，想要汉化和自定义监控平台随意添加仪表板的话还是首选prometheus+grafana，汉化在本资源主要介绍的前端vue的汉化，prometheus主要用于采集微服务数据或者...
08prometheus监控docker主机
Prometheus监控elk实现1、elk环境以及prometheus环境已经部署完成2、下载第三方监控插件，实现prometheus与elk的对接插件下载地
prometheus：用于Prometheus监控的docker-compose堆栈
本资料包提供了Prometheus监控Redis的详细教程，包括部署文档和相关软件包，旨在帮助用户理解如何设置和使用Prometheus来监控Redis，并配置报警规则以及时发现并处理问题。 首先，你需要了解如何安装Prometheus。...
### 脚本部分介绍 - 运行之前先查看脚本，修改一下Basrdir的路径为你想安装的路径 - 把这个项目拷贝，记得是"拷贝"到Basedir...部署Prometheus、node_exporter、grafana、alertmanager Prometheus -----数据采集 ```
在这个“使用prometheus监控k8s的grafana模板”主题中，我们将深入探讨如何利用Grafana的模板功能来有效监控Kubernetes（k8s）集群。 一、Prometheus与Kubernetes的集成 Prometheus可以通过Service Discovery机制...
使用 cadvisor、prometheus、redis 和 grafana 监控 docker 容器docker 监控使用 cadvisor、prometheus、redis 和 grafana 监控 docker 容器运行-docker-compose up如果一切正常，你应该看到这条消息msg="服务器已...
Docker 的 Prometheus 监控 使用 Prometheus 和 fig 监控 Docker 主机。 要求 码头工人 图 安装 此存储库包含参考存储库作为子模块。 要使用子模块下载此存储库，您必须使用 git clone命令的--recursive标志： git...
**Prometheus监控系统详解** Prometheus是一款开源的系统监控和报警工具，由SoundCloud公司于2012年创建，并逐渐发展成为云原生计算基金会（CNCF）的重要项目之一。Prometheus以其强大的时间序列数据库、灵活的查询...
docker-compose快速搭建 Prometheus+Grafana监控系统
prometheus docker 容器监控 k8s kubernetes 好东西 岗岗的
- **Prometheus**：基于时间序列数据的容器监控解决方案，适合监控Docker容器和Kubernetes集群。 在实施监控系统时，公司应根据自身的业务特点和监控需求，选择适合的监控工具和实施策略。Prometheus由于其对容器和...
【docker-compose-prometheus包】是一个基于Docker Compose的解决方案，用于快速搭建和管理Prometheus监控系统。这个压缩包包含了所有必要的配置文件和服务，旨在帮助IT运维人员在本地或云端环境中轻松部署一套全面...
然而，在部署过程中，你可能会碰到与镜像相关的问题，这些问题通常与 Kubernetes 集群无法拉取正确的 Docker 镜像有关。 问题背景 kube-prometheus 项目通常包含多个组件，每个组件都需要特定的 Docker 镜像。例如...
Cadvisor（Container Advisor）是Google开发的一个开源项目，用于监控Docker容器的资源使用情况，包括CPU、内存、磁盘I/O和网络等。Cadvisor集成到Prometheus中，提供了对Docker容器内资源消耗的详细视图。 ...
Prometheus 是一个开源的监控和警报工具，用于收集、处理、聚合以及查询时间序列数据。Alertmanager 是 Prometheus 的组成部分，负责处理 Prometheus 发送的警报，并将其分组、静默、重定向或通过多种通知方式进行...
在Prometheus的架构中，数据采集主要通过拉取（Pulling）的方式进行，也就是说，Prometheus服务器会定期向被监控的目标发送请求，从而获取目标的性能指标数据。当系统规模变大时，为了高效获取和存储数据，...