• 搭建 Hadoop 生态集群大数据监控告警平台

    目录

    一、部署 prometheus 环境

    1.1 下载安装包

    1.2 解压安装

    1.3 修改配置文件

    1.3.1 hadoop-env.sh

    1.3.2 prometheus_config.yml

    1.3.3 zkServer.sh

    1.3.4 prometheus_zookeeper.yaml

    1.3.5 alertmanager.yml 

    1.3.6 prometheus.yml 

    1.3.7 config.yml 

    1.3.8 template.tmpl 

    1.3.9 告警规则

    1.3.10 /etc/profile

    1.3.11 prometheus_spark.yml

    1.3.12 spark-env.sh

    1.3.13 hive 

    1.3.14 prometheus_metastore.yaml、prometheus_hs2.yaml 

    1.4 创建 systemd 服务

    1.4.1 创建 prometheus 用户

    1.4.2 alertmanager.service 

    1.4.3 prometheus.service 

    1.4.4 node_exporter.service 

    1.4.5 pushgateway.service 

    1.4.6 grafana.service 

    1.5 启动服务

    二、补充

    2.1 告警规则和 grafana 仪表盘文件下载

    2.2 服务进程监控脚本

    2.2.1 prometheus-webhook-dingtalk

    2.2.2 hive 

    2.2.3 日志切割 

    2.3 常用命令 

    2.4 参考文档

    2.5 内网环境

    2.5.1 nginx（公网）

    2.5.2 config 

    2.5.3 /etc/hosts 

    Hadoop 集群规模：Hadoop YARN HA 集群安装部署详细图文教程_Stars.Sky的博客-CSDN博客

    Spark 集群规模：Spark-3.2.4 高可用集群安装部署详细图文教程_Stars.Sky的博客-CSDN博客

    IP

    主机名

    运行角色

    192.168.170.136

    hadoop01

    namenode datanode resourcemanager nodemanager JournalNode DFSZKFailoverController QuorumPeerMain spark hive

    192.168.170.137

    hadoop02

    namenode datanode resourcemanager nodemanager JournalNode DFSZKFailoverController QuorumPeerMain spark

    192.168.170.138

    hadoop03

    datanode nodemanage JournalNode QuorumPeerMain spark

    一、部署 prometheus 环境

    1.1 下载安装包

    • prometheus、alertmanager、pushgateway、node_exporter：https://prometheus.io/download/

    • prometheus-webhook-dingtalk：https://github.com/timonwong/prometheus-webhook-dingtalk/tree/main

    • grafana：https://grafana.com/grafana/download

    • jmx_exporter：https://github.com/prometheus/jmx_exporter

    1.2 解压安装

            新建一个 /monitor 目录，把上面下载的 tar.gz 包都解压安装在 /monitor 目录下，并重命名如下名字：

    1.3 修改配置文件

    1.3.1 hadoop-env.sh

    修改完后要把这个文件 scp 给各个 Hadoop 节点！

    1. [root@hadoop01 ~]# cd /bigdata/hadoop/server/hadoop-3.2.4/etc/hadoop/
    2. [root@hadoop01 /bigdata/hadoop/server/hadoop-3.2.4/etc/hadoop]# vim hadoop-env.sh
    3. if ! grep -q <<<"$HDFS_NAMENODE_OPTS" jmx_prometheus_javaagent; then
    4. HDFS_NAMENODE_OPTS="$HDFS_NAMENODE_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30002:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    5. fi
    6. if ! grep -q <<<"$HDFS_DATANODE_OPTS" jmx_prometheus_javaagent; then
    7. HDFS_DATANODE_OPTS="$HDFS_DATANODE_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30003:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    8. fi
    9. if ! grep -q <<<"$YARN_RESOURCEMANAGER_OPTS" jmx_prometheus_javaagent; then
    10. YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30004:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    11. fi
    12. if ! grep -q <<<"$YARN_NODEMANAGER_OPTS" jmx_prometheus_javaagent; then
    13. YARN_NODEMANAGER_OPTS="$YARN_NODEMANAGER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30005:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    14. fi
    15. if ! grep -q <<<"$HDFS_JOURNALNODE_OPTS" jmx_prometheus_javaagent; then
    16. HDFS_JOURNALNODE_OPTS="$HDFS_JOURNALNODE_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30006:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    17. fi
    18. if ! grep -q <<<"$HDFS_ZKFC_OPTS" jmx_prometheus_javaagent; then
    19. HDFS_ZKFC_OPTS="$HDFS_ZKFC_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30007:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    20. fi
    21. if ! grep -q <<<"$HDFS_HTTPFS_OPTS" jmx_prometheus_javaagent; then
    22. HDFS_HTTPFS_OPTS="$HDFS_HTTPFS_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30008:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    23. fi
    24. if ! grep -q <<<"$YARN_PROXYSERVER_OPTS" jmx_prometheus_javaagent; then
    25. YARN_PROXYSERVER_OPTS="$YARN_PROXYSERVER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30009:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    26. fi
    27. if ! grep -q <<<"$MAPRED_HISTORYSERVER_OPTS" jmx_prometheus_javaagent; then
    28. MAPRED_HISTORYSERVER_OPTS="$MAPRED_HISTORYSERVER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30010:/bigdata/hadoop/server/hadoop-3.2.4/prometheus_config.yml"
    29. fi

    1.3.2 prometheus_config.yml

    修改完后要把这个文件 scp 给各个 Hadoop 节点！

    1. [root@hadoop01 ~]# cd /bigdata/hadoop/server/hadoop-3.2.4/
    2. [root@hadoop01 /bigdata/hadoop/server/hadoop-3.2.4]# vim prometheus_config.yml
    3. rules:
    4. - pattern: ".*"

    1.3.3 zkServer.sh

    修改完后要把这个文件 scp 给各个 zookeeper 节点！

    1. [root@hadoop01 ~]# cd /bigdata/hadoop/zookeeper/zookeeper-3.7.1/bin/
    2. [root@hadoop01 /bigdata/hadoop/zookeeper/zookeeper-3.7.1/bin]# vim zkServer.sh
    3. if [ "x$JMXLOCALONLY" = "x" ]
    4. then
    5.     JMXLOCALONLY=false
    6. fi
    7.  
    8. JMX_DIR="/monitor"
    9. JVMFLAGS="$JVMFLAGS -javaagent:$JMX_DIR/jmx_prometheus_javaagent-0.19.0.jar=30011:/bigdata/hadoop/zookeeper/zookeeper-3.7.1/prometheus_zookeeper.yaml"

    1.3.4 prometheus_zookeeper.yaml

    修改完后要把这个文件 scp 给各个 zookeeper 节点！

    1. [root@hadoop01 ~]# cd /bigdata/hadoop/zookeeper/zookeeper-3.7.1/
    2. [root@hadoop01 /bigdata/hadoop/zookeeper/zookeeper-3.7.1]# vim prometheus_zookeeper.yaml 
    3. rules:
    4.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    5.     name: "zookeeper_$2"
    6.     type: GAUGE
    7.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    8.     name: "zookeeper_$3"
    9.     type: GAUGE
    10.     labels:
    11.       replicaId: "$2"
    12.   - pattern: "org.apache.ZooKeeperService<>(Packets\\w+)"
    13.     name: "zookeeper_$4"
    14.     type: COUNTER
    15.     labels:
    16.       replicaId: "$2"
    17.       memberType: "$3"
    18.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    19.     name: "zookeeper_$4"
    20.     type: GAUGE
    21.     labels:
    22.       replicaId: "$2"
    23.       memberType: "$3"
    24.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    25.     name: "zookeeper_$4_$5"
    26.     type: GAUGE
    27.     labels:
    28.       replicaId: "$2"
    29.       memberType: "$3"
    30.  
    31.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    32.     type: GAUGE
    33.     name: "zookeeper_$2"
    34.   - pattern: "org.apache.ZooKeeperService<>(\\w+)"
    35.     type: GAUGE
    36.     name: "zookeeper_$2"

    1.3.5 alertmanager.yml 

    1. [root@hadoop01 ~]# cd /monitor/alertmanager/
    2. [root@hadoop01 /monitor/alertmanager]# ls
    3. alertmanager  alertmanager.yml  amtool  data  LICENSE  NOTICE
    4. [root@hadoop01 /monitor/alertmanager]# vim alertmanager.yml 
    5. global:
    6.   resolve_timeout: 5m
    7.  
    8. templates:
    9.   - '/monitor/prometheus-webhook-dingtalk/contrib/templates/legacy/*.tmpl'
    10.  
    11. route:
    12.   group_by: ['job', 'severity']
    13.   group_wait: 30s
    14.   group_interval: 5m
    15.   repeat_interval: 3h
    16.   receiver: 'webhook1'
    17.  
    18. receivers:
    19.   - name: 'webhook1'
    20.     webhook_configs:
    21.     - url: 'http://192.168.170.136:8060/dingtalk/webhook1/send'
    22.       send_resolved: true

    1.3.6 prometheus.yml 

    1. [root@hadoop01 ~]# cd /monitor/prometheus
    2. [root@hadoop01 /monitor/prometheus]# ls
    3. console_libraries  consoles  data  LICENSE  NOTICE  prometheus  prometheus.yml  promtool  rule
    4. [root@hadoop01 /monitor/prometheus]# vim prometheus.yml 
    5. # my global config
    6. global:
    7.   scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    8.   evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
    9.  
    10. # Alertmanager configuration
    11. alerting:
    12.   alertmanagers:
    13.     - static_configs:
    14.         - targets: ['192.168.170.136:9093']
    15.  
    16. # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    17. rule_files:
    18.   - "rule/*.yml"
    19.  
    20. # A scrape configuration containing exactly one endpoint to scrape:
    21. # Here it's Prometheus itself.
    22. scrape_configs:
    23.   - job_name: "prometheus"
    24.     scrape_interval: 30s
    25.     static_configs:
    26.       - targets: ["hadoop01:9090"]
    28.   # zookeeper 集群配置
    29.   - job_name: "zookeeper"
    30.     scrape_interval: 30s
    31.     static_configs:
    32.       - targets: ['hadoop01:30011', 'hadoop02:30011', 'hadoop03:30011']
    34.   # node_exporter 配置
    35.   - job_name: "pushgatewawy"
    36.     scrape_interval: 30s
    37.     static_configs:
    38.       - targets: ["hadoop01:9091"]
    39.                                                                         
    40.   # node_exporter 配置
    41.   - job_name: "node_exporter"
    42.     scrape_interval: 30s
    43.     static_configs:
    44.       - targets: ['hadoop01:9100', 'hadoop02:9100', 'hadoop03:9100']
    46.   - job_name: " namenode "
    47.     scrape_interval: 30s
    48.     static_configs:
    49.       - targets: ['hadoop01:30002', 'hadoop02:30002']
    50. #        labels:
    51. #          instance: namenode 服务器
    53.   - job_name: "datanode"
    54.     scrape_interval: 30s
    55.     static_configs:
    56.       - targets: ['hadoop01:30003', 'hadoop02:30003', 'hadoop03:30003']
    58.   - job_name: "resourcemanager"
    59.     scrape_interval: 30s
    60.     static_configs:
    61.       - targets: ['hadoop01:30004', 'hadoop02:30004']
    63.   - job_name: "nodemanager"
    64.     scrape_interval: 30s
    65.     static_configs:
    66.       - targets: ['hadoop01:30005', 'hadoop02:30005', 'hadoop03:30005']
    68.   - job_name: "journalnode"
    69.     scrape_interval: 30s
    70.     static_configs:
    71.       - targets: ['hadoop01:30006', 'hadoop02:30006', 'hadoop03:30006']
    73.   - job_name: "zkfc"
    74.     scrape_interval: 30s
    75.     static_configs:
    76.       - targets: ['hadoop01:30007', 'hadoop02:30007']
    78.   - job_name: "jobhistoryserver"
    79.     scrape_interval: 30s
    80.     static_configs:
    81.       - targets: ["hadoop01:30010"]
    82.       
    83.   - job_name: "spark_master"
    84.     scrape_interval: 30s
    85.     static_configs:
    86.       - targets: ['hadoop01:30012', 'hadoop02:30012']
    88.   - job_name: "spark_worker"
    89.     scrape_interval: 30s
    90.     static_configs:
    91.       - targets: ['hadoop01:30013', 'hadoop02:30013', 'hadoop03:30013'] 
    92.   
    93.   - job_name: "hive_metastore"
    94.     scrape_interval: 30s
    95.     static_configs:
    96.       - targets: ["hadoop01:30014"]
    98.   - job_name: "hive_hs2"
    99.     scrape_interval: 30s
    100.     static_configs:
    101.       - targets: ["hadoop01:30015"]

    1.3.7 config.yml 

    1. [root@hadoop01 ~]# cd /monitor/prometheus-webhook-dingtalk/
    2. [root@hadoop01 /monitor/prometheus-webhook-dingtalk]# ls
    3. config.example.yml  config.yml  contrib  LICENSE  nohup.out  prometheus-webhook-dingtalk
    4. [root@hadoop01 /monitor/prometheus-webhook-dingtalk]# vim config.yml
    5. ## Request timeout
    6. # timeout: 5s
    7.  
    8. ## Uncomment following line in order to write template from scratch (be careful!)
    9. #no_builtin_template: true
    10.  
    11. ## Customizable templates path
    12. templates:
    13.   - /monitor/prometheus-webhook-dingtalk/contrib/templates/legacy/template.tmpl
    14.  
    15. ## You can also override default template using `default_message`
    16. ## The following example to use the 'legacy' template from v0.3.0
    17. #default_message:
    18. #  title: '{{ template "legacy.title" . }}'
    19. #  text: '{{ template "legacy.content" . }}'
    20.  
    21. ## Targets, previously was known as "profiles"
    22. targets:
    23.   webhook1:
    24.     url: https://oapi.dingtalk.com/robot/send?access_token=0d6c5dc25fa3f79cf2f83c92705fe4594dcxxx
    25.     # secret for signature
    26.     secret: SECecdbfff858ab8f3195dc34b7e225fee9341bc9xxx
    27.     message:
    28.       title: '{{ template "ops.title" . }}'
    29.       text: '{{ template "ops.content" . }}'

    1.3.8 template.tmpl 

    1. [root@hadoop01 ~]# cd /monitor/prometheus-webhook-dingtalk/contrib/templates/legacy/
    2. [root@hadoop01 /monitor/prometheus-webhook-dingtalk/contrib/templates/legacy]# vim template.tmpl
    3. {{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
    4. {{ end }}
    5.  
    6. {{ define "__alert_list" }}{{ range . }}
    7. ---
    8.     **告警类型**: {{ .Labels.alertname }}
    9.     **告警级别**: {{ .Labels.severity }}
    10.     **故障主机**: {{ .Labels.instance }}
    11.     **告警信息**: {{ .Annotations.description }}
    12.     **触发时间**: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
    13. {{ end }}{{ end }}
    14.  
    15. {{ define "__resolved_list" }}{{ range . }}
    16. ---
    17.     **告警类型**: {{ .Labels.alertname }}
    18.     **告警级别**: {{ .Labels.severity }}
    19.     **故障主机**: {{ .Labels.instance }}
    20.     **触发时间**: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
    21.     **恢复时间**: {{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
    22. {{ end }}{{ end }}
    23.  
    24.  
    25. {{ define "ops.title" }}
    26. {{ template "__subject" . }}
    27. {{ end }}
    28.  
    29. {{ define "ops.content" }}
    30. {{ if gt (len .Alerts.Firing) 0 }}
    31. **====侦测到{{ .Alerts.Firing | len  }}个故障====**
    32. {{ template "__alert_list" .Alerts.Firing }}
    33. ---
    34. {{ end }}
    35.  
    36. {{ if gt (len .Alerts.Resolved) 0 }}
    37. **====恢复{{ .Alerts.Resolved | len  }}个故障====**
    38. {{ template "__resolved_list" .Alerts.Resolved }}
    39. {{ end }}
    40. {{ end }}
    41.  
    42. {{ define "ops.link.title" }}{{ template "ops.title" . }}{{ end }}
    43. {{ define "ops.link.content" }}{{ template "ops.content" . }}{{ end }}
    44. {{ template "ops.title" . }}
    45. {{ template "ops.content" . }}

    1.3.9 告警规则

    在第二点下面的文件里下载即可。

    1.3.10 /etc/profile

    修改完后要把这个文件 scp 给各个 Hadoop 节点！

    1. # JDK 1.8
    2. JAVA_HOME=/usr/java/jdk1.8.0_381
    3. PATH=$PATH:$JAVA_HOME/bin
    4. CLASSPATH=.:$JAVA_HOME/lib
    5. export JAVA_HOME PATH CLASSPATH
    6.  
    7. # hadoop
    8. export HADOOP_HOME=/bigdata/hadoop/server/hadoop-3.2.4/
    9. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    10.  
    11. # spark
    12. export SPARK_HOME=/bigdata/spark-3.2.4
    13. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
    14. export PYSPARK_PYTHON=/usr/local/anaconda3/envs/pyspark/bin/python3.10
    15. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native

    1.3.11 prometheus_spark.yml

    修改完后要把这个文件 scp 给各个 spark 节点！

    1. (base) [root@hadoop01 ~]# cd /bigdata/spark-3.2.4/
    2. (base) [root@hadoop01 /bigdata/spark-3.2.4]# vim prometheus_spark.yml
    3. rules:
    4.  
    5.   # These come from the master
    6.   # Example: master.aliveWorkers
    7.   - pattern: "metrics<>Value"
    8.     name: spark_master_$1
    9.  
    10.   # These come from the worker
    11.   # Example: worker.coresFree
    12.   - pattern: "metrics<>Value"
    13.     name: spark_worker_$1
    14.  
    15.   # These come from the application driver
    16.   # Example: app-20160809000059-0000.driver.DAGScheduler.stage.failedStages
    17.   - pattern: "metrics<>Value"
    18.     name: spark_driver_$2_$3
    19.     type: GAUGE
    20.     labels:
    21.       app_id: "$1"
    22.  
    23.   # These come from the application driver
    24.   # Emulate timers for DAGScheduler like messagePRocessingTime
    25.   - pattern: "metrics<>Count"
    26.     name: spark_driver_DAGScheduler_$2_total
    27.     type: COUNTER
    28.     labels:
    29.       app_id: "$1"
    30.  
    31.   - pattern: "metrics<>Count"
    32.     name: spark_driver_HiveExternalCatalog_$2_total
    33.     type: COUNTER
    34.     labels:
    35.       app_id: "$1"
    36.  
    37.   # These come from the application driver
    38.   # Emulate histograms for CodeGenerator
    39.   - pattern: "metrics<>Count"
    40.     name: spark_driver_CodeGenerator_$2_total
    41.     type: COUNTER
    42.     labels:
    43.       app_id: "$1"
    44.  
    45.   # These come from the application driver
    46.   # Emulate timer (keep only count attribute) plus counters for LiveListenerBus
    47.   - pattern: "metrics<>Count"
    48.     name: spark_driver_LiveListenerBus_$2_total
    49.     type: COUNTER
    50.     labels:
    51.       app_id: "$1"
    52.  
    53.   # Get Gauge type metrics for LiveListenerBus
    54.   - pattern: "metrics<>Value"
    55.     name: spark_driver_LiveListenerBus_$2
    56.     type: GAUGE
    57.     labels:
    58.       app_id: "$1"
    59.  
    60.   # These come from the application driver if it's a streaming application
    61.   # Example: app-20160809000059-0000.driver.com.example.ClassName.StreamingMetrics.streaming.lastCompletedBatch_schedulingDelay
    62.   - pattern: "metrics<>Value"
    63.     name: spark_driver_streaming_$3
    64.     labels:
    65.       app_id: "$1"
    66.       app_name: "$2"
    67.  
    68.   # These come from the application driver if it's a structured streaming application
    69.   # Example: app-20160809000059-0000.driver.spark.streaming.QueryName.inputRate-total
    70.   - pattern: "metrics<>Value"
    71.     name: spark_driver_structured_streaming_$3
    72.     labels:
    73.       app_id: "$1"
    74.       query_name: "$2"
    75.  
    76.   # These come from the application executors
    77.   # Examples:
    78.   #  app-20160809000059-0000.0.executor.threadpool.activeTasks (value)
    79.   #  app-20160809000059-0000.0.executor.JvmGCtime (counter)
    80.  
    81.   # filesystem metrics are declared as gauge metrics, but are actually counters
    82.   - pattern: "metrics<>Value"
    83.     name: spark_executor_filesystem_$3_total
    84.     type: COUNTER
    85.     labels:
    86.       app_id: "$1"
    87.       executor_id: "$2"
    88.  
    89.   - pattern: "metrics<>Value"
    90.     name: spark_executor_$3
    91.     type: GAUGE
    92.     labels:
    93.       app_id: "$1"
    94.       executor_id: "$2"
    95.  
    96.   - pattern: "metrics<>Count"
    97.     name: spark_executor_$3_total
    98.     type: COUNTER
    99.     labels:
    100.       app_id: "$1"
    101.       executor_id: "$2"
    102.  
    103.   - pattern: "metrics<>Value"
    104.     name: spark_executor_$3
    105.     type: GAUGE
    106.     labels:
    107.       app_id: "$1"
    108.       executor_id: "$2"
    109.  
    110.   # These come from the application executors
    111.   # Example: app-20160809000059-0000.0.jvm.threadpool.activeTasks
    112.   - pattern: "metrics<>Value"
    113.     name: spark_executor_$3_$4
    114.     type: GAUGE
    115.     labels:
    116.       app_id: "$1"
    117.       executor_id: "$2"
    118.  
    119.   - pattern: "metrics<>Count"
    120.     name: spark_executor_HiveExternalCatalog_$3_total
    121.     type: COUNTER
    122.     labels:
    123.       app_id: "$1"
    124.       executor_id: "$2"
    125.  
    126.   # These come from the application driver
    127.   # Emulate histograms for CodeGenerator
    128.   - pattern: "metrics<>Count"
    129.     name: spark_executor_CodeGenerator_$3_total
    130.     type: COUNTER
    131.     labels:
    132.       app_id: "$1"
    133.       executor_id: "$2"

    1.3.12 spark-env.sh

    修改完后要把这个文件 scp 给各个 spark 节点！

    1. (base) [root@hadoop01 ~]# cd /bigdata/spark-3.2.4/conf/
    2. (base) [root@hadoop01 /bigdata/spark-3.2.4/conf]# vim spark-env.sh
    3. export SPARK_MASTER_OPTS="$SPARK_MASTER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30012:/bigdata/spark-3.2.4/prometheus_spark.yml"
    4. export SPARK_WORKER_OPTS="$SPARK_WORKER_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30013:/bigdata/spark-3.2.4/prometheus_spark.yml"

    1.3.13 hive 

    1. (base) [root@hadoop01 ~]# cd /bigdata/apache-hive-3.1.2/
    2. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# vim bin/hive
    3. ···
    4. if [[ "$SERVICE" =~ ^(help|version|orcfiledump|rcfilecat|schemaTool|cleardanglingscratchdir|metastore|beeline|llapstatus|llap)$ ]] ; then
    5.   # 如果是 metastore 服务，则修改 HADOOP_CLIENT_OPTS
    6.   if [[ "$SERVICE" == "metastore" ]] ; then
    7.     export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30014:/bigdata/apache-hive-3.1.2/prometheus_metastore.yaml"
    8.   fi
    9.   SKIP_HBASECP=true
    10. fi
    11.  
    12. ···
    13. if [[ "$SERVICE" =~ ^(hiveserver2|beeline|cli)$ ]] ; then
    14.   # 如果是 hiveserver2 服务，则修改 HADOOP_CLIENT_OPTS
    15.   if [[ "$SERVICE" == "hiveserver2" ]] ; then
    16.     export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -javaagent:/monitor/jmx_prometheus_javaagent-0.19.0.jar=30015:/bigdata/apache-hive-3.1.2/prometheus_hs2.yaml"
    17.   fi
    18.   # If process is backgrounded, don't change terminal settings
    19.   if [[ ( ! $(ps -o stat= -p $$) =~ "+" ) && ! ( -p /dev/stdin ) && ( ! $(ps -o tty= -p $$) =~ "?" ) ]]; then
    20.     export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Djline.terminal=jline.UnsupportedTerminal"
    21.   fi
    22. fi
    23. ···

    1.3.14 prometheus_metastore.yaml、prometheus_hs2.yaml 

    1. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# vim prometheus_metastore.yaml 
    2. ---
    3. startDelaySeconds: 0
    4. ssl: false
    5. lowercaseOutputName: false
    6. lowercaseOutputLabelNames: false
    7. rules:
    8. - pattern: ".*"
    9.  
    10. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# vim prometheus_hs2.yaml 
    11. ---
    12. startDelaySeconds: 0
    13. ssl: false
    14. lowercaseOutputName: false
    15. lowercaseOutputLabelNames: false
    16. rules:
    17. - pattern: ".*"

    1.4 创建 systemd 服务

    1.4.1 创建 prometheus 用户

            各个节点都需要创建！！！（也可以不创建 prometheus 用户，把后面 service 文件的 prometheus 改为 root 即可！）

    1. useradd -M -s /usr/sbin/nologin prometheus
    2. chown -R prometheus:prometheus /monitor

    1.4.2 alertmanager.service 

    1. [root@hadoop01 ~]# vim /usr/lib/systemd/system/alertmanager.service 
    2. [Unit]
    3. Description=Alertmanager
    4. Documentation=https://prometheus.io/docs/alerting/alertmanager/
    5. After=network-online.target
    6. Wants=network-online.target
    7.  
    8. [Service]
    9. User=prometheus
    10. Group=prometheus
    11. Type=simple
    12. ExecStart=/monitor/alertmanager/alertmanager \
    13.     --config.file=/monitor/alertmanager/alertmanager.yml \
    14.     --storage.path=/monitor/alertmanager/data \
    15.     --web.listen-address=0.0.0.0:9093
    16. ExecReload=/bin/kill -HUP $MAINPID
    17. Restart=always
    18.  
    19. [Install]
    20. WantedBy=multi-user.target

    1.4.3 prometheus.service 

    1. [root@hadoop01 ~]# vim /usr/lib/systemd/system/prometheus.service 
    2. [Unit]
    3. Description=Prometheus Server
    4. Documentation=https://prometheus.io/docs/introduction/overview/
    5. After=network-online.target
    6.  
    7. [Service]
    8. Type=simple
    9. User=prometheus
    10. Group=prometheus
    11. WorkingDirectory=/monitor/prometheus
    12. ExecStart=/monitor/prometheus/prometheus \
    13.   --web.listen-address=0.0.0.0:9090 \
    14.   --storage.tsdb.path=/monitor/prometheus/data \
    15.   --storage.tsdb.retention.time=30d \
    16.   --config.file=prometheus.yml \
    17.   --web.enable-lifecycle
    18. ExecReload=/bin/kill -s HUP $MAINPID
    19. ExecStop=/bin/kill -s QUIT $MAINPID
    20. Restart=on-failure
    21.  
    22. [Install]
    23. WantedBy=multi-user.target

    1.4.4 node_exporter.service 

    1. [root@hadoop01 ~]# vim /usr/lib/systemd/system/node_exporter.service 
    2. [Unit]
    3. Description=Node Exporter
    4. Documentation=https://github.com/prometheus/node_exporter
    5. After=network-online.target
    6. Wants=network-online.target
    7.  
    8. [Service]
    9. User=prometheus
    10. Group=prometheus
    11. Type=simple
    12. ExecStart=/monitor/node_exporter/node_exporter
    13.  
    14. [Install]
    15. WantedBy=multi-user.target

    1.4.5 pushgateway.service 

    1. [root@hadoop01 ~]# vim /usr/lib/systemd/system/pushgateway.service 
    2. [Unit]
    3. Description=Pushgateway Server
    4. Documentation=https://github.com/prometheus/pushgateway
    5. After=network-online.target
    6. Wants=network-online.target
    7.  
    8. [Service]
    9. User=prometheus
    10. Group=prometheus
    11. Type=simple
    12. ExecStart=/monitor/pushgateway/pushgateway \
    13.           --web.listen-address=:9091 \
    14.           --web.telemetry-path=/metrics
    15. Restart=always
    16.  
    17. [Install]
    18. WantedBy=multi-user.target

    1.4.6 grafana.service 

    1. [root@hadoop01 ~]# vim /usr/lib/systemd/system/grafana.service 
    2. [Unit]
    3. Description=Grafana Server
    4. Documentation=http://docs.grafana.org
    5. After=network-online.target
    6. Wants=network-online.target
    7.  
    8. [Service]
    9. Type=simple
    10. User=prometheus
    11. Group=prometheus
    12. ExecStart=/monitor/grafana/bin/grafana-server \
    13.     --config=/monitor/grafana/conf/defaults.ini \
    14.     --homepath=/monitor/grafana
    15. Restart=on-failure
    16. RestartSec=10
    17. StandardOutput=syslog
    18. StandardError=syslog
    19. SyslogIdentifier=grafana
    20. Environment=GRAFANA_HOME=/monitor/grafana \
    21.             GRAFANA_USER=prometheus \
    22.             GRAFANA_GROUP=prometheus
    23.  
    24. [Install]
    25. WantedBy=multi-user.target

    1.5 启动服务

    把上述服务启动即可！

    注意：prometheus-webhook-dingtalk 服务需要用下面方式启动：

    1. cd /monitor/prometheus-webhook-dingtalk/
    2. nohup ./prometheus-webhook-dingtalk --web.listen-address="0.0.0.0:8060" --config.file="/monitor/prometheus-webhook-dingtalk/config.yml" &

    二、补充

    2.1 告警规则和 grafana 仪表盘文件下载

    1. [root@hadoop01 ~]# cd /monitor/prometheus/
    2. [root@hadoop01 /monitor/prometheus]# ls
    3. console_libraries  consoles  data  LICENSE  NOTICE  prometheus  prometheus.yml  promtool  rule
    4. [root@hadoop01 /monitor/prometheus]# ls rule/
    5. HDFS.yml  node.yml  spark_master.yml  spark_worker.yml  yarn.yml  zookeeper.yml

    文件下载链接：【免费】prometheus告警规则文件和grafana仪表盘文件资源-CSDN文库

    2.2 服务进程监控脚本

    2.2.1 prometheus-webhook-dingtalk

    1. [root@hadoop01 ~]# cd /monitor/prometheus-webhook-dingtalk/
    2. [root@hadoop01 /monitor/prometheus-webhook-dingtalk]# vim monitor_prometheus_webhook_dingtalk.sh
    3. #!/bin/bash
    4.  
    5. # 获取当前系统时间
    6. current_time=$(date "+%Y-%m-%d %H:%M:%S")
    7.  
    8. # 定义日志文件路径
    9. log_file="/monitor/prometheus-webhook-dingtalk/monitor.log"
    10.  
    11. echo "[$current_time] Checking if prometheus-webhook-dingtalk process is running..." >> $log_file
    13. # 检查进程是否在运行
    14. if ! /usr/bin/pgrep -fx "/monitor/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --web.listen-address=0.0.0.0:8060 --config.file=/monitor/prometheus-webhook-dingtalk/config.yml" >> $log_file; then
    15.     echo "[$current_time] prometheus-webhook-dingtalk process is not running. Starting it now..." >> $log_file
    16.     # 使用绝对路径和 nohup 在后台启动进程
    17.     /usr/bin/nohup /monitor/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --web.listen-address="0.0.0.0:8060" --config.file="/monitor/prometheus-webhook-dingtalk/config.yml" >> /monitor/prometheus-webhook-dingtalk/output.log 2>&1 &
    19. else
    20.     echo "[$current_time] prometheus-webhook-dingtalk process is running." >> $log_file
    21. fi
    22.  
    23. [root@hadoop01 /monitor/prometheus-webhook-dingtalk]# chmod 777 monitor_prometheus_webhook_dingtalk.sh 
    24.  
    25. [root@hadoop01 /monitor/prometheus-webhook-dingtalk]# crontab -e
    26. * * * * * /usr/bin/bash /monitor/prometheus-webhook-dingtalk/monitor_prometheus_webhook_dingtalk.sh

    2.2.2 hive 

    1. (base) [root@hadoop01 ~]# cd /bigdata/apache-hive-3.1.2/
    2. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# vim monitor_hive.sh
    3. #!/bin/bash
    4.  
    5. # 获取当前系统时间
    6. current_time=$(date "+%Y-%m-%d %H:%M:%S")
    7.  
    8. # 定义日志文件路径
    9. log_file_metastore="/bigdata/apache-hive-3.1.2/monitor_metastore.log"
    10. log_file_hs2="/bigdata/apache-hive-3.1.2/monitor_hs2.log"
    11.  
    12. echo "[$current_time] Checking if hive metastore and hs2 processes are running..." 
    13.  
    14. # 检查 Hive Metastore 是否在运行
    15. echo "[$current_time] Checking if hive metastore process is running..." >> $log_file_metastore
    16. if ! /usr/bin/pgrep -f "hive-metastore-3.1.2.jar" >> $log_file_metastore; then
    17.     echo "[$current_time] hive metastore process is not running. Starting it now..." >> $log_file_metastore
    18.     # 使用绝对路径和 nohup 在后台启动进程
    19.     /usr/bin/nohup /bigdata/apache-hive-3.1.2/bin/hive --service metastore >> /bigdata/apache-hive-3.1.2/metastore_output.log 2>&1 &
    20.     # 等待一点时间以确保 metastore 完全启动
    21.     sleep 30
    22. else
    23.     echo "[$current_time] hive metastore process is running." >> $log_file_metastore
    24. fi
    25.  
    26. # 检查 HiveServer2 是否在运行
    27. echo "[$current_time] Checking if hive hs2 process is running..." >> $log_file_hs2
    28. if ! /usr/bin/pgrep -f "HiveServer2" >> $log_file_hs2; then
    29.     echo "[$current_time] hive hs2 process is not running. Starting it now..." >> $log_file_hs2
    30.     # 使用绝对路径和 nohup 在后台启动进程
    31.     /usr/bin/nohup /bigdata/apache-hive-3.1.2/bin/hive --service hiveserver2 >> /bigdata/apache-hive-3.1.2/hs2_output.log 2>&1 &
    32. else
    33.     echo "[$current_time] hive hs2 process is running." >> $log_file_hs2
    34. fi
    35.  
    36. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# chmod 777 montior_metastore.sh 
    37.  
    38. (base) [root@hadoop01 /bigdata/apache-hive-3.1.2]# crontab -e
    39. * * * * * /usr/bin/bash /bigdata/apache-hive-3.1.2/monitor_hive.sh

    2.2.3 日志切割 

    1. [root@hadoop01 ~]# vim /etc/logrotate.d/prometheus-webhook-dingtalk
    2. /monitor/prometheus-webhook-dingtalk/monitor.log \
    3. /bigdata/apache-hive-3.1.2/monitor_metastore.log \
    4. /bigdata/apache-hive-3.1.2/monitor_hs2.log {
    5.     daily
    6.     rotate 7
    7.     size 150M
    8.     compress
    9.     maxage 30
    10.     missingok
    11.     notifempty
    12.     create 0644 root root
    13.     copytruncate
    14. }
    15.  
    16. # 测试调式 logrotate 配置
    17. [root@hadoop01 ~]# logrotate -d /etc/logrotate.d/prometheus-webhook-dingtalk
    18. # 手动执行日志轮换
    19. logrotate -f /etc/logrotate.d/prometheus-webhook-dingtalk

    2.3 常用命令 

    1. # 检查 prometheus 配置文件，包括告警规则文件
    2. [root@hadoop01 ~]# cd /monitor/prometheus
    3. ./promtool check config prometheus.yml
    4.  
    5. # 重启 prometheus 配置
    6. curl -X POST http://localhost:9090/-/reload
    8. # 测试发送信息到机器人
    9. curl 'https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxx' \
    10.  -H 'Content-Type: application/json' \
    11.  -d '{"msgtype": "text","text": {"content":"我就是我, 是不一样的烟火"}}'

    2.4 参考文档

    • Hadoop 官方监控指标：https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/Metrics.html

    • 阿里云监控指标：https://help.aliyun.com/zh/emr/emr-on-ecs/user-guide/hdfs-metrics?spm=a2c4g.11186623.0.0.11ba6daalnmBWn

    • 阿里云 grafana 仪表盘：https://help.aliyun.com/document_detail/2326798.html?spm=a2c4g.462292.0.0.4c4c5d35uXCP6k#section-1bn-bzq-fw3

    • jmx_exporter 配置文件参考：https://github.com/prometheus/jmx_exporter/tree/main/example_configs

    • 钉钉机器人文档：https://open.dingtalk.com/document/robots/custom-robot-access

    2.5 内网环境

    2.5.1 nginx（公网）

    需要一台可以访问公网的 nginx 服务器来代理钉钉 api：

    1. [root@idc-master-02 ~]# cat /etc/nginx/conf.d/uat-prometheus-webhook-dingtalk.conf
    2. server {
    3.     listen 30080;
    4.  
    5.     location /robot/send {
    6.         proxy_pass https://oapi.dingtalk.com;
    7.         proxy_set_header Host oapi.dingtalk.com;
    8.         proxy_set_header X-Real-IP $remote_addr;
    9.         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    10.         proxy_set_header X-Forwarded-Proto $scheme;
    11.     }
    12. }

    2.5.2 config 

    1. [root@localhost-13 ~]# cat /opt/prometheus-webhook-dingtalk/config.yml 
    2.  
    3. targets:
    4.   webhook1:
    5.     url: http://10.0.4.11:30080/robot/send?access_token=0d6c5dc25fa3f79cf2f83c92705fe4594dcc5b3xxx
    6.     secret: SECecdbfff858ab8f3195dc34b7e225fee93xxx
    7.     message:
    8.       title: '{{ template "ops.title" . }}'
    9.       text: '{{ template "ops.content" . }}'

    2.5.3 /etc/hosts 

    1. [root@localhost-13 ~]# cat /etc/hosts
    2. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    3. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    4. 10.0.4.11 oapi.dingtalk.com
  • 相关阅读:
    sql 5
    c++ 类的实例化顺序
    Json的使用，以及@JsonProperty和@JSONFiled注解的混淆
    挑战 Google 搜索？OpenAI 发布最强 AI 对话系统 ChatGPT
    数字化时代，企业转型发展可能会有哪些特征？
    百度网盘的扩容
    《LeetCode力扣练习》代码随想录——二叉树（平衡二叉树---Java）
    MindSponge分子动力学模拟——定义Collective Variables（2024.02）
    marven项目打包第三方jar包
    用golang container/list 实现队列并控制并发
  • 原文地址：https://blog.csdn.net/weixin_46560589/article/details/133811785