ELK集群 日志中心集群、kafka、logstash

ES：用来日志存储

Logstash:用来日志的搜集，进行日志格式转换并且传送给别人（转发）

Kibana:主要用于日志的展示和分析

kafka

Filebeat:搜集文件数据

es-1

本地解析

vi /etc/hosts

scp /etc/hosts es-2:/etc/hosts

scp /etc/hosts es-3:/etc/hosts

 yum -y install wget

安装配置jdk

wget 8u191

scp -3

tar xf jdk-8u191-linux-x64.tar.gz -C /usr/local/

[root@es-1 ~]# vim /etc/profile
JAVA_HOME=/usr/local/java
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH
[root@es-1~]# source /etc/profile

安装配置ES

useradd elsearch

ssh es-2 useradd elsearch

ssh es-3 useradd elsearch

echo "123456" | passwd --stdin "elsearch"

wget els包

vim /usr/local/elasticsearch/config/elasticsearch.yml

（都删了）

cluster.name: xingdiancloud-elk
node.name: es-1
node.master: true
node.data: true
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["es-1", "es-2","es-3"]
discovery.zen.ping_timeout: 150s
discovery.zen.fd.ping_retries: 10
client.transport.ping_timeout: 60s
http.cors.enabled: true
http.cors.allow-origin: "*"

创建ES数据及日志存储目录

mkdir -p /data/elasticsearch/{logs,data}

scp /usr/local/elasticsearch/config/elasticsearch.yml es-3:/usr/local/elasticsearch/config/elasticsearch.yml 

修改安装目录及存储目录权限

系统优化

vi /etc/security/limits.conf

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

sysctl -p

su - elsearch -c "cd /usr/local/elasticsearch && nohup bin/elasticsearch &"

es-2

echo "123456" | passwd --stdin "elsearch"

mkdir -p /data/elasticsearch/{logs,data}

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

sysctl -p

su - elsearch -c "cd /usr/local/elasticsearch && nohup bin/elasticsearch &"

ss -antpl  //看9200

es-3

echo "123456" | passwd --stdin "elsearch"

mkdir -p /data/elasticsearch/{logs,data}

vim /usr/local/elasticsearch-6.5.4/config/elasticsearch.yml   (改名字)

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

sysctl -p

su - elsearch -c "cd /usr/local/elasticsearch && nohup bin/elasticsearch &"

ss -antpl  //看9200

浏览器访问：IP：9300

kibana

本地解析

vi /etc/hosts  

IP  es-1、2、3

上传kibana包

vi /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "172.16.244.28"
elasticsearch.url: "http://172.16.244.25:9200"
kibana.index: ".kibana"

jobs

ss -antpl  看端口5600

浏览器访问  IP ：5601

Nginx安装反向代理

安装

yum -y install nginx httpd-tools

验证节点与kibana时间

三：logstash部署(新机器)

安装jdk8

vi /etc/hosts

ip es-1

vim /etc/profile
JAVA_HOME=/usr/local/java
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH
 source /etc/profile

java -version

2.安装logstash

上传logstash包

vi /opt/messages.conf

执行logstash

logstash -f /opt/messages.conf

数据的测试

1）基本的输入输出

logstash ：

终端输入

es查看

Kibana上

管理 -------------------下一步-------

监控nginx logs 

选RPM

es-1、2、3

新机子

上传filebeat.rpm包

vi /etc/hosts

ip +es-1

vi nginx.yml

yum -y install nginx

systemctl start nginx 

curl localhost

vu nginx.yml

生效

date(时间统一)

排错中(不出错不做)

干掉es

 

yum -y install psmisc

systemctl restart filebeat