EFK(ELK)日志收集

1 说明

部署方式：Docker-compose。
Docker版本：Server Version: 19.03.12
通过elasticsearch,filebeat,kibana收集windows系统日志
通过elasticsearch,logstash,kibana收集嘉定项目开发环境tomcat日志，便于开发分析问题
部署位置：192.168.0.241
访问方式：http://192.168.0.241:5601/

2 部署es

2.1 创建本地目录

mkdir -p /opt/docker_volume/elasticsearch/data

2.1 创建es的docker-compose.yaml

cat <<EOF>> /opt/docker_volume/elasticsearch/docker-compose.yamlversion: "3.1"services:  elasticsearch:    image: elasticsearch:6.5.4    restart: always    container_name: es    privileged: true    volumes:      - /opt/docker_volume/elasticsearch/data:/usr/share/elasticsearch/data    ports:      - 9200:9200  kibana:    image: kibana:6.5.4    restart: always    container_name: kibana    ports:      - 5601:5601    environment:      - elasticsearch_url=http://es:9200    depends_on:      - elasticsearch

2.2 设置防火墙

fiewall-cmd --zone=public --add-port=9200/tcp --permanentfiewall-cmd --zone=public --add-port=5601/tcp --permanent

2.3 启动容器

cd /opt/docker_volume/elasticsearch/ && docker-compose up -d

3 部署filebeat、logstash客户端

3.1 windows部署filebeat客户端

windows下载与es对应的filebeat版本，解压到C:\根目录，修改winlogbeat.yml

winlogbeat.event_logs:    - name: Application    - name: Security      level: critical, error, warning    - name: Systemsetup.template.settings:  index.number_of_shards: 1output.elasticsearch:  hosts: ["192.168.0.241:9200"]  index: "windows_log-%{+yyyy}"setup.template.name: "windows"setup.template.pattern: "windows_*"setup.template.enabled: falsesetup.template.overwrite: true

3.2 linux中logstash或者运行Docker容器

嘉定项目运行在容器中，日志目录已挂载到本地目录，logstash的容器只需挂载日志目录即可

chmod 644 /var/log/messages && chmod 644 /var/log/secure && tomcat的日志目录 docker run \-d \--restart=always \--name logstash \--volumes-from tomcat \-v /var/log/messages:/var/log/messages:ro \-v /var/log/secure:/var/log/secure:ro \-v /etc/logstash.conf:/etc/logstash.conf \logstash:6.5.4 -f /etc/logstash.conf

3.2.1 编辑 logstash配置文件

vim /etc/logstash.conf input {    file {        path => "/var/log/messages"        type => "systemlog"        start_position => "beginning"        stat_interval => "3"    }    file {        path => "/var/log/secure"        type => "securelog"        start_position => "beginning"        stat_interval => "3"    }    file {        path => "/usr/local/tomcat/logs/catalina.out"        type => "catalina.out"    }}output {    if [type] == "systemlog" {         elasticsearch {            hosts => ["192.168.0.241:9200"]            index => "system-log-%{+YYYY.MM}"        }    }    if [type] == "securelog" {         elasticsearch {            hosts => ["192.168.0.241:9200"]            index => "secure-log-%{+YYYY.MM}"        }    }    if [type] == "catalina.out" {         elasticsearch {            hosts => ["192.168.0.241:9200"]            index => "catalina.out-%{+YYYY.MM}"        }    }}