Centos7.9 Docker-compose方式安装Logstash7

安装Docker

curl -sSL https://get.daocloud.io/docker | sh

systemctl start docker

安装docker-compose

curl -L https://get.daocloud.io/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

Logstash7 Compose文件

vim logstash.yaml

logstash7:
  image: harbor.ikun.blog:444/chair/logstash:7.16.3
  volumes:
  - /opt/elk/logstash/config/conf/:/usr/share/logstash/pipeline/
  - /opt/elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
  - /opt/elk/logstash/config/log4j2.properties:/usr/share/logstash/config/log4j2.properties
  - /opt/elk/log/logstash/:/usr/share/logstash/logs/

创建目录并给予权限

mkdir -p /opt/elk/logstash/config/conf/
mkdir -p /opt/elk/log/logstash/
chmod 777 -R /opt/elk/logstash/config/conf/
chmod 777 -R /opt/elk/log/logstash/

下载logstash.yml，log4j2.properties，logstash.conf文件,并mv到指定位置

wget https://www.ikun.blog/downloads/logstash/logstash.yml
wget https://www.ikun.blog/downloads/logstash/log4j2.properties
wget https://www.ikun.blog/downloads/logstash/conf/logstash.conf
mv logstash.conf /opt/elk/logstash/config/conf/
mv log4j2.properties /opt/elk/logstash/config/
mv logstash.yml /opt/elk/logstash/config/

使用上面编写的Compose文件启动容器

docker-compose -f logstash.yaml up -d

Logstash7 Kafka之间持续同步数据

Logstash7配置文件

input {
        kafka {
                bootstrap_servers => ["192.168.x.x:9092,192.168.x.x:9092,192.168.x.x:9092"]
                group_id => "profile"
                client_id => "logstash-kafka-1"
                auto_offset_reset => "latest"
                consumer_threads => 8
                topics => ["mapping_event_json"]
                codec => "json"
        }
}

output {
              kafka  {
                bootstrap_servers => "192.168.x.x:9092"
                topic_id => "user_event_json"
                }
}

使用logstash跨版本迁移es

logstash配置文件

input {
  elasticsearch {
    hosts => ["172.16.xx.xx:9200", "172.16.xx.xx:9200", "172.16.xx.xx:9200"]
    user => "xxx"
    password => "xxx"
    index => "*"
    codec => "json"
    docinfo => true
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["172.16.xx.xx:9200", "172.16.xx.xx:9200", "172.16.xx.xx:9200"]
    user => "xxx"
    password => "xxx"
    document_type => "%{[@metadownloads][_type]}"    #type和来源相同
    document_id => "%{[@metadownloads][_id]}"        #id和原来相同
    index => "%{[@metadownloads][_index]}"   #index和来源相同
  }
}