外观
Elasticsearch
约 2917 字大约 10 分钟
2025-02-20
Docker-Compose安装ES7集群
安装Docker
curl -sSL curl -fsSL https://get.docker.com | sh
启动Docker
systemctl enable docker && systemctl start docker
安装Docker-Compose
curl -SL https://github.com/docker/compose/releases/download/v2.34.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
ES01
编写elasticsearch 01Docker-Compose文件
vim elasticsearch.yaml
version: '3.8'
services:
es:
image: harbor.ikun.blog/chair/elasticsearch:7.16.2
container_name: es
restart: always
environment:
- node.name=ES01 # 节点名称,集群模式下每个节点名称唯一
- network.host=0.0.0.0 # 设置绑定的ip地址,可以是ipv4或ipv6的,默认为0.0.0.0,即本机
- network.publish_host=10.203.16.13 # 用于集群内各机器间通信,对外使用,其他机器访问本机器的es服务,一般为本机宿主机IP
- discovery.seed_hosts=10.203.16.13,192.168.6.203,192.168.7.232 # 写入候选主节点的设备地址
- cluster.initial_master_nodes=ES01 # ES8.x需要指定初始主节点列表
- cluster.name=docker-cluster # 集群名称,相同名称为一个集群, 三个es节点须一致
- bootstrap.memory_lock=true # 内存交换的选项,官网建议为true
- http.port=9200
- transport.port=9300
- ES_JAVA_OPTS=-Xms2g -Xmx2g
ulimits: # 栈内存的上限
memlock:
soft: -1 # 不限制
hard: -1 # 不限制
volumes:
- /opt/elasticsearch/config/es.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /data/es:/usr/share/elasticsearch/data
- /downloads/es:/usr/share/elasticsearch/downloads
ports:
- 9200:9200
- 9300:9300
下载Elastic配置文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
wget https://www.ikun.blog/downloads/es/es.yml
创建文件夹
mkdir -p /opt/elasticsearch/config/
mkdir -p /opt/log/elasticsearch
mkdir -p /downloads/es
mkdir -p /data/es
mv配置文件到指定文件夹
mv log4j2.properties es.yml /opt/elasticsearch/config/
给予权限
chmod 777 -R /downloads/es/
chmod 755 -R /opt/elasticsearch/config/
chmod 755 -R /opt/log/elasticsearch/
chown -R 1000:1000 /opt/elasticsearch/config/
chown -R 1000:1000 /opt/log/elasticsearch/
chown -R 1000:1000 /data/es
启动容器
docker-compose -f elasticsearch.yaml up -d
配置密码&生成密钥
docker exec -it elasticSearch /bin/bash
/usr/share/elasticsearch/bin/elasticsearch-certutil ca
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
注
两条命令一路回车就行
mv elastic-* /usr/share/elasticsearch/config/
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
退出容器,修改es配置文件,取消注释项
exit
vim /opt/elasticsearch/config/es.yml
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,Content-Type
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
重启es容器
docker restart es
ES02
编写elasticsearch 02Docker-Compose文件
vim elasticsearch.yaml
version: '3.8'
services:
es:
image: harbor.ikun.blog/chair/elasticsearch:7.16.2
container_name: es
restart: always
environment:
- node.name=ES02 # 节点名称,集群模式下每个节点名称唯一
- network.host=0.0.0.0 # 设置绑定的ip地址,可以是ipv4或ipv6的,默认为0.0.0.0,即本机
- network.publish_host=192.168.6.203 # 用于集群内各机器间通信,对外使用,其他机器访问本机器的es服务,一般为本机宿主机IP
- discovery.seed_hosts=192.168.7.164,192.168.6.203,192.168.7.232 # 写入候选主节点的设备地址
- cluster.name=docker-cluster # 集群名称,相同名称为一个集群, 三个es节点须一致
- bootstrap.memory_lock=true # 内存交换的选项,官网建议为true
- node.master=true
- node.downloads=true
- http.port=9200
- transport.tcp.port=9300
- ES_JAVA_OPTS=-Xms2g -Xmx2g
ulimits: # 栈内存的上限
memlock:
soft: -1 # 不限制
hard: -1 # 不限制
volumes:
- /opt/elasticsearch/config/es.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /downloads/es:/usr/share/elasticsearch/downloads
- /data/es:/usr/share/elasticsearch/data
- /opt/elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
ports:
- 9200:9200
- 9300:9300
下载Elastic配置文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
wget https://www.ikun.blog/downloads/es/es.yml
创建文件夹并给予权限
```bash
mkdir -p /opt/elasticsearch/config/
mkdir -p /opt/log/elasticsearch
mkdir -p /downloads/es
mkdir -p /data/es
mv配置文件到指定文件夹
mv log4j2.properties es.yml /opt/elasticsearch/config/
复制第一台ES生成的秘钥到/opt/elasticsearch/config/目录下
docker cp elasticsearch:/usr/share/elasticsearch/config/elastic-certificates.p12 /opt/elasticsearch/config/
给予权限
chmod 777 -R /downloads/es/
chmod 755 -R /opt/elasticsearch/config/
chmod 755 -R /opt/log/elasticsearch/
chown -R 1000:1000 /opt/elasticsearch/config/
chown -R 1000:1000 /opt/log/elasticsearch/
chown -R 1000:1000 /data/es
修改es配置文件,取消注释项
vim /opt/elasticsearch/config/es.yml
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,Content-Type
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
启动es容器
docker-compose -f elasticsearch.yaml up -d
ES03
编写elasticsearch 03Docker-Compose文件
vim elasticsearch.yaml
version: '3.8'
services:
es:
image: harbor.ikun.blog/chair/elasticsearch:7.16.2
container_name: elasticsearch
restart: always
environment:
- node.name=ES03 # 节点名称,集群模式下每个节点名称唯一
- network.host=0.0.0.0 # 设置绑定的ip地址,可以是ipv4或ipv6的,默认为0.0.0.0,即本机
- network.publish_host=192.168.7.232 # 用于集群内各机器间通信,对外使用,其他机器访问本机器的es服务,一般为本机宿主机IP
- discovery.seed_hosts=192.168.7.164,192.168.6.203,192.168.7.232 # 写入候选主节点的设备地址
- cluster.name=docker-cluster # 集群名称,相同名称为一个集群, 三个es节点须一致
- bootstrap.memory_lock=true # 内存交换的选项,官网建议为true
- node.master=true
- node.downloads=true
- http.port=9200
- transport.tcp.port=9300
- ES_JAVA_OPTS=-Xms2g -Xmx2g # 设置内存,如内存不足,可以尝试调低点
ulimits: # 栈内存的上限
memlock:
soft: -1 # 不限制
hard: -1 # 不限制
volumes:
- /opt/elasticsearch/config/es.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /downloads/es:/usr/share/elasticsearch/downloads
- /data/es:/usr/share/elasticsearch/data
- /opt/elasticsearch/config/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
ports:
- 9200:9200
- 9300:9300
下载Elastic配置文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
wget https://www.ikun.blog/downloads/es/es.yml
创建文件夹并给予权限
mkdir -p /opt/elasticsearch/config/
mkdir -p /opt/log/elasticsearch
mkdir -p /downloads/es
mkdir -p /data/es
mv配置文件到指定文件夹
mv log4j2.properties es.yml /opt/elasticsearch/config/
复制第一台ES生成的密钥文件到/opt/elasticsearch/config/目录下 给予权限
chmod 777 -R /downloads/es/
chmod 755 -R /opt/elasticsearch/config/
chmod 755 -R /opt/log/elasticsearch/
chown -R 1000:1000 /opt/elasticsearch/config/
chown -R 1000:1000 /opt/log/elasticsearch/
chown -R 1000:1000 /data/es
修改es配置文件,取消注释项
vim /opt/elasticsearch/config/es.yml
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,Content-Type
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
复制第一台ES生成的秘钥到/opt/elasticsearch/config/目录下
docker cp elasticsearch:/usr/share/elasticsearch/config/elastic-certificates.p12 /opt/elasticsearch/config/
启动es容器
docker-compose -f elasticsearch.yaml up -d
CURL常用查询
查询索引的settings
curl http://x.x.x.x:9200/_settings\?pretty
修改索引的settings
curl -XPUT -d '{ "key": value }' 'localhost:9200/index/_settings'
使用Docker-Compose部署ElasticSearch 8.17.4
ES01
安装Docker
curl -sSL curl -fsSL https://get.docker.com | sh
安装Docker-Compose
curl -SL https://github.com/docker/compose/releases/download/v2.34.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
创建文件夹
mkdir -p /opt/log/elasticsearch
mkdir -p /data/es
mkdir -p /opt/elasticsearch/config/certs/
编写elasticsearchDocker-Compose文件
vim elasticsearch.yaml
version: '3.7'
services:
elasticsearch:
image: harbor.ikun.blog/chair/elasticsearch:8.17.4
container_name: es
environment:
- node.name=es01
- network.host=0.0.0.0
- cluster.name=chair
- network.publish_host=10.203.0.17
- discovery.seed_hosts=10.203.0.18,10.203.0.19 # 其它两节点的IP地址
- cluster.initial_master_nodes=10.203.0.17,10.203.0.18,10.203.0.19
- ELASTIC_PASSWORD=Wa33Wa331!
- xpack.security.enabled=true
# 启用传输层SSL
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.client_authentication=required
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
volumes:
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /data/es:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
ulimits:
memlock:
soft: -1
hard: -1
下载log4j2.properties文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
mv log4j2.properties /opt/elasticsearch/config/
启动一台临时ES容器用作生成证书
docker run --name es -itd -v /opt/elasticsearch/config/certs:/certs
docker exec -it es /bin/bash
生成ca证书
bin/elasticsearch-certutil ca
生成X.509证书和私钥
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
cp elastic-certificates.p12 /certs/
退出并删除此临时容器
exit
docker stop es && docker rm es
给予权限
chown -R 1000:1000 /opt/elasticsearch
chown -R 1000:1000 /opt/log/elasticsearch
chown -R 1000:1000 /data/es
启动ES容器
docker-compose -f elasticsearch.yaml up -d
ES02
安装Docker
curl -sSL curl -fsSL https://get.docker.com | sh
安装Docker-Compose
curl -SL https://github.com/docker/compose/releases/download/v2.34.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
创建文件夹
mkdir -p /opt/log/elasticsearch
mkdir -p /data/es
mkdir -p /opt/elasticsearch/config/certs/
编写elasticsearchDocker-Compose文件
vim elasticsearch.yaml
version: '3.7'
services:
elasticsearch:
image: harbor.ikun.blog/chair/elasticsearch:8.17.4
container_name: es
environment:
- node.name=es02
- network.host=0.0.0.0
- cluster.name=chair
- network.publish_host=10.203.0.18
- discovery.seed_hosts=10.203.0.17,10.203.0.19 # 其它两节点的IP地址
- cluster.initial_master_nodes=10.203.0.17,10.203.0.18,10.203.0.19
- xpack.security.enabled=true
# 启用传输层SSL
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.client_authentication=required
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
volumes:
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /data/es:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
ulimits:
memlock:
soft: -1
hard: -1
下载log4j2.properties文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
mv log4j2.properties /opt/elasticsearch/config/
把第一台服务器上的证书复制到另外两台服务器
scp ./elastic-certificates.p12 root@10.203.0.18:/opt/elasticsearch/config/certs/elastic-certificates.p12
相关信息
在ES01执行
给予权限
chown -R 1000:1000 /opt/elasticsearch
chown -R 1000:1000 /opt/log/elasticsearch
chown -R 1000:1000 /data/es
启动ES容器
docker-compose -f elasticsearch.yaml up -d
ES03
安装Docker
curl -sSL curl -fsSL https://get.docker.com | sh
安装Docker-Compose
curl -SL https://github.com/docker/compose/releases/download/v2.34.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
创建文件夹
mkdir -p /opt/log/elasticsearch
mkdir -p /data/es
mkdir -p /opt/elasticsearch/config/certs/
编写elasticsearchDocker-Compose文件
vim elasticsearch.yaml
version: '3.7'
services:
elasticsearch:
image: harbor.ikun.blog/chair/elasticsearch:8.17.4
container_name: es
environment:
- node.name=es03
- network.host=0.0.0.0
- cluster.name=chair
- network.publish_host=10.203.0.19
- discovery.seed_hosts=10.203.0.17,10.203.0.18 # 其它两节点的IP地址
- cluster.initial_master_nodes=10.203.0.17,10.203.0.18,10.203.0.19
- xpack.security.enabled=true
# 启用传输层SSL
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.client_authentication=required
- xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
volumes:
- /opt/elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
- /opt/elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12
- /opt/log/elasticsearch:/usr/share/elasticsearch/logs
- /data/es:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
ulimits:
memlock:
soft: -1
hard: -1
下载log4j2.properties文件
wget https://www.ikun.blog/downloads/es/log4j2.properties
mv log4j2.properties /opt/elasticsearch/config/
把第一台服务器上的证书复制到这台服务器
scp ./elastic-certificates.p12 root@10.203.0.19:/opt/elasticsearch/config/certs/elastic-certificates.p12
给予权限
chown -R 1000:1000 /opt/elasticsearch
chown -R 1000:1000 /opt/log/elasticsearch
chown -R 1000:1000 /data/es
启动ES容器
docker-compose -f elasticsearch.yaml up -d
使用Elasticdump迁移数据并修改Mapping或者Settings
因为客户环境不能通外网,所以此文档是以离线方式部署
相关信息
以下操作是在通外网的机器做的
下载Nodejs安装包
wget https://nodejs.org/dist/v16.14.0/node-v16.14.0-linux-x64.tar.xz
解压
tar xvf node-v16.14.0-linux-x64.tar.xz
建立软连接
ln -s /root/node-v16.14.0-linux-x64/bin/node /usr/bin/node
ln -s /root/node-v16.14.0-linux-x64/bin/npm /usr/bin/npm
确认安装是否成功
node -v
npm -v
安装npm-pack-all
npm install -g npm-pack-all
安装Elasticdump
npm install elasticdump -g
进入安装目录:
cd /root/node-v16.14.0-linux-x64/lib/node_modules/elasticdump
使用npm-pack-all进行打包
npm-pack-all
提示
执行时如果报错找不到命令,那就去安装目录下执行:/root/node-v16.14.0-linux-x64/bin/ 执行npm-pack-all后会生成elasticdump压缩包,把这两个压缩包(Nodejs,Elasticdump)放到目标服务器上去
相关信息
以下操作在目标服务器执行
索引备份
curl -XPOST -u "elastic:xxxx" -H "Content-Type: application/json" '172.17.xx.xx:9200/_reindex' -d '{"source": {"index": "xxxxxxx"},"dest": {"index": "xxx_back"}}'
提示
mapping如果是es5迁移到es7那就要事先创建索引和mapping。相同版本之间新index不用事先创建。
Kibana Console创建索引并设置mapping
PUT /test_index_202210111646?include_type_name=true
{
"mappings": {
}
}
解压Nodejs包
tar xvf node-v16.14.0-linux-x64.tar.xz
设置软连接
ln -s /root/node-v16.14.0-linux-x64/bin/node /usr/bin/node
ln -s /root/node-v16.14.0-linux-x64/bin/npm /usr/bin/npm
安装Elasticdump
npm install /root/elasticdump-6.88.0.tgz
进入elasticdump/bin
cd /root/node-v16.14.0-linux-x64/bin/node_modules/elasticdump/bin
接下来执行Elasticdump命令来进行迁移
./elasticdump --input=http://elastic:elastic@192.168.89.181:9200/my_video_index --output=http://elastic:elastic@192.168.89.180:9200/my_video_index_dump --type=settings
./elasticdump --input=http://elastic:elastic@192.168.89.181:9200/my_video_index --output=http://elastic:elastic@192.168.89.180:9200/my_video_index_dump --type=mapping
./elasticdump --input=http://elastic:elastic@192.168.89.181:9200/my_video_index --output=http://elastic:elastic@192.168.89.180:9200/my_video_index_dump --type=downloads
提示
参数:--limit 用于限定每一批量操作时进行迁移的文档对象数目,默认值是100,也就是每一次从SOURCE到DESTINATION文档对象数量。 需要迁移哪项就执行那条命令,最好是mapping在downloads前面迁移。可以从不同的index导入数据,比如说从1导入settings,从2导入mapping
使用curl迁移数据并修改Mapping
注意
此方法有个弊端:无法迁移,迁移中进入的数据
创建备份索引,并配置Mapping
curl -XPUT -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/bak' -d '{"mappings": }'
将原始索引数据复制到备份索引,确认返回状态
curl -XPOST -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/_reindex' -d '{"source": {"index": "indexName"},"dest": {"index": "IndexName"}}'
检查确认,备份索引含有数据,总数量与原始索引总数量一致
curl -XPOST -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/IndexName/_search' -d '{"track_total_hits":true,"query":{"match_all":{}}}'
curl -XPOST -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/IndexName/_search' -d '{"track_total_hits":true,"query":{"match_all":{}}}'
经第三步反复确认,数据总数一致后,删除原始索引(务必确认多次,保证数据都被复制成功)
curl -XDELETE -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/IndexName'
重新创建原始索引,含mapping
curl -XPUT -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/bak' -d '{"mappings": }'
将备份索引复制到原始索引
curl -XPOST -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/_reindex' -d '{"source": {"index": "IndexName"},"dest": {"index": "IndexName"}}'
检查确认,原始索引含有数据,数量与原先一致
curl -XPOST -H 'Content-Type:application/json' 'elastic:xxxx@x.x.x.x:9200/IndexName/_search' -d '{"track_total_hits":true,"query":{"match_all":{}}}'
删除备份索引
curl -XDELETE 'elastic:xxxx@x.x.x.x:9200/IndexName'
以时间为单位删除索引数据
完整命令
curl -u xxx:xxx -H'Content-Type: application/json' -d'{"query": {"range": {"@timestamp": {"lt": "now-7d","format": "epoch_millis"}}}}' -XPOST "http://127.0.0.1:9200/nginx-access-log*/_delete_by_query?pretty"
参数解析
-u:指定用户名和密码
-H:设置请求头
-XPOST:指定用POST方式请求
-d:指定发送的body内容
body内容解析
{
"query": {
"range": { // 范围
"@timestamp": { // 时间段
"lt": "now-7d", // lt是小于(<),lte是小于等于(<=),gt是大于(>),gte是大于等于(>=),now-7d是当前时间减7天
"format": "epoch_millis"
}
}
}
}
注意
如果索引很大而且没做分片,那么就会占用很多io和网络,而且删除的时间也会很长,200G,357337097多条数据删了四五天还没删完。