欢迎您访问 最编程 本站为您分享编程语言代码，编程技术文章！

热门搜索/Hot Search

您现在的位置是： 首页

ETCD 部署说明

最编程 2024-06-22 09:32:39
...

记录一下用systemd管理TLS ETCD集群的服务, 及对NODE结点的扩缩容

环境准备

准备三台机器，配置信息如下:

操作系统 IP地址 主机名
CentOS 8.5.2111 192.168.14.188 c8k1
CentOS 8.5.2111 192.168.14.189 c8k2
CentOS 8.5.2111 192.168.14.190 c8k3

在每个机器上运行下面命令, 下载并安装最新版本的ETCD(当前版本是3.5.4):

ETCD_VER=v3.5.4
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local
rm -f /usr/local/etcd-${ETCD_VER}-linux-amd64.tar.gz
ln /usr/local/etcd-${ETCD_VER}-linux-amd64/etcd  /usr/local/bin/etcd
ln /usr/local/etcd-${ETCD_VER}-linux-amd64/etcdctl  /usr/local/bin/etcdctl

测试当前ETCD的信息:

[root@c8k2 ~]# etcd version 
{"level":"info","ts":"2022-05-26T06:21:15.131+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","version"]}
{"level":"warn","ts":"2022-05-26T06:21:15.131+0800","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"'version' is not a valid flag"}
[root@c8k2 ~]# etcdctl version 
etcdctl version: 3.5.4
API version: 3.5

在每台机器上, 创建必要的目录:

mkdir -p /var/data/etcd/
mkdir -p /etc/etcd/pki/

配置TLS ETCD高可用集群

把c8k1和c8k2组成一个TLS ETCD的高可用集群.

生成证书

创建脚本

内容如下:

#!/usr/bin/bash

export NAME1=c8k2
export ADDRESS1=192.168.14.189

export NAME2=c8k1
export ADDRESS2=192.168.14.188

export NAME3=c8k3
export ADDRESS3=192.168.14.190

days=3650

cat > openssl.conf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = $NAME1
DNS.2 = $NAME2
DNS.3 = $NAME3
IP.1 = 127.0.0.1
IP.2 = $ADDRESS1
IP.3 = $ADDRESS2
IP.4 = $ADDRESS3
EOF

[ -f ca.key ] || openssl genrsa -out ca.key 2048
[ -f ca.crt ] || openssl req -x509 -new -nodes -key ca.key -subj "/CN=etcd-ca" -days ${days} -out ca.crt

[ -f client.key ] || openssl genrsa -out client.key 2048
[ -f client.csr ] || openssl req -new -key client.key -subj "/CN=kube-etcd" -out client.csr -config openssl.conf
[ -f client.crt ] || openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days ${days} -extensions v3_req  -extfile openssl.conf


[ -f peer.key ] || openssl genrsa -out peer.key 2048
[ -f peer.csr ] || openssl req -new -key peer.key -subj "/CN=kube-etcd-peer" -out peer.csr -config openssl.conf
[ -f peer.crt ] || openssl x509 -req -in peer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out peer.crt -days ${days} -extensions v3_req  -extfile openssl.conf

运行脚本

把上面的脚本放在任一台机器中的/etc/etcd/pki/目录下, 直接运行. 运行脚本之后, 在/etc/etcd/pki/目录下的文件列表如下:

[root@c8k1 pki]# ls
ca.crt  ca.key  ca.srl  client.crt  client.csr  client.key  gen_etcdca.sh  openssl.conf  peer.crt  peer.csr  peer.key

把/etc/etcd/pki目录, 并把此目录分别拷贝到另外三台机器中的/etc/etcd/目录下. 以保证四台机器中的/etc/etcd/pki/目录中的内容是一样的.

创建etcd服务

创建/usr/lib/systemd/system/etcd.service文件, 内容如下:

[Unit]
Description=Etcd Server

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

创建配置文件

创建/etc/etcd/etcd.conf文件, 内容如下:

[root@c8k2 ~]# cat /etc/etcd/etcd.conf
ETCD_NAME="c8k2"
ETCD_DATA_DIR="/var/data/etcd/"
ETCD_LISTEN_PEER_URLS="https://192.168.14.189:32380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.14.189:32379,http://127.0.0.1:32379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.14.189:32380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.14.189:32379"
ETCD_INITIAL_CLUSTER="c8k1=https://192.168.14.188:32380,c8k2=https://192.168.14.189:32380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt
ETCD_CERT_FILE=/etc/etcd/pki/client.crt
ETCD_KEY_FILE=/etc/etcd/pki/client.key
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

注:

  1. 在c8k1机器也要创建一样的文件.
  2. 在配置文件中, ETCD_NAME要修改成对应主机名.
  3. 在配置文件中, ETCD_LISTEN_PEER_URLS, ETCD_INITIAL_ADVERTISE_PEER_URLS和ETCD_ADVERTISE_CLIENT_URLS三项要修改对应的IP地址.

运行服务

运行下面命令, 以重载服务配置, 并开机启动服务.

systemctl daemon-reload
systemctl enable etcd

启动服务, 使etcd作为系统服务运行.

systemctl start etcd

注:

  1. c8k1, c8k2两台机器要同时运行, 否则启动服务的命令会一直hang住.

检查集群状态

运行下面命令:

[root@c8k2 etcd]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.188:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member list 
30562fa23a71c781, started, c8k1, https://192.168.14.188:32380, https://192.168.14.188:32379, false
c7a9cd6ce683f76a, started, c8k2, https://192.168.14.189:32380, https://192.168.14.189:32379, false
[root@c8k2 etcd]# etcdctl --endpoints=https://192.168.14.188:32379,https://192.168.14.189:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key endpoint status 
https://192.168.14.188:32379, 30562fa23a71c781, 3.5.4, 20 kB, false, false, 2, 6, 6, 
https://192.168.14.189:32379, c7a9cd6ce683f76a, 3.5.4, 20 kB, true, false, 2, 6, 6,

集群扩缩容

集群扩容

把c8k3加到上面的集群里面.

创建配置文件

在c8k3中, 创建一个配置/etc/etcd/etcd.conf. 内容如下:

[root@c8k3 ~]# cat /etc/etcd/etcd.conf
ETCD_NAME="c8k3"
ETCD_DATA_DIR="/var/data/etcd/"
ETCD_LISTEN_PEER_URLS="https://192.168.14.190:32380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.14.190:32379,http://127.0.0.1:32379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.14.190:32380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.14.190:32379"
ETCD_INITIAL_CLUSTER="c8k1=https://192.168.14.188:32380,c8k2=https://192.168.14.189:32380,c8k3=https://192.168.14.190:32380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt
ETCD_CERT_FILE=/etc/etcd/pki/client.crt
ETCD_KEY_FILE=/etc/etcd/pki/client.key
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

加入集群

运行member add命令把c8k3加入到集群里. 如下:

[root@c8k2 etcd]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.190:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member add c8k3 --peer-urls=https://192.168.14.190:32380 
Member ffaacb3e3b861a2d added to cluster d1cee4f300849f84

ETCD_NAME="c8k3"
ETCD_INITIAL_CLUSTER="c8k1=https://192.168.14.188:32380,c8k2=https://192.168.14.189:32380,c8k3=https://192.168.14.190:32380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.14.190:32380"
ETCD_INITIAL_CLUSTER_STATE="existing"

查看当前member. 要注意的是c8k3处在unstarted状态. 这是因为c8k3的etcd服务还没有开启.

[root@c8k2 etcd]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.188:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member list 
30562fa23a71c781, started, c8k1, https://192.168.14.188:32380, https://192.168.14.188:32379, false
c7a9cd6ce683f76a, started, c8k2, https://192.168.14.189:32380, https://192.168.14.189:32379, false
ffaacb3e3b861a2d, unstarted, , https://192.168.14.190:32380, , false

启动服务

在c8k3上，启动etcd服务.

systemctl start etcd

查看member状态.

[root@c8k3 ~]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.188:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member list 
30562fa23a71c781, started, c8k1, https://192.168.14.188:32380, https://192.168.14.188:32379, false
c7a9cd6ce683f76a, started, c8k2, https://192.168.14.189:32380, https://192.168.14.189:32379, false
ffaacb3e3b861a2d, started, c8k3, https://192.168.14.190:32380, https://192.168.14.190:32379, false

集群缩容

直接运行member remove命令就行. 操作如下:

[root@c8k2 ~]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.188:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member list 
30562fa23a71c781, started, c8k1, https://192.168.14.188:32380, https://192.168.14.188:32379, false
c7a9cd6ce683f76a, started, c8k2, https://192.168.14.189:32380, https://192.168.14.189:32379, false
ffaacb3e3b861a2d, started, c8k3, https://192.168.14.190:32380, https://192.168.14.190:32379, false
[root@c8k2 ~]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.190:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member remove ffaacb3e3b861a2d
Member ffaacb3e3b861a2d removed from cluster d1cee4f300849f84
[root@c8k2 ~]# etcdctl --endpoints=https://192.168.14.189:32379,https://192.168.14.188:32379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key member list 
30562fa23a71c781, started, c8k1, https://192.168.14.188:32380, https://192.168.14.188:32379, false
c7a9cd6ce683f76a, started, c8k2, https://192.168.14.189:32380, https://192.168.14.189:32379, false

参考

  • https://www.zhaowenyu.com/etcd-doc/ops/etcd-install-systemd.html
  • https://www.zhaowenyu.com/etcd-doc/ops/etcd-tls-install.html
  • https://www.zhaowenyu.com/etcd-doc/ops/runtime-configuration.html
  • https://cloud.tencent.com/developer/article/1825488

上一篇： Etcd 可视化管理工具

下一篇： 云计算和云原生 - ETCD 数据库详解 - ETCD 的架构

推荐阅读