.1. ETCD介绍

  ETCD是一个K-V分布式数据存储库，为分布式系统提供关键数据的存储服务。主要特点

  1. 简单：设计简单、部署简单，API简单
  2. 安全：可部署为自认证系统
  3. 快速：写入性能测试达到了10k/s
  4. 可靠：基于Raft算法

  Raft算法的介绍，可以参考：The Raft Consensus Algorithm。Raft的提出，主要是为了解决Paxos算法的如下短板

  1. 晦涩难懂
  2. 落地困难

因此，Raft设计之初就是为了易与理解而设计。在上述参考文献中，详细介绍了Raft算的机制/原理等内容。

.2. 部署ETCD集群

  ETCD集群的部署，也较为简单。不安全的集群部署，这里不再讨论。这里的部署，主要是指自认证的ETCD集群。

  ETCD集群部署配置，可以参考：ETCD集群部署、etcd多台部署:启用https以及ca自签名。参考上述两篇博文的配置、操作步骤，但配置过程中略微存在差异。

.2.1. 编译cfssl组件

  CFSSL主要是为了实现自认证配置，生成CA证书、公钥、私钥等内容。
最新版本cfssl已经不再提供通过官方网站下载，可以直接通过Github下载编译或直接安装

### 直接下载某个版本
wget https://github.com/cloudflare/cfssl/archive/1.3.4.tar.gz

### 下载Master分支，切换为某个tag的代码
1. Download：https://github.com/cloudflare/cfssl.git
2. 安装依赖组件：
    go get -u github.com/cloudflare/cfssl/cli \
              github.com/go-sql-driver/mysql \
              github.com/lib/pq github.com/mattn/go-sqlite3 \
              github.com/cloudflare/go-metrics
3. 编译安装：make ## 会在bin目录下面创建常用命令
### 切换分支命令：参考
git checkout tags/v1.0 -b v1.0-branch

.2.2. 创建CA证书

  创建CA证书，主要涉及到两个操作步骤。通过CFSSL来创建证书，一般只需要编辑JSON配置文件即可。

.2.2.1. 创建CA证书

### 创建CA，配置文件如下
#### ca-config.json
{
    "signing": {
        "default": {
            "expiry": "86400h"
        },
        "profiles": {
            "server": {
                "expiry": "86400h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "86400h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "86400h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

#### 创建CA相关文件
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

.2.2.2. 创建业务Key

  创建业务需求的公钥\私钥，需要明确业务是那种情况：server(peer)、client，这里配置了两种不同的适应场景。另外一点，需要定义一下业务的配置JSON文件。

### 配置业务JSON文件：etcd-csr.json
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "host1_ip",
        "host2_ip",
        "host3_ip"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shanghai",
            "L": "Shanghai",
            "O": "test.com",
            "OU": "System"
        }
    ]
}

### 创建业务场景的公钥/私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd-csr.json | cfssljson -bare etcd

  至此，CA配置以及业务需要的配置的自认证配置已获取到。

.2.3. 集群配置部署

  这里配置为service的形式，并使用ansible部署集群：主要涉及到配置文件(etcd.conf)、服务配置(etcd.service)，以及ansible部署脚本。

.2.3.1. etcd.service.j2

### 配置路径：/usr/lib/systemd/system/etcd.service
{%- set hosts = []  -%}
{%- for host in groups['ETCD'] | sort -%}
 {{ hosts.append( hostvars[host]['etcd_name'] + '=https://' + host + ':' + cluster_port|string ) }}
{%- endfor -%}
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
  --name=${ETCD_NAME} \
  --cert-file=/etc/etcd/ssl/cfssl/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/cfssl/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/cfssl/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/cfssl/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/cfssl/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/cfssl/ssl/ca.pem \
  --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
  --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster={{ hosts | join(',') }}
  --initial-cluster-state=new \
  --data-dir=${ETCD_DATA_DIR}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

.2.3.2. etcd.conf.j2

#### 配置文件路径：/etc/etcd/etcd.conf
[member]
ETCD_NAME={{ etcd_name }}
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://{{ inventory_hostname }}:{{ cluster_port }}"
ETCD_LISTEN_CLIENT_URLS="https://{{ inventory_hostname }}:{{ client_port }},https://127.0.0.1:{{ client_port }}"

[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://{{ inventory_hostname }}:{{ cluster_port }}"
ETCD_INITIAL_CLUSTER_TOKEN="{{ cluster_token }}"
ETCD_ADVERTISE_CLIENT_URLS="https://{{ inventory_hostname }}:{{ client_port }}"

编译好的脚本，可以在这里下载：cfssl.tar.gz

.2.3.3. 分发部署Playbook

---
- hosts: ETCD
  tasks:
  - name: install etcd
    yum:
        name: etcd
        state: installed

  - name: mkdir target dir
    shell: mkdir -p /etc/etcd/ssl

  - name: copy ssl files
    copy:
        src: cfssl.tar.gz
        dest: /tmp/cfssl.tar.gz

  - name: untar ssl files
    shell: tar xvf /tmp/cfssl.tar.gz -C /etc/etcd/ssl

  - name: set privileges for ca files
    shell: chmod -R 755 /etc/etcd/ssl

  - name: copy template configure - conf
    template:
        src: etcd.service.j2
        dest: /usr/lib/systemd/system/etcd.service

  - name: copy template configure - service
    template:
        src: etcd.conf.j2
        dest: /etc/etcd/etcd.conf

  - name: systemctl reload before start service
    shell: systemctl daemon-reload

  - name: start etcd service
    service:
        name: etcd
        state: restarted

  - name: etcdctl with ssl
    template:
        src: etcdctls.j2
        dest: /usr/bin/etcdctls
        mode: "0755"

.3. 集群状态检查及基本使用

### 健康状态检查: etcdctl 命令行访问
etcdctl   \
    --ca-file=/etc/etcd/ssl/cfssl/ssl/ca.pem   \
    --cert-file=/etc/etcd/ssl/cfssl/ssl/etcd.pem   \
    --key-file=/etc/etcd/ssl/cfssl/ssl/etcd-key.pem   \
    --endpoint https://127.0.0.1:2379   \
    cluster-health

#### 输出
member c3ca8ce4516450a is healthy: got healthy result from https://host1:2379
member 22c833e1835a442d is healthy: got healthy result from https://host2:2379
member 3254807a1a7e65ac is healthy: got healthy result from https://host3:2379
cluster is healthy

### 查看集群成员信息: rest api 访问
curl -k --cert /etc/etcd/ssl/cfssl/ssl/etcd.pem  --key /etc/etcd/ssl/cfssl/ssl/etcd-key.pem  https://host1:2379/v2/members

#### 输出
{
    "members": [
        {
            "id": "c3ca8ce4516450a",
            "name": "infra01",
            "peerURLs": [
                "https://host1:2380"
            ],
            "clientURLs": [
                "https://host1:2379"
            ]
        },
        {
            "id": "22c833e1835a442d",
            "name": "infra03",
            "peerURLs": [
                "https://host2:2380"
            ],
            "clientURLs": [
                "https://host2:2379"
            ]
        },
        {
            "id": "3254807a1a7e65ac",
            "name": "infra02",
            "peerURLs": [
                "https://host3:2380"
            ],
            "clientURLs": [
                "https://host3:2379"
            ]
        }
    ]
}

如果新搭建的集群可以正常返回，那么基本可以表明搭建了一个健康、可用的ETCD集群。

大数据 云原生
调研 ETCD

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！

工具知识总结 上一篇
Python2日期时间操作 下一篇