Kubernetes etcd 技术指南

1. 简介与核心作用etcd 是 K8S 集群的核心数据存储采用 Raft 共识算法保证分布式一致性。存储内容Pod、Service、ConfigMap、Secret 等资源对象元数据API Server 所有状态信息集群自举信息、调度器与 Controller Manager 状态存储架构plaintext┌────────────────────────────────────────────────────────────┐ │ etcd 存储架构 │ ├────────────────────────────────────────────────────────────┤ │ Client → API Server → WAL (预写日志) → Boltdb (KV存储) │ │ ↓ │ │ MVCC 多版本控制 Watch 机制 │ └────────────────────────────────────────────────────────────┘2. 工作原理2.1 Raft 选举与日志复制Leader 选举plaintextFollower ──(选举超时150-300ms)──► Candidate ──(获得多数票)──► Leader ↑ └──(未获多数)──► 重试选举Term 概念每个任期最多一个 Leader节点通过 Term 判断过期信息。日志复制流程plaintextClient → Leader: PUT keyvalue │ ▼ ┌─────────────┐ │ 写入本地 WAL │ └──────┬──────┘ ▼ 广播 AppendEntries RPC │ │ │ │ ▼ ▼ ▼ ▼ Node2 Node3 Node4 Node5 │ │ │ │ └────────┴────────┴────────┘ │ (多数派确认 3/5) ▼ ┌─────────────┐ │ 应用到状态机 │ │ 返回客户端 │ └─────────────┘ 日志条目结构: Index │ Term │ Data │ Committed 1 │ 1 │ kv1 │ ✓ 2 │ 1 │ kv2 │ ✓ 3 │ 2 │ kv3 │ ✓ 4 │ 2 │ kv4 │ -2.2 读写流程写: Leader → WAL → 广播 Followers → 多数派确认 → 状态机读: Leader 直接读本地Follower 可转发 Leader 读Watch: gRPC 流实时推送 key 变化事件2.3 MVCC 多版本控制bashetcdctl get --rev100 /registry/pods/default # 读取历史版本 etcdctl compaction 10000 # 压缩历史3. 集群部署3.1 kubeadm 静态 Pod 部署yaml# /etc/kubernetes/manifests/etcd.yaml apiVersion: v1 kind: Pod metadata: labels: component: etcd tier: control-plane name: etcd namespace: kube-system spec: containers: - command: - etcd - --data-dir/var/lib/etcd - --wal-dir/var/lib/etcd/wal - --namenode1 - --cert-file/etc/kubernetes/pki/etcd/server.crt - --key-file/etc/kubernetes/pki/etcd/server.key - --trusted-ca-file/etc/kubernetes/pki/etcd/ca.crt - --client-cert-authtrue - --peer-cert-file/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file/etc/kubernetes/pki/etcd/peer.key - --peer-trusted-ca-file/etc/kubernetes/pki/etcd/ca.crt - --peer-client-cert-authtrue - --listen-peer-urlshttps://0.0.0.0:2380 - --listen-client-urlshttps://0.0.0.0:2379 - --advertise-client-urlshttps://192.168.1.10:2379 - --initial-clusternode1https://192.168.1.10:2380,node2https://192.168.1.11:2380,node3https://192.168.1.12:2380 - --initial-cluster-statenew - --initial-cluster-tokenetcd-cluster - --quota-backend-bytes8589934592 - --auto-compaction-retention1 - --heartbeat-interval500 - --election-timeout2500 image: registry.k8s.io/etcd:3.5.9 livenessProbe: httpGet: host: 192.168.1.10 path: /health port: 2379 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 volumeMounts: - mountPath: /var/lib/etcd name: etcd-data - mountPath: /etc/kubernetes/pki/etcd name: etcd-certs volumes: - hostPath: path: /var/lib/etcd type: DirectoryOrCreate name: etcd-data - hostPath: path: /etc/kubernetes/pki/etcd type: DirectoryOrCreate name: etcd-certs3.2 关键参数表格参数说明推荐值--quota-backend-bytes数据库配额8GB--heartbeat-interval心跳间隔(ms)500--election-timeout选举超时(ms)5000--auto-compaction-retention自动压缩(小时)1--snapshot-count快照触发事务数50004. 常用操作命令4.1 健康检查bashexport ETCDCTL_API3 export ETCDCTL_CACERT/etc/kubernetes/pki/etcd/ca.crt export ETCDCTL_CERT/etc/kubernetes/pki/etcd/server.crt export ETCDCTL_KEY/etc/kubernetes/pki/etcd/server.key etcdctl member list -w table # 成员列表 etcdctl endpoint health -w table # 健康状态 etcdctl endpoint status -w table # 状态详情4.2 数据操作bashetcdctl put /registry/pods/default/nginx {apiVersion:v1} # 写入 etcdctl get /registry/pods/default/nginx # 读取 etcdctl get --prefix /registry/pods/ # 前缀查询 etcdctl del /registry/pods/default/nginx # 删除 etcdctl get --rev100 /registry/pods/default # 历史版本4.3 备份与恢复bash# 快照备份 etcdctl snapshot save /backup/etcd-snapshot-$(date %Y%m%d).db # 检查快照 etcdctl snapshot status /backup/etcd-snapshot.db -w table # 恢复快照 etcdctl snapshot restore /backup/etcd-snapshot.db \ --namenode1 \ --initial-clusternode1https://192.168.1.10:2380,node2https://192.168.1.11:2380 \ --initial-cluster-tokenetcd-cluster \ --initial-advertise-peer-urlshttps://192.168.1.10:2380 \ --data-dir/var/lib/etcd4.4 成员管理bashetcdctl member add node4 --peer-urlshttps://192.168.1.13:2380 # 添加 etcdctl member remove member_id # 移除 etcdctl member update member_id --peer-urlshttps://... # 更新4.5 维护操作bashetcdctl defrag --endpoints$ENDPOINTS # 碎片整理(必须定期) etcdctl compaction revision # 压缩历史版本 etcdctl alarm disarm # 取消空间告警5. 常见问题与排查5.1 集群脑裂 / 成员故障排查步骤bash# 1. 检查日志 kubectl logs -n kube-system etcd-node1 --tail100 # 2. 成员状态 etcdctl member list # 3. 网络连通性 nc -zv peer_ip 2380 # 4. 检查选举超时一致性 grep -E heartbeat|election /etc/kubernetes/manifests/etcd.yaml解决方案bash# 重启故障节点 sudo systemctl restart etcd # 移除不可用节点 etcdctl member remove 故障节点ID # 完全不可用时从快照恢复5.2 数据库空间不足错误:etcdserver: mvcc: database space exceeded排查bashetcdctl endpoint status -w table # 查看配额使用 du -sh /var/lib/etcd/ # 查看实际大小解决bash# 1. 取消告警 etcdctl alarm disarm # 2. 压缩历史版本 REVISION$(etcdctl endpoint status --write-outjson | jq -r .[0].Status.header.revision) etcdctl compaction $((REVISION - 1000)) # 3. 碎片整理 etcdctl defrag --endpoints$ENDPOINTS # 4. 验证 etcdctl endpoint status -w table预防: 配额设 8GB auto-compaction 定期备份5.3 API Server 连接超时排查bash# 1. etcd 服务状态 kubectl get pods -n kube-system -l componentetcd # 2. 健康状态 etcdctl endpoint health # 3. API Server 日志 kubectl logs -n kube-system kube-apiserver-node --tail50 | grep -i etcd # 4. 证书检查 openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates常见原因: 证书过期、CN 不匹配、网络不通、防火墙阻断6. 最佳实践6.1 集群规模建议表格规模节点CPU内存磁盘100节点1-32C4G50GB SSD100-50034C8G100GB SSD5003-58C16G200GB SSD6.2 硬件配置磁盘: NVMe SSDWAL 独立分区网络: 10Gbps建议独立网络隔离内存: 8GB 以上内存直接影响性能6.3 备份策略bash#!/bin/bash # /usr/local/bin/etcd-backup.sh BACKUP_DIR/backup/etcd DATE$(date %Y%m%d%H%M%S) export ETCDCTL_API3 export ETCDCTL_CACERT/etc/kubernetes/pki/etcd/ca.crt export ETCDCTL_CERT/etc/kubernetes/pki/etcd/server.crt export ETCDCTL_KEY/etc/kubernetes/pki/etcd/server.key etcdctl snapshot save ${BACKUP_DIR}/etcd-snapshot-${DATE}.db find ${BACKUP_DIR} -name etcd-snapshot-*.db -mtime 7 -delete6.4 灾难恢复流程bash# 1. 停止控制平面 sudo systemctl stop kube-apiserver kube-controller-manager kube-scheduler # 2. 停止 etcd sudo systemctl stop etcd # 3. 清理数据 sudo mv /var/lib/etcd /var/lib/etcd.bak # 4. 恢复快照 etcdctl snapshot restore /backup/etcd-snapshot.db \ --namenode1 \ --initial-clusternode1https://192.168.1.10:2380,node2https://192.168.1.11:2380 \ --initial-cluster-tokenetcd-cluster \ --initial-advertise-peer-urlshttps://192.168.1.10:2380 \ --data-dir/var/lib/etcd # 5. 启动并验证 sudo systemctl start etcd etcdctl endpoint health kubectl get nodes6.5 监控指标关键 Prometheus 指标:etcd_server_leader_changes_seen_total- Leader 变更次数etcd_mvcc_db_total_size_in_bytes- 数据库大小etcd_server_quota_backend_bytes- 配额使用率etcd_network_peer_round_trip_time_seconds- 节点延迟参考: K8S etcd 文档 | etcd 官方