5 台服务器（一）：系统初始化与证书底座

5 台服务器二进制 K8s 系列第一篇：角色与网络规划、系统初始化、containerd、证书体系，把控制面要用的底座一次配好。

技术文章2026 年 9 月 9 日约 12 分钟1 次阅读

系列导航：系列总览 · 下一篇：5 台服务器（二）：etcd 与控制面高可用

项目	说明
系列	5 台服务器二进制 K8s（第 1/5 篇）
本篇	角色与网段规划、系统初始化、containerd、证书体系
前置	无；建议先读系列总览里的原理层
版本	Kubernetes v1.36.4 / etcd v3.6.13 / containerd v2.3.3 LTS / Calico v3.32.2

#一、前置条件：发行版、版本与角色划分

以下操作以 Rocky Linux 9.x（RHEL 系通吃，内核 ≥ 5.10，配套 K8s v1.36.4、etcd v3.6.13、containerd v2.3.3 LTS、Calico v3.32.2）为例，版本是 2026 年 9 月的取值，动手前建议去各组件 release 页面核对补丁号。5 台机器，3 台控制平面 + 2 台工作节点，etcd 与控制平面堆叠部署（stacked），不单独拆 etcd 节点：

主机名	IP	角色	建议配置
k8s-m1	10.0.0.11	control-plane + etcd + NFS	8C16G，200G SSD
k8s-m2	10.0.0.12	control-plane + etcd	8C16G，200G SSD
k8s-m3	10.0.0.13	control-plane + etcd	8C16G，200G SSD
k8s-n1	10.0.0.14	worker	16C64G，1T NVMe
k8s-n2	10.0.0.15	worker	16C64G，1T NVMe
k8s-vip	10.0.0.10	apiserver VIP（keepalived）	—

网络规划：Pod CIDR 用 172.16.0.0/16，Calico 按节点切 /24（每节点最多 254 个 Pod）；Service CIDR 用 10.96.0.0/12，CoreDNS 固定 10.96.0.10。这两个网段不能和宿主机网段 10.0.0.0/24 重叠；NodePort 范围保持默认 30000-32767。

几个取舍必须说明：不拆独立 etcd 节点，因为 5 台拆出 3 台纯 etcd 就只剩 2 台跑业务，堆叠部署下 apiserver 走 127.0.0.1 延迟最低，风险是 master 整机宕机时同时丢一个 etcd 成员，3 节点还扛得住；工作节点给 64G，是因为 GitLab 级别的仓库先不算（Gitea 轻很多），Harbor、Jenkins、Nexus、监控、日志、Nacos、MySQL、Redis 全都要落在 n1/n2 上，32G 会很紧张；只有 2 个 worker 也要 3 个 master，因为 etcd 需要奇数成员做 quorum。内网域名统一走 *.apps.lxhhub.top，在内网 DNS 上 A 记录指向 n1/n2；所有命令里的 10.0.0.x、apps.lxhhub.top 按你自己的环境替换。

#二、第 0 步：5 台机器的系统初始化

这段在 m1、m2、m3、n1、n2 上全部执行一遍，参数只有一个主机名：

cat > /root/init-node.sh <<'SCRIPT'
#!/bin/bash
set -e
HOST=$1
[ -z "$HOST" ] && { echo "用法: $0 <主机名>"; exit 1; }

hostnamectl set-hostname ${HOST}

cat >> /etc/hosts <<'EOF'
10.0.0.10  k8s-vip
10.0.0.11  k8s-m1
10.0.0.12  k8s-m2
10.0.0.13  k8s-m3
10.0.0.14  k8s-n1
10.0.0.15  k8s-n2
EOF

# 防火墙：有严格的合规要求就别关，改成放行 6443/2379/2380/10250/179/8472 等端口
systemctl disable --now firewalld || true

setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

swapoff -a
sed -i '/ swap /s/^/#/' /etc/fstab

dnf install -y ipvsadm ipset conntrack-tools socat ebtables chrony \
               curl wget tar jq bash-completion nfs-utils

systemctl enable --now chronyd

cat > /etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF
modprobe overlay
modprobe br_netfilter
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh

cat > /etc/sysctl.d/99-k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
net.ipv4.ip_local_port_range        = 1024 65535
net.netfilter.nf_conntrack_max      = 1000000
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.core.somaxconn                  = 32768
net.ipv4.neigh.default.gc_thresh1   = 8192
net.ipv4.neigh.default.gc_thresh2   = 32768
net.ipv4.neigh.default.gc_thresh3   = 65536
fs.file-max                         = 1000000
fs.inotify.max_user_instances       = 8192
fs.inotify.max_user_watches         = 524288
vm.swappiness                       = 0
vm.max_map_count                    = 262144
EOF
sysctl --system

cat > /etc/security/limits.d/99-k8s.conf <<'EOF'
*       soft    nofile    1048576
*       hard    nofile    1048576
*       soft    nproc     655360
*       hard    nproc     655360
*       soft    memlock   unlimited
*       hard    memlock   unlimited
EOF

mkdir -p /etc/kubernetes/pki /var/lib/kubelet /var/log/kubernetes
echo "init done: ${HOST}"
SCRIPT

chmod +x /root/init-node.sh

按主机分别执行并重启：

# 各自执行
/root/init-node.sh k8s-m1   # 10.0.0.11
/root/init-node.sh k8s-m2   # 10.0.0.12
/root/init-node.sh k8s-m3   # 10.0.0.13
/root/init-node.sh k8s-n1   # 10.0.0.14
/root/init-node.sh k8s-n2   # 10.0.0.15

reboot

重启后验证三件事：getenforce 输出 Permissive 或 Disabled；free -h | grep Swap 为 0；chronyc tracking 显示时间同步正常；五台 hostname 各不相同。时间同步必须单独强调：证书校验强依赖时间，5 台机器差几分钟，etcd 的 peer 通信和 kubelet 的 bootstrap 会以各种姿势失败，而且报错信息里一个字都不会提"时间"。

getenforce
free -h | grep Swap
chronyc tracking | grep Leap
hostname

#三、容器运行时：containerd

containerd 2.x 的配置文件 schema 是 version = 3，从 1.x 升级上来的老配置文件直接套会报 unsupported config version，别偷懒复用。关键点有两个：sandbox（pause）镜像在 2.x 里通过 pinned_images 指定，老文章写的 sandbox_image 字段已经挪位置；cgroup 驱动必须和 kubelet 一致用 systemd。

cat > /root/install-containerd.sh <<'SCRIPT'
#!/bin/bash
set -e
CD_VER=2.3.3
RUNC_VER=1.3.0      # 换成官方 release 页最新的 1.3.x
CNI_VER=1.7.0       # 换成官方 release 页最新的 1.7.x
PAUSE_VER=3.10

cd /usr/local/src

wget -q https://github.com/containerd/containerd/releases/download/v${CD_VER}/containerd-${CD_VER}-linux-amd64.tar.gz
tar -C /usr/local -xzf containerd-${CD_VER}-linux-amd64.tar.gz

wget -q -O /usr/local/sbin/runc https://github.com/opencontainers/runc/releases/download/v${RUNC_VER}/runc.amd64
chmod +x /usr/local/sbin/runc

mkdir -p /opt/cni/bin /etc/cni/net.d
wget -q https://github.com/containernetworking/plugins/releases/download/v${CNI_VER}/cni-plugins-linux-amd64-v${CNI_VER}.tgz
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v${CNI_VER}.tgz

cat > /usr/lib/systemd/system/containerd.service <<'EOF'
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

mkdir -p /etc/containerd /etc/containerd/certs.d

cat > /etc/containerd/config.toml <<EOF
version = 3
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  level = "warn"

[plugins."io.containerd.cri.v1.images"]
  snapshotter = "overlayfs"
  [plugins."io.containerd.cri.v1.images".pinned_images]
    sandbox = "registry.k8s.io/pause:${PAUSE_VER}"

[plugins."io.containerd.cri.v1.runtime"]
  [plugins."io.containerd.cri.v1.runtime".containerd]
    default_runtime_name = "runc"
    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
        SystemdCgroup = true

[plugins."io.containerd.cri.v1.runtime".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"
EOF

systemctl daemon-reload
systemctl enable --now containerd
systemctl is-active containerd
SCRIPT

chmod +x /root/install-containerd.sh && /root/install-containerd.sh

pinned_images.sandbox 写错的话，Pod 沙箱会卡在 ImagePullBackOff 且报的是 pause 镜像拉不下来。镜像信任和加速：公共镜像按你的网络环境配一个或几个加速源；内网 Harbor 走 HTTP，就必须在每个节点告诉 containerd 这个仓库可信——这个配置对 kubelet 拉镜像同样生效，漏配一台，调度到那台的 Pod 就起不来：

mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'EOF'
server = "https://registry-1.docker.io"
[host."https://docker.m.daocloud.io"]
  capabilities = ["pull", "resolve"]
[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
EOF

mkdir -p /etc/containerd/certs.d/harbor.apps.lxhhub.top
cat > /etc/containerd/certs.d/harbor.apps.lxhhub.top/hosts.toml <<'EOF'
server = "http://harbor.apps.lxhhub.top"
[host."http://harbor.apps.lxhhub.top"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF

systemctl restart containerd

装 crictl，后面排障全靠它：

CRICTL_VER=v1.32.0
wget -q https://github.com/kubernetes-sigs/crictl/releases/download/${CRICTL_VER}/crictl-${CRICTL_VER}-linux-amd64.tar.gz
tar -C /usr/local/bin -xzf crictl-${CRICTL_VER}-linux-amd64.tar.gz

cat > /etc/crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

crictl info | head -20
crictl pull registry.k8s.io/pause:3.10

crictl pull 成功说明运行时链路通了；失败先查 journalctl -u containerd -f，九成是镜像源或者证书的问题。

#四、证书体系：在 m1 上一次签完

证书的组织方式是两套独立 CA：etcd 一套、Kubernetes 一套。一套 CA 打天下也能跑，但 etcd 的证书和 K8s 的证书混在一起，将来轮换 etcd CA 时会连带影响 apiserver，拆开省事。全部签名只在 m1 上做一次，然后 scp 到各节点。先装 cfssl 并生成签名配置：

# m1 上执行
wget -q -O /usr/local/bin/cfssl     https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64
wget -q -O /usr/local/bin/cfssljson https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

mkdir -p /root/ssl && cd /root/ssl

cat > ca-config.json <<'EOF'
{
  "signing": {
    "default": { "expiry": "87600h" },
    "profiles": {
      "kubernetes": {
        "expiry": "87600h",
        "usages": ["signing", "key encipherment", "server auth", "client auth"]
      },
      "etcd": {
        "expiry": "87600h",
        "usages": ["signing", "key encipherment", "server auth", "client auth"]
      }
    }
  }
}
EOF

cat > etcd-ca-csr.json <<'EOF'
{
  "CN": "etcd-ca",
  "key": { "algo": "rsa", "size": 4096 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Ops" }]
}
EOF

cat > k8s-ca-csr.json <<'EOF'
{
  "CN": "kubernetes-ca",
  "key": { "algo": "rsa", "size": 4096 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Ops" }]
}
EOF

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
cfssl gencert -initca k8s-ca-csr.json  | cfssljson -bare k8s-ca

etcd 的 server 证书要同时用于 peer（2380）和 client（2379）通信，SANs 必须把三个节点 IP、127.0.0.1、主机名和对应域名全写进去：

cat > etcd-server-csr.json <<'EOF'
{
  "CN": "etcd-server",
  "hosts": [
    "127.0.0.1",
    "10.0.0.11", "10.0.0.12", "10.0.0.13",
    "k8s-m1", "k8s-m2", "k8s-m3",
    "k8s-m1.apps.lxhhub.top", "k8s-m2.apps.lxhhub.top", "k8s-m3.apps.lxhhub.top"
  ],
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Ops" }]
}
EOF

cat > etcd-client-csr.json <<'EOF'
{
  "CN": "etcd-client",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Ops" }]
}
EOF

cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=etcd \
  etcd-server-csr.json | cfssljson -bare etcd-server
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=etcd \
  etcd-client-csr.json | cfssljson -bare etcd-client

apiserver 的证书是整篇重点：SANs 少写一个，kubectl 用 VIP 访问就会报 x509: certificate is valid for ..., not k8s-vip。除了三台 master 的 IP 和 VIP，还要把 Service 网段里的 10.96.0.1 与集群内部域名都写进去，否则集群内访问 apiserver 同样校验失败：

cat > apiserver-csr.json <<'EOF'
{
  "CN": "kube-apiserver",
  "hosts": [
    "127.0.0.1",
    "10.0.0.10", "10.0.0.11", "10.0.0.12", "10.0.0.13",
    "10.96.0.1",
    "k8s-vip", "k8s-m1", "k8s-m2", "k8s-m3",
    "kubernetes", "kubernetes.default", "kubernetes.default.svc",
    "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local"
  ],
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Ops" }]
}
EOF

cfssl gencert -ca=k8s-ca.pem -ca-key=k8s-ca-key.pem -config=ca-config.json -profile=kubernetes \
  apiserver-csr.json | cfssljson -bare apiserver

其余客户端证书用一个函数批量生成。注意 O 字段决定 RBAC group：system:kube-controller-manager、system:masters 等都是 apiserver 内置绑定的组名：

cat > apiserver-kubelet-client-csr.json <<'EOF'
{
  "CN": "apiserver-kubelet-client",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "system:masters", "OU": "Ops" }]
}
EOF
cfssl gencert -ca=k8s-ca.pem -ca-key=k8s-ca-key.pem -config=ca-config.json -profile=kubernetes \
  apiserver-kubelet-client-csr.json | cfssljson -bare apiserver-kubelet-client

gen_client () {
  cat > ${1}-csr.json <<EOF
{
  "CN": "${2}",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "${3}", "OU": "Ops" }]
}
EOF
  cfssl gencert -ca=k8s-ca.pem -ca-key=k8s-ca-key.pem -config=ca-config.json -profile=kubernetes \
    ${1}-csr.json | cfssljson -bare ${1}
}

gen_client kube-controller-manager  system:kube-controller-manager  system:kube-controller-manager
gen_client kube-scheduler           system:kube-scheduler           system:kube-scheduler
gen_client admin                    admin                           system:masters
gen_client kube-proxy               system:kube-proxy               system:node-proxier

front-proxy CA 给聚合层用。metrics-server 以及任何以 APIService 方式聚合进来的组件都靠它，不配的话 kubectl top 会报 unauthorized。ServiceAccount 的密钥对不是 x509，直接用 openssl 生成：

cat > front-proxy-ca-csr.json <<'EOF'
{
  "CN": "front-proxy-ca",
  "key": { "algo": "rsa", "size": 4096 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Ops" }]
}
EOF
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca

cat > front-proxy-client-csr.json <<'EOF'
{
  "CN": "front-proxy-client",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "Kubernetes", "OU": "Ops" }]
}
EOF
cfssl gencert -ca=front-proxy-ca.pem -ca-key=front-proxy-ca-key.pem -config=ca-config.json \
  -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare front-proxy-client

openssl genrsa -out sa.key 2048
openssl rsa -in sa.key -pubout -out sa.pub

分发规则：三个 master 拿 etcd 证书 + k8s 证书 + sa 密钥；两个 worker 只需要 CA 公钥。k8s-ca-key.pem 只放 master，worker 上不需要——私钥泄漏等于集群门户大开，能不放就别放：

cd /root/ssl

for h in k8s-m1 k8s-m2 k8s-m3; do
  ssh $h "mkdir -p /etc/etcd/ssl /etc/kubernetes/pki"
  scp etcd-ca.pem etcd-server.pem etcd-server-key.pem etcd-client.pem etcd-client-key.pem \
      $h:/etc/etcd/ssl/
  scp k8s-ca.pem k8s-ca-key.pem apiserver.pem apiserver-key.pem \
      apiserver-kubelet-client.pem apiserver-kubelet-client-key.pem \
      front-proxy-ca.pem front-proxy-client.pem front-proxy-client-key.pem \
      sa.key sa.pub $h:/etc/kubernetes/pki/
done

for h in k8s-n1 k8s-n2; do
  ssh $h "mkdir -p /etc/kubernetes/pki"
  scp k8s-ca.pem $h:/etc/kubernetes/pki/
done

下一篇：5 台服务器（二）：etcd 与控制面高可用 —— 三成员 etcd、控制平面三件套与 apiserver VIP。

#Kubernetes #二进制部署 #容器 #Linux

留言讨论

0 条

还没有留言，来说点什么吧。

#一、前置条件：发行版、版本与角色划分

#二、第 0 步：5 台机器的系统初始化

#三、容器运行时：containerd

#四、证书体系：在 m1 上一次签完

相关阅读

留言讨论