启用 istio sidecar 后，我们在 Kubernetes 中部署 rook-ceph 时遇到问题。问题是 OSD 没有出现，因为 crashcollector 没有正确初始化。如下图卡住了。
rook-ceph csi-cephfsplugin-7jcr9 3/3 Running 0 63m
rook-ceph csi-cephfsplugin-c4dnd 3/3 Running 0 63m
rook-ceph csi-cephfsplugin-provisioner-8658f67749-6gzkk 7/7 Running 2 63m
rook-ceph csi-cephfsplugin-provisioner-8658f67749-bgdpx 7/7 Running 1 63m
rook-ceph csi-cephfsplugin-zj9xm 3/3 Running 0 63m
rook-ceph csi-rbdplugin-58xf4 3/3 Running 0 63m
rook-ceph csi-rbdplugin-87rjn 3/3 Running 0 63m
rook-ceph csi-rbdplugin-provisioner-94f699d86-rh2r6 7/7 Running 1 63m
rook-ceph csi-rbdplugin-provisioner-94f699d86-xkv6h 7/7 Running 1 63m
rook-ceph csi-rbdplugin-tvjvz 3/3 Running 0 63m
rook-ceph rook-ceph-crashcollector-node1-f7f6c6f8d-lfs6d 0/2 Init:0/3 0 63m
rook-ceph rook-ceph-crashcollector-node2-998bb8769-pspnn 0/2 Init:0/3 0 51m
rook-ceph rook-ceph-crashcollector-node3-6c48c99c8-7bbl6 0/2 Init:0/3 0 40m
rook-ceph rook-ceph-mon-a-7966994c76-z9phm 2/2 Running 0 51m
rook-ceph rook-ceph-mon-b-8cbf8579f-g6nd9 2/2 Running 0 51m
rook-ceph rook-ceph-mon-c-d65968cc4-wcpmr 2/2 Running 0 40m
rook-ceph rook-ceph-operator-5c47844cf-z9jcb 2/2 Running 1 67m
当我们在这个 pod 上执行 kubectl describe 时，我们会遇到以下问题：
Warning FailedMount 59m kubelet, node1 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[rook-config-override rook-ceph-log rook-ceph-crash-collector-keyring istio-data istio-podinfo istiod-ca-cert istio-envoy rook-ceph-crash default-token-htvcq]: timed out waiting for the condition
还注意到没有创建秘密“rook-ceph-crash-collector-keyring”。
经过大量调试后，注意到“mon” pod 无法通过服务端点访问。但所有其他通信，如 Kubernetes API、其他命名空间中的其他服务等都运行良好。
当我们执行到“mon”pod 并执行 curl 时，如果我们使用它连接的主机名。
sh-4.4# curl -f rook-ceph-mon-b-8cbf8579f-g6nd9:6789
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
但使用服务名称不起作用
sh-4.4# curl -f rook-ceph-mon-a:6789
curl: (56) Recv failure: Connection reset by peer
在 rook-ceph-operator 日志中也注意到，有一些潜在的线索表明没有让 mons 进入 quorum。
2021-02-13 06:11:23.532494 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update
2021-02-13 06:11:23.532658 I | op-mon: waiting for mon quorum with [a c b]
2021-02-13 06:11:24.123965 I | op-mon: mons running: [a c b]
2021-02-13 06:11:44.354283 I | op-mon: mons running: [a c b]
2021-02-13 06:12:04.553052 I | op-mon: mons running: [a c b]
2021-02-13 06:12:24.760423 I | op-mon: mons running: [a c b]
2021-02-13 06:12:44.953344 I | op-mon: mons running: [a c b]
2021-02-13 06:13:05.153151 I | op-mon: mons running: [a c b]
2021-02-13 06:13:25.354678 I | op-mon: mons running: [a c b]
2021-02-13 06:13:45.551489 I | op-mon: mons running: [a c b]
2021-02-13 06:14:05.910343 I | op-mon: mons running: [a c b]
2021-02-13 06:14:26.188100 I | op-mon: mons running: [a c b]
2021-02-13 06:14:46.377549 I | op-mon: mons running: [a c b]
2021-02-13 06:15:06.563272 I | op-mon: mons running: [a c b]
2021-02-13 06:15:27.119178 I | op-mon: mons running: [a c b]
2021-02-13 06:15:47.372562 I | op-mon: mons running: [a c b]
2021-02-13 06:16:07.565653 I | op-mon: mons running: [a c b]
2021-02-13 06:16:27.751456 I | op-mon: mons running: [a c b]
2021-02-13 06:16:47.952091 I | op-mon: mons running: [a c b]
2021-02-13 06:17:08.168884 I | op-mon: mons running: [a c b]
2021-02-13 06:17:28.358448 I | op-mon: mons running: [a c b]
2021-02-13 06:17:48.559239 I | op-mon: mons running: [a c b]
2021-02-13 06:18:08.767715 I | op-mon: mons running: [a c b]
2021-02-13 06:18:28.987579 I | op-mon: mons running: [a c b]
2021-02-13 06:18:49.242784 I | op-mon: mons running: [a c b]
2021-02-13 06:19:09.456809 I | op-mon: mons running: [a c b]
2021-02-13 06:19:29.671632 I | op-mon: mons running: [a c b]
2021-02-13 06:19:49.871453 I | op-mon: mons running: [a c b]
2021-02-13 06:20:10.062897 I | op-mon: mons running: [a c b]
2021-02-13 06:20:30.258163 I | op-mon: mons running: [a c b]
2021-02-13 06:20:50.452097 I | op-mon: mons running: [a c b]
2021-02-13 06:21:10.655282 I | op-mon: mons running: [a c b]
2021-02-13 06:21:25.854570 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to start mon pods: failed to check mon quorum a: failed to wait for mon quorum: exceeded max retry count waiting for monitors to reach quorum
看起来 mons 不再可以通过服务端点访问，这使得整个初始化过程卡住了。
以下是在 rook-ceph 命名空间下运行的服务。
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics ClusterIP 10.233.30.235 <none> 8080/TCP,8081/TCP 83m
csi-rbdplugin-metrics ClusterIP 10.233.61.8 <none> 8080/TCP,8081/TCP 83m
rook-ceph-mon-a ClusterIP 10.233.2.224 <none> 6789/TCP,3300/TCP 83m
rook-ceph-mon-b ClusterIP 10.233.39.129 <none> 6789/TCP,3300/TCP 72m
rook-ceph-mon-c ClusterIP 10.233.51.59 <none> 6789/TCP,3300/TCP 61m
其他注意事项： Wee 使用所有最新版本的 istio、rook-ceph 等。集群是使用 Kubespray 创建的，在 Ubuntu bionic 上运行，具有 3 个节点。使用印花布。
如果您需要更多详细信息，请告诉我们。提前致谢。