我是 Ceph 技术的新手，所以我可能不知道明显的东西。我开始使用 cephadm 部署 ceph 集群，我做到了。在我的第一次尝试中，我给了每个节点 3 GB RAM（一段时间后我发现它需要更多）。当节点的 ram 和交换之一填满 100% 时，我的集群挂起。现在我给每个节点 8GB 内存和 10GB SSD 交换，它是固定的：
Node01：4x CPU，8GB RAM，60GB SSD
Node02：4x CPU，6GB RAM，60GB SSD
Node04: 4x CPU , 8GB RAM , 60GB HDD
我通过创建 CephFS 开始使用它（它创建了 2 个池，一个用于数据，一个用于元数据（3x 副本规则））。我使用 ceph-common 在 Ubuntu 20.04 上安装了这个 FS：
>> vim /etc/fstab
...
ceph-n01,ceph-n02:/ /ceph ceph _netdev,name=amin,secretfile=/home/amin/.ceph 0 0
它工作正常。我通过运行渲染地图并将切片保存在文件系统（我的 CephFS 池）中的服务来使用这个 fs。它工作大约 1 天半并生成 ~56.65GB 文件）。第二天，我看到了 1 个 OSD（带有 HDD 的 OSD）并且只有两个 OSD 在运行。我检查了 3 个节点的 RAM 和 CPU 状态。在 2 个节点中使用了 50% 的 RAM，在一个节点（节点 01）中使用了 85% 的 RAM，交换空间约为 4GB。我试图通过重新启动 OSD 来解决这个问题。当我重新启动它们时，关闭的 OSD 一直在崩溃。（之前运行的 OSD，重启后成功启动。我查看了 OSD 日志：
debug -11> 2022-01-06T12:43:01.620+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981624364, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 3219, "file_size": 6014, "table_properties": {"data_size": 4653, "index_size": 64, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 2663, "raw_average_key_size": 17, "raw_value_size": 2408, "raw_average_value_size": 16, "num_data_blocks": 2, "num_entries": 148, "num_deletions": 0, "num_merge_operands": 147, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": ".T:int64_array.b:bitwise_xor", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -10> 2022-01-06T12:43:01.652+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981656222, "cf_name": "p-0", "job": 1, "event": "table_file_creation", "file_number": 3220, "file_size": 5104328, "table_properties": {"data_size": 4982595, "index_size": 70910, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 49989, "raw_key_size": 973446, "raw_average_key_size": 48, "raw_value_size": 4492366, "raw_average_value_size": 224, "num_data_blocks": 1298, "num_entries": 19980, "num_deletions": 10845, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "p-0", "column_family_id": 4, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -9> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981692873, "cf_name": "p-2", "job": 1, "event": "table_file_creation", "file_number": 3221, "file_size": 5840600, "table_properties": {"data_size": 5701923, "index_size": 81198, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 56645, "raw_key_size": 1103994, "raw_average_key_size": 48, "raw_value_size": 5146222, "raw_average_value_size": 227, "num_data_blocks": 1485, "num_entries": 22623, "num_deletions": 12166, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "p-2", "column_family_id": 6, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -8> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981694121, "cf_name": "O-0", "job": 1, "event": "table_file_creation", "file_number": 3222, "file_size": 73885, "table_properties": {"data_size": 72021, "index_size": 588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9444, "raw_average_key_size": 60, "raw_value_size": 63028, "raw_average_value_size": 406, "num_data_blocks": 18, "num_entries": 155, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-0", "column_family_id": 7, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -7> 2022-01-06T12:43:01.688+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981695243, "cf_name": "O-1", "job": 1, "event": "table_file_creation", "file_number": 3223, "file_size": 71023, "table_properties": {"data_size": 69158, "index_size": 589, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9028, "raw_average_key_size": 61, "raw_value_size": 60508, "raw_average_value_size": 408, "num_data_blocks": 18, "num_entries": 148, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-1", "column_family_id": 8, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -6> 2022-01-06T12:43:01.692+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981696397, "cf_name": "O-2", "job": 1, "event": "table_file_creation", "file_number": 3224, "file_size": 75263, "table_properties": {"data_size": 73370, "index_size": 617, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 453, "raw_key_size": 9679, "raw_average_key_size": 60, "raw_value_size": 64238, "raw_average_value_size": 404, "num_data_blocks": 19, "num_entries": 159, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "O-2", "column_family_id": 9, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -5> 2022-01-06T12:43:01.696+0000 7feaa5390080 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1641472981700113, "cf_name": "L", "job": 1, "event": "table_file_creation", "file_number": 3225, "file_size": 338198, "table_properties": {"data_size": 335953, "index_size": 1100, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 325, "raw_key_size": 1712, "raw_average_key_size": 16, "raw_value_size": 333803, "raw_average_value_size": 3119, "num_data_blocks": 39, "num_entries": 107, "num_deletions": 68, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "L", "column_family_id": 10, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1641472981, "oldest_key_time": 3, "file_creation_time": 0}}
debug -4> 2022-01-06T12:43:01.784+0000 7feaa5390080 1 bluefs _allocate unable to allocate 0x80000 on bdev 1, allocator name block, allocator type bitmap, capacity 0xeffc00000, block size 0x1000, free 0x10c7c2000, fragmentation 1, allocated 0x0
debug -3> 2022-01-06T12:43:01.784+0000 7feaa5390080 -1 bluefs _allocate allocation failed, needed 0x7b2e8
debug -2> 2022-01-06T12:43:01.784+0000 7feaa5390080 -1 bluefs _flush_range allocated: 0xc90000 offset: 0xc8a944 length: 0x809a4
debug -1> 2022-01-06T12:43:01.792+0000 7feaa5390080 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7feaa5390080 time 2022-01-06T12:43:01.789216+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc")
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x558699f6ac8c]
2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x55869a661901]
3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55869a661be0]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x55869a672cf2]
5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x55869a68b32b]
6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55869ab1dacf]
7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x55869ac2f81a]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x55869ac30c70]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x55869ad4c416]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x55869ad4cd5c]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x55869ad4d47c]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x55869ad4d50d]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x55869ad50978]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x55869acfb3d5]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x55869ab60415]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x55869ab62b4e]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x55869ab63ea8]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x59d) [0x55869ab5dbcd]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x15) [0x55869ab5ef65]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x55869aad6ec1]
21: (BlueStore::_open_db(bool, bool, bool)+0x948) [0x55869a55c4d8]
22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x55869a5c6657]
23: (BlueStore::_mount()+0x204) [0x55869a5c9514]
24: (OSD::init()+0x380) [0x55869a09ea10]
25: main()
26: __libc_start_main()
27: _start()
debug 0> 2022-01-06T12:43:01.796+0000 7feaa5390080 -1 *** Caught signal (Aborted) **
in thread 7feaa5390080 thread_name:ceph-osd
以上是down OSD中的日志。我开始阅读它并找到一个有用的日志以在 google 中搜索：
bluefs _allocate unable to allocate 0x80000 on bdev 1, allocator name block, allocator type bitmap, capacity 0xeffc00000, block size 0x1000, free 0x10c7c2000, fragmentation 1, allocated 0x0
我发现了一个与版本 16.2.1 相关的错误（我使用 16.2.6）： https ://tracker.ceph.com/issues/50656
我想从我的 OSD 中得到一个转储（我不完全理解他们在说什么）：
ceph daemon osd.1 bluestore allocator dump block
Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n"
我使用使用容器的 cephadm 部署集群，所以我认为我无法以这种方式访问套接字。这个命令引导我使用 ceph-bluestore-tool 查看我的物理磁盘的状态（查看容量、运行 fsck 或修复）但运行 ceph-bluestore-tool 需要指定我无法运行的 osd 的 --path主机（我的容器不断崩溃，所以我无法在容器内运行此命令）。我尝试使用 cephadm 在 osd 容器中运行命令，但无论如何都找不到这样做。
如果您需要此处的完整日志，请告诉我（由于字符限制，我无法发送）但它是相同的崩溃日志
我真的不明白发生了什么事。
我尝试使用 ceph-volume 将块设备挂载到主机上使用 ceph-bluestore-tool 在其上使用fsck或repair在其上使用。（它需要一个 --path 参数来指向 osd 文件）（我什至不知道以这种方式使用 ceph-volume 是否正确，或者它是为此而构建的 - 正如我告诉我的 Ceph 新手）
我尝试使用 cephadm 在崩溃的 OSD 中运行 ceph-bluestore-tool 命令，但我做不到。（我上面提到的socket错误）
我的 SSD OSD 已填满 94%，所以其他的 OSD 上仍有可用空间（我猜）。
我可以在互联网上找到的唯一线索不起作用。
我真的很想找到答案。如果你能帮助我，我会很高兴。甚至告诉我阅读文档或学习一些东西。
我将在此处发布有关我的集群的一些信息：
[Ceph 仪表板][1]
>> ceph -s
cluster:
id: 1ad06d18-3e72-11ec-8684-fd37cdad1703
health: HEALTH_WARN
mons ceph-n01,ceph-n02,ceph-n04 are low on available space
2 backfillfull osd(s)
Degraded data redundancy: 4282822/12848466 objects degraded (33.333%), 64 pgs degraded, 96 pgs undersized
3 pool(s) backfillfull
6 daemons have recently crashed
services:
mon: 3 daemons, quorum ceph-n01,ceph-n04,ceph-n02 (age 7h)
mgr: ceph-n02.xyrntr(active, since 4w), standbys: ceph-n04.srrvqt
mds: 1/1 daemons up, 1 standby
osd: 3 osds: 2 up (since 6h), 2 in (since 6h)
data:
volumes: 1/1 healthy
pools: 3 pools, 96 pgs
objects: 4.28M objects, 42 GiB
usage: 113 GiB used, 6.7 GiB / 120 GiB avail
pgs: 4282822/12848466 objects degraded (33.333%)
64 active+undersized+degraded
32 active+undersized
>> ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 4m ago 8w count:1
crash 3/3 6m ago 8w *
grafana ?:3000 1/1 4m ago 8w count:1
mds.cephfs 2/2 6m ago 7w label:mds
mgr 2/2 6m ago 8w label:mgr
mon 3/5 6m ago 3w count:5
node-exporter ?:9100 3/3 6m ago 8w *
osd 3/3 6m ago - <unmanaged>
prometheus ?:9095 1/1 4m ago 8w count:1
>> ceph orch host ls
HOST ADDR LABELS STATUS
ceph-n01 192.168.2.20 _admin mon mds
ceph-n02 192.168.2.21 mon mgr
ceph-n04 192.168.2.23 _admin mon mds mgr
>> ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.17578 root default
-3 0.05859 host ceph-n01
1 ssd 0.05859 osd.1 up 1.00000 1.00000
-5 0.05859 host ceph-n02
0 ssd 0.05859 osd.0 up 1.00000 1.00000
-10 0.05859 host ceph-n04
2 hdd 0.05859 osd.2 down 0 1.00000
>> ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 ssd 0.05859 1.00000 60 GiB 57 GiB 50 GiB 2.1 GiB 4.6 GiB 3.4 GiB 94.37 1.00 96 up
0 ssd 0.05859 1.00000 60 GiB 57 GiB 50 GiB 2.1 GiB 4.6 GiB 3.4 GiB 94.41 1.00 96 up
2 hdd 0.05859 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
TOTAL 120 GiB 113 GiB 100 GiB 4.3 GiB 9.1 GiB 6.7 GiB 94.39
MIN/MAX VAR: 1.00/1.00 STDDEV: 0.02
node01: >> free -m
total used free shared buff/cache available
Mem: 7956 7121 175 0 659 573
Swap: 10238 3748 6490
node02: >> free -m
total used free shared buff/cache available
Mem: 7956 7121 175 0 659 573
Swap: 10238 3748 6490
node04: >> free -m
total used free shared buff/cache available
Mem: 7922 2260 3970 1 1690 5371
Swap: 10238 642 9596