singlestore - How much memory should be reserved to store 100GB data

Question

I have 5 leaf nodes with each node having setting

maximum_memory = 51200m
maximum_table_memory = 40960m
So total leaf table memory = 200GB
and total  Memsql Memmory = 250 GB

I am trying to load 100GB data of TPCH but not able to do so. LineItem Table loading is failed. Its count is 0.

Below is the picture of status table of memsql ops. It varies between 90GB and 108GB for leaf Table memory

When I tried to load LINEItem table again whose size is 74GB, status tab shows memsql is consuming all available memory

So even after setting more that double amount of memory (200GB), Memsql is still not able to load 100GB data. What is wrong here? Looks like I am missing on some obvious parameters.Please guide me for correct settings.

Answer 1

将数据存储在行存储表中通常需要比原始数据大小更多的空间，因为索引、开销等消耗了额外的空间。MemSQL 和其他数据库都是如此。例如，要减少空间使用，请考虑删除架构中的一些索引。或者您可能还想检查您是否使用了正确的数据类型——例如，使用 bigint 而不是 int 也会浪费空间。

您可以根据 MemSQL 实际为您的数据使用多少空间来估计需要多少空间 - 从您在上面发布的信息看来，MemSQL 使用大约 200GB 来存储 74GB 原始数据，因此对于 100GB 原始数据，乘以 ~270GB数据。

有关 MemSQL 使用内存的更多详细信息，您可以通过 Cluster 页面查看 Ops 中的单个 MemSQL 叶节点并查看内存使用信息。