java - Batch size in Storm Trident

Question

I would like to know how to set BATCH SIZE (example I want a batch to have 10000 records) in TRIDENT. I have the below configuration in my code and this fetches approximately 250 records per batch from Kafka. Can I increase this to 10000*1024? or is there a way to set No of records per batch.

TridentKafkaConfig tridentKafkaConfig = new TridentKafkaConfig(hosts, topic, consumerGroupId);
tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new XScheme()); 
tridentKafkaConfig.fetchSizeBytes = 1024*1024;

Answer 1

您可以通过更改“ tridentKafkaConfig.fetchSizeBytes ”属性来增加批量大小。此外，批量大小与代理数量和分区数量有关。例如，如果您有 2 个代理，每个代理有 3 个分区，这意味着分区的总数为 6。这样，批量大小等于tridentKafkaConfig.fetchSizeBytes X 总分区计数。如果我们假设 tridentKafkaConfig.fetchSizeBytes 为 1024X1024，则批量大小等于 6 MB.(3x2x1024x1024)bytes