1

I have setup a mirrormaker2 on 2 clusters (CL1,CL2) and the mirroring seems to work properly except with an issue with duplicates in a test topic in the following scenario:

While both clusters are up and running i simulate an incident, stopping one by one the brokers of the CL2 cluster. Stopping the first two brokers does not generate any issue. All messages of my test topic are mirrored without problems on CL1.topic of CL2 cluster. After stopping the last broker, obviously will stop messages in the CL2 side as all brokers are down. There is always active a producer that feeds with messages during the test on topic of CL1.

The problem starts on restarting the brokers. After starting the first broker i note that some messages (about 5%) are duplicated. I have connected a client on CL1.topic and i can confirm that indeed there are duplicated messages in my mirrored topic. Kindly suggest how i could avoid these duplicates. Idempotence may not work correctly during broker shutdown?

In the following you can find my MM2 relative config

clusters = CL1, CL2
CL1.bootstrap.servers = broker1CL1:9092, broker2CL1:9092, broker3CL1:9092
CL2.bootstrap.servers = broker1CL2:9092, broker2CL2:9092, broker3CL2:9092

PRIM->DSTR.enabled = true
DSTR->PRIM.enabled = true

CL1.producer.enable.idempotence = true
CL1.producer.acks=all
CL1.producer.max.in.flight.requests.per.connection=5
CL1.producer.retries=2147483647
CL1.consumer.isolation.level=read_committed
CL2.producer.enable.idempotence = true
CL2.producer.acks=all
CL2.producer.max.in.flight.requests.per.connection=5
CL2.producer.retries=2147483647
CL2.consumer.isolation.level=read_committed
4

1 回答 1

1

这个问题已经在apache kafka 官方邮件列表上得到了回应。

答案是这样的：Kafka 交付保证可以分为三组，包括“最多一次”、“至少一次”和“恰好一次”。默认情况下，当前的 MM 2.0 保证“至少”一次交付保证，这意味着在某些失败场景下会出现重复消息。对于不了解的更多信息，我建议阅读此链接

于 2021-04-02T09:50:19.387 回答