0

作为 DR 计划的一部分,我正在实施 Kafka MirrorMaker 2.0 以在 2 个集群之间复制数据。

在这一点上,我只是想做一个简单的主动/被动设计。

这是我的 mm2.properties 文件:

# Licensed to the Apache Software Foundation (ASF) under A or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see org.apache.kafka.clients.consumer.ConsumerConfig for more details

# Sample MirrorMaker 2.0 top-level configuration file
# Run with ./bin/connect-mirror-maker.sh connect-mirror-maker.properties 

# specify any number of cluster aliases
clusters = west, central

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
west.bootstrap.servers = kafka-west.com:30002
central.bootstrap.servers = kafka-central.com:30002

# enable and configure individual replication flows
west->central.enabled = true
central->west.enabled = false

# Setting replication factor of newly created remote topics
replication.factor=3

############################# Internal Topic Settings  #############################
# The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3

# The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3

# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5

问题在这里(我认为?):

west->central.enabled = true
central->west.enabled = false

无论我如何配置流程,我只能看到正在写入west集群的数据。即使两个流都为真,west集群也会得到west.topiccentral.topic。同时,central集群永远不会得到任何复制。

我还在两个集群中运行了这个 Mirror-Maker 配置,它总是west由于某种原因被写入集群。

我不明白为什么会发生这种情况,提前感谢您的帮助。

EDIT: I'm losing my mind a bit. I switched both URLs to be central like this:

west.bootstrap.servers = kafka-central.com:30002
central.bootstrap.servers = kafka-central.com:30002

and it STILL writes to the west cluster... how is this possible, when he doesn't even know where the west cluster is? I even checked the DNSs and they are correct.

4

1 回答 1

0

Turns out there was something in my brokers on the central cluster that were pointing to the Loadbalancer on the west cluster.

After fixing the brokers the replication flow was correct again

于 2021-09-23T13:37:24.343 回答