作为 DR 计划的一部分,我正在实施 Kafka MirrorMaker 2.0 以在 2 个集群之间复制数据。
在这一点上,我只是想做一个简单的主动/被动设计。
这是我的 mm2.properties 文件:
# Licensed to the Apache Software Foundation (ASF) under A or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see org.apache.kafka.clients.consumer.ConsumerConfig for more details
# Sample MirrorMaker 2.0 top-level configuration file
# Run with ./bin/connect-mirror-maker.sh connect-mirror-maker.properties
# specify any number of cluster aliases
clusters = west, central
# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
west.bootstrap.servers = kafka-west.com:30002
central.bootstrap.servers = kafka-central.com:30002
# enable and configure individual replication flows
west->central.enabled = true
central->west.enabled = false
# Setting replication factor of newly created remote topics
replication.factor=3
############################# Internal Topic Settings #############################
# The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
# The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3
# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5
问题在这里(我认为?):
west->central.enabled = true
central->west.enabled = false
无论我如何配置流程,我只能看到正在写入west
集群的数据。即使两个流都为真,west
集群也会得到west.topic
和central.topic
。同时,central
集群永远不会得到任何复制。
我还在两个集群中运行了这个 Mirror-Maker 配置,它总是west
由于某种原因被写入集群。
我不明白为什么会发生这种情况,提前感谢您的帮助。
EDIT:
I'm losing my mind a bit. I switched both URLs to be central
like this:
west.bootstrap.servers = kafka-central.com:30002
central.bootstrap.servers = kafka-central.com:30002
and it STILL writes to the west
cluster... how is this possible, when he doesn't even know where the west
cluster is? I even checked the DNSs and they are correct.