kafka迁移与扩容工具使用
参考官网site:https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
说明:
当我们对kafka集群扩容时,需要满足2点要求:
- 将指定topic迁移到集群内新增的node上。
- 将topic的指定partition迁移到新增的node上。
1. 迁移topic到新增的node上
--topics-to-move-json-file migration-push-token-topic.json --broker-list "104,105,106" --generate
脚本migration-push-token-topic.json文件内容如下:
{
"topics":
[
{
"topic": "push-token-topic"
}
],
"version":1
}
生成分配partitions的json脚本:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[8]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":3,"replicas":[5]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":9,"replicas":[5]},{"topic":"cluster-switch-topic","partition":1,"replicas":[5]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[5]},{"topic":"cluster-switch-topic","partition":2,"replicas":[4]},{"topic":"cluster-switch-topic","partition":0,"replicas":[4]},{"topic":"cluster-switch-topic","partition":6,"replicas":[4]},{"topic":"cluster-switch-topic","partition":8,"replicas":[4]}]}
重新分配parttions的json脚本如下:
migration-topic-cluster-switch-topic.json
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[5]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":3,"replicas":[4]},{"topic":"cluster-switch-topic","partition":9,"replicas":[4]},{"topic":"cluster-switch-topic","partition":1,"replicas":[4]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[4]},{"topic":"cluster-switch-topic","partition":2,"replicas":[5]},{"topic":"cluster-switch-topic","partition":0,"replicas":[5]},{"topic":"cluster-switch-topic","partition":6,"replicas":[5]},{"topic":"cluster-switch-topic","partition":8,"replicas":[5]}]}
lizhitao@localhost:$ bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka01 --reassignment-json-file migration-topic-cluster-switch-topic.json --execute
2.topic修改(replicats-factor)副本个数
lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka
--reassignment-json-file replicas-update-push-token-topic.json --execute
假如初始时push-token-topic为一个副本,为了提高可用性,需要改为2副本模式。
脚本replicas-push-token-topic.json文件内容如下:
{
"partitions":
[
{
"topic": "log.mobile_nginx",
"partition": 0,
"replicas": [101,102,104]
},
{
"topic": "log.mobile_nginx",
"partition": 1,
"replicas": [102,103,106]
},
{
"topic": "xxxx",
"partition": 数字,
"replicas": [数组]
}
],
"version":1
}
3.topic的分区扩容用法
a.先扩容分区数量,脚本如下:
例如:push-token-topic初始分区数量为12,目前到增加到15个
lizhitao@localhost:$ ./bin/kafka-topics.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka --alter --partitions 15 --topic push-token-topic
b.设置topic分区副本
lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka
--reassignment-json-file partitions-extension-push-token-topic.json --execute
脚本partitions-extension-push-token-topic.json文件内容如下:
- {
- "partitions":
- [
- {
- "topic": "push-token-topic",
- "partition": 12,
- "replicas": [101,102]
- },
- {
- "topic": "push-token-topic",
- "partition": 13,
- "replicas": [103,104]
- },
- {
- "topic": "push-token-topic",
- "partition": 14,
- "replicas": [105,106]
- }
- ],
- "version":1
- }
相关推荐
使用Kafka的监控工具(如Kafka Metrics API、JMX或Prometheus)可以对系统的健康状况进行实时监控。 通过阅读《Apache Kafka实战》这本书,你可以深入了解Kafka的工作原理,学习如何设计、部署和维护一个高效、可靠...
This book is here to help you get familiar with Apache Kafka and to solve your challenges related to the consumption of millions of messages in publisher-subscriber architectures. It is aimed at ...
Spring for Apache Kafka API。 Spring for Apache Kafka 开发文档。
在部署Kafka时,可以使用多种工具和方法来搭建Kafka集群,确保集群的高可用性和可扩展性。同时,根据不同的使用场景,Kafka也支持多种配置选项,如消息保留策略、分区数量、副本因子等,以满足不同的业务需求。 ...
Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records and process them in ...
总之,Apache Kafka和MapR Streams结合使用,为构建现代化的流处理架构提供了强大的工具集。这些技术有助于企业实时捕获、处理和分析海量数据,从而提升业务效率,实现快速响应市场变化的能力。对于希望在大数据时代...
《Apache Kafka Cookbook》是由PACKT在2015年出版的一本专著,专注于介绍Apache Kafka这一分布式流处理平台的实战技巧和最佳实践。Apache Kafka是一个高性能、可扩展且容错性强的消息中间件,它被广泛应用于大数据...
3. **创建主题**:使用 `kafka-topics.sh` 工具创建一个主题(topic)。 4. **生产者和消费者测试**:分别运行生产者和消费者示例程序,观察数据传输过程。 #### 四、API 概览 Kafka 提供了多种 API 来满足不同的...
Building Data Streaming Applications with Apache Kafka 英文azw3 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
### Apache Kafka:设置集群与开发消息生产者及消费者 #### Apache Kafka概述 Apache Kafka是一种开源流处理平台,由LinkedIn公司创建并捐赠给Apache软件基金会。Kafka被设计为一种高吞吐量、分布式的消息系统,它...
Learning Apache Kafka Second Edition provides you with step-by-step, practical examples that help you take advantage of the real power of Kafka and handle hundreds of megabytes of messages per second ...
《Apache Kafka实战》一书是针对分布式流处理平台Apache Kafka的专业指南,旨在帮助读者深入理解和应用Kafka在实际项目中的各种场景。Kafka是一个高吞吐量、低延迟的消息发布订阅系统,最初由LinkedIn开发,后来成为...
Kafka是一个对于像Hadoop的一样的日志数据和离线分析系统,但又要求实时处理的限制,这是一个可行的解决方案。 1.Kafka集群包含一个或多个服务器,这种服务器被称为broker 2.Partition是物理上的概念,每个Topic...
【Apache Jmeter Kafka Jar包】是Apache JMeter的扩展组件,专为测试和性能评估Apache Kafka消息系统设计。这个jar包允许用户在JMeter测试计划中直接发送和接收Kafka消息,从而对Kafka集群的性能进行深入分析。 ...
Kafka 是由 LinkedIn 开发的一个分布式的消息系统, 使用 Scala 编写, 它以可水平扩展和高吞吐率而被广泛使用。 目前越来越多的开源分布式处理系统如Cloudera、 Apache Storm、Spark 都支持与 Kafka 集成。
Apache Kafka是一种分布式...总之,Apache Kafka是大数据领域的重要工具,广泛应用于实时数据处理、日志收集和事件驱动的系统设计中。了解其核心概念、应用场景和最佳实践,能够帮助我们更好地利用Kafka解决实际问题。