Apache Kafka is a famous distributed event streaming platform that ships with plenty of features & functionalities. For example, Replication, JBOD, Authn/z, Encryption, Compression, At-most/At-least/Exactly once delivery, and Transaction. Some common use-cases covered by Kafka include high-performance data pipelines, streaming analytics, and data integration.
A typical Kafka cluster will experience business requirement changes and resource allocation changes. A resource imbalance issue will stem once a Kafka cluster suffers from these everyday situations. If one doesn’t handle these issues, a cluster will experience performance problems, which eventually, makes the upper-layer application work unstable.
To resolve this issue, we proposed a new open-source cluster balance tool for Apache Kafka. The Astraea Balancer. We take advantage of custom rebalance plan generation and various load balance goals to maintain the stability of a cluster. This will ease the maintenance work for cluster administrators.