Kafka

Configuration settings for Kafka

1: Howtos

1.1: How to generating cluster ID

Kubernetes logging helm chart deploys Apache Kafka as a message broker between FluentBit and Logstash for improving stability and loadbalance in big deployments.

From helm chart version 4.6.0 we omited Apache ZooKeeper as Kafka dependency. Kafka from version 2.8.0 introduced KRaft aka ZooKeeper-less mode. From Kafka version 3.3.0 is KRaft marked as production ready, so, we decide to adopt it in the logging helm chart to save some resources and deploying time. Kafka in Raft mode need to have generated cluster ID. Please check how to generating cluster ID.

1 - Howtos

How to achieve …

Here you can find guidelines, how to manipulate with Kafka in context of logging helm chart.

1.1 - How to generating cluster ID

How to generating cluster ID

Cluster ID is required for each Kafka instance to know, to which instance it must to connect and make cluster. It is also used to preparing storage space. We need to set all Kafka instance same cluster ID. If you omit this settings, all Kafka instance generate their own ID and reject make cluster.

There is many ways, how to generate the ID, but I recommend use this chain of Bash command:

$ cat /proc/sys/kernel/random/uuid | tr -d '-' | base64 | cut -b 1-22

You can also use built-in script:

$ bin/kafka-storage.sh random-uuid

Or just start one Kafka instance without the cluster ID settings and it will be generated to the logs.

Sources:

Blog sleeplessbeastie.eu
Apache Kafka Documentation