Quick command reference for Apache Kafka

kafka-cheat-sheet.md

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

Delete a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic

Get number of messages in a topic ???

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

Get the earliest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

Get the latest offset still in a topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

Consume messages with the console consumer

bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic mytopic --from-beginning

Get the consumer offsets for a topic

bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group

Read from __consumer_offsets

Add the following property to config/consumer.properties: exclude.internal.topics=false

bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

Kafka Consumer Groups

List the consumer groups known to Kafka

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list (old api)

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list (new api)

View the details of a consumer group

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group <group name>

kafkacat

Getting the last five message of a topic

kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e

Zookeeper

Starting the Zookeeper Shell

bin/zookeeper-shell.sh localhost:2181

The command for "Get number of messages in a topic ???" will only work if our earliest offsets are zero, correct?
If we have a topic, whose message retention period already passed (meaning some messages were discarded and new ones were added), we would have to get the earliest and latest offsets, subtract them for each partition accordingly and then add them, right?

You just need to wrap the two calculations in a script, and then calculate the difference. An example:
https://gist.github.com/davewat/b3c5c4a383d1d9d26df33ecbb6ddd668

View the details of a consumer group: passing --zookeeper shows old api consumer groups only (as for List Consumer Groups)
For new API's consumers use --bootstrap-server
It might be obvious, but it is worth mentioning it explicitly :)
(long time no see @ursuad :) )

Is their something that would list latest offsets by partitions?
Thanks in advance.

If someone wants to see a consumer group's progress, you may use this.

Some of these commands are not working. For example --zookeeper is not a valid option for listing consumer groups. Instead, need to pass broker as argument.
Working command:
kafka-consumer-groups --bootstrap-server {Broker_List} --list

One important note to following scripts and gist mentioned by @davewat - these counts does not reflect deleted messages in compacted topic. We learned this the hard way...

List the consumer groups known to Kafka
new api
you can omit the option --new-consumer (kafka 1.1.0)

The [new-consumer] option is deprecated and will be removed in a future major release...

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list (new api)

Offsets for a group

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group mygroup

........kafka_2.12-2.1.0\bin\windows>kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe

some commands need to be modified

Get number of messages in a topic

echo "$(($(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 | awk -F ":" '{sum += $3} END {print sum}')-$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2 | awk -F ":" '{sum += $3} END {print sum}')))"

Topic size:
kafka-log-dirs.sh --describe --bootstrap-server $kafkaHost:9092 --topic-list $topicName

https://gist.github.com/ursuad/e5b8542024a15e4db601f34906b30bb5#read-from-__consumer_offsets

For Kafka 11 onwards, the formatter to be used is kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter

Get number of messages in a Topic
GetOffsetShell only worked in PLAINTEXT, how to achieve this in SASL_PALINTEXT?

--command-config /path/to/file.properties with the files.properties containing the ```security.protocol=SASL_SSL````

PLAIN versus PLAINTEXT: Do not confuse the SASL mechanism PLAIN with the no TLS/SSL encryption option, which is called PLAINTEXT. Configuration parameters such as sasl.enabled.mechanisms or sasl.mechanism.inter.broker.protocol may be configured to use the SASL mechanism PLAIN, whereas security.inter.broker.protocol or listeners may be configured to use the no TLS/SSL encryption option, SASL_PLAINTEXT.