How to Install Apache Kafka on Rocky Linux 8

Apache Kafka is an open-source stream processing and message broker software application that allows you to process data streams via a distributed streaming platform. It acts as a messaging system between the sender and the recipient. Apache Kafka is based on a distributed architecture, so it provides high fault tolerance and scalability capabilities. It was originally developed by LinkedIn, but now it is a project of Apache Software Foundation. Apache Kafka provides an interface to read and write data to Kafka clusters or to import and export data to and from third-party systems.

In this post, we will explain how to install Apache Kafka on Rocky Linux 8.

Step 1- Install Java

Apache Kafka is a Java-based application, so Java must be installed on your server. If not installed, you can install it using the following command:

dnf update -y
dnf install java-11-openjdk-devel -y

Once Java is installed, verify the Java installation using the following command:

java --version

You will get the Java version in the following output:

openjdk 11.0.13 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)

Step 2 – Install Apache Kafka on Rocky Linux 8

First, go to the Apache official website and download the latest version of Apache Kafka using the wget command:

wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

Once the download is completed, extract the downloaded file using the following command:

tar -xvzf kafka_2.13-3.0.0.tgz

Once the downloaded file is extracted, move the extracted directory to /usr/local directory:

mv kafka_2.13-3.0.0/ /usr/local/kafka

Once you are finished, you can proceed to the next step.

Step 3 – Create Systemd Service File for Zookeeper and Kafka

For the production environment, it is recommended to create a systemd service file to run both Zookeeper and Kafka in the background.

First, create a systemd service file for Zookeeper with the following command:

nano /etc/systemd/system/zookeeper.service

Add the following lines:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file, then create a systemd service file for Kafka using the following command:

nano /etc/systemd/system/kafka.service

Add the following lines:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon with the following command:

systemctl daemon-reload

Next, start both Zookeeper and Kafka services and enable them to start at system reboot:

systemctl start zookeeper
systemctl start kafka
systemctl enable zookeeper
systemctl enable kafka

You can also check both services using the following command:

systemctl status zookeeper kafka

You will get the following output:

● zookeeper.service - Apache Zookeeper server
   Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-22 07:51:56 UTC; 42s ago
     Docs: http://zookeeper.apache.org
 Main PID: 33802 (java)
    Tasks: 28 (limit: 11411)
   Memory: 91.4M
   CGroup: /system.slice/zookeeper.service
           └─33802 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCIn>

Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,676] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,676] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.z>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,679] INFO Snapshot loaded in 22 ms, highest zxid is 0x0, digest is 1371985504 (o>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,679] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.z>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,683] INFO Snapshot taken in 3 ms (org.apache.zookeeper.server.ZooKeeperServer)
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,703] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookee>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,741] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apach>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,764] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,765] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvid>
Oct 22 07:52:12 RockyLinux8 bash[33802]: [2021-10-22 07:52:12,102] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.>

● kafka.service - Apache Kafka Server
   Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-22 07:52:09 UTC; 29s ago
     Docs: http://kafka.apache.org/documentation.html
 Main PID: 34147 (java)
    Tasks: 69 (limit: 11411)
   Memory: 331.6M
   CGroup: /system.slice/kafka.service
           └─34147 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancy>

Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,881] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeC>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,899] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Starting socket server>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,981] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acc>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,988] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started socket server >
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser)
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInf>
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka startTimeMs: 1634889134988 (org.apache.kafka.

Step 4 – Create a Topic on Kafka

To test Apache Kafka, you will need to create at least one topic on the server.

Change the directory to Apache Kafka and create a test topic named topic1 with the following command:

cd /usr/local/kafka/
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic1

You can now verify your created topic using the following command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You will get the following output:

topic1

Kafka provides two APIs: Producer and Consumer. The Producer is responsible for creating events and the Consumer displays them on the screen:

First, run the following command to create an event named event1 using the following command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic event1

Type some text that you want to stream and display on the Consumer.

>Hi, this is my first event

Sample output:

[2021-10-22 07:58:05,318] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {event1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Open another terminal and run the following command to display the generated event data in real-time:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event1 --from-beginning

You will get the following output:

Hi, this is my first event

Conclusion

In the above guide, you learned how to install Apache Kafka on Rocky Linux 8. For more information, you can visit the Apache Kafka documentation page. Get started with Apache Kafka on VPS hosting from Atlantic.Net!