How to Install Apache Kafka on Oracle Linux 8

Apache Kafka is a free, open-source, distributed event streaming platform that is used by thousands of companies for high-performance data pipelines. It is a message broker software application primarily used to build real-time streaming data pipelines and applications. Apache Kafka allows you to process data streams via a distributed streaming platform. It is based on a distributed architecture, so it provides high fault tolerance and scalability capabilities.

In this post, we will explain how to install Apache Kafka on Oracle Linux 8.

Step 1 – Install Java

Apache Kafka is based on Java, so you will need to install Java on your server. You can install it by running the following command:

dnf install java-11-openjdk-devel -y

Once Java is installed, verify the Java installation using the following command:

java --version

You will get the Java version in the following output:

openjdk 11.0.15 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10-LTS, mixed mode, sharing)

Step 2 – Install Apache Kafka on Oracle Linux 8

First, go to the Apache official website and download the latest version of Apache Kafka using the wget command:

wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

Once the download is completed, extract the downloaded file using the following command:

tar -xvzf kafka_2.13-3.2.0.tgz

Next, move the extracted directory to /usr/local directory:

mv kafka_2.13-3.2.0/ /usr/local/kafka

Once you are finished, you can proceed to the next step.

Step 3 – Create Systemd Service File for Zookeeper and Kafka

It is a good idea to create a systemd service file to run both Zookeeper and Kafka in the background.

First, create a systemd service file for Zookeeper with the following command:

nano /etc/systemd/system/zookeeper.service

Add the following lines:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file, then create a systemd service file for Kafka using the following command:

nano /etc/systemd/system/kafka.service

Add the following lines:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon with the following command:

systemctl daemon-reload

Next, start both the Zookeeper and Kafka services and enable them to start at system reboot:

systemctl start zookeeper
systemctl start kafka
systemctl enable zookeeper
systemctl enable kafka

To check the Zookeeper service, run the following command:

systemctl status zookeeper

You will get the following output:

● zookeeper.service - Apache Zookeeper server
   Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-11 01:06:50 EDT; 28s ago
     Docs: http://zookeeper.apache.org
 Main PID: 29634 (java)
    Tasks: 26 (limit: 11409)
   Memory: 67.9M
   CGroup: /system.slice/zookeeper.service
           └─29634 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCI>

Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,302] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,311] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeepe>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,312] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshot loaded in 18 ms, highest zxid is 0x0, digest is 1371985504 >
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,324] INFO Snapshot taken in 2 ms (org.apache.zookeeper.server.ZooKeeperServer)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,378] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zook>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,388] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apa>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,446] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalM>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,447] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProv>
lines 1-20/20 (END)

To check the Kafka service, run the following command:

systemctl status kafka

You should see the following output:

● kafka.service - Apache Kafka Server
   Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-11 01:07:17 EDT; 55s ago
     Docs: http://kafka.apache.org/documentation.html
 Main PID: 29994 (java)
    Tasks: 69 (limit: 11409)
   Memory: 327.3M
   CGroup: /system.slice/kafka.service
           └─29994 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupanc>

Jun 11 01:07:23 oraclelinux8 bash[29994]: [2022-06-11 01:07:23,975] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNod>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,001] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Starting socket serv>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane a>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started socket serve>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka version: 3.2.0 (org.apache.kafka.common.utils.AppInfoParser)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka commitId: 38103ffaa962ef50 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka startTimeMs: 1654924044055 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,158] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,377] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorde>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,378] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Rec>

Step 4 – Create Your First Topic on Kafka

Next, you will need to create at least one topic on the server to test Apache Kafka.

Change the directory to Apache Kafka and create a test topic named app1 with the following command:

cd /usr/local/kafka/
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic app1

You can now verify your created topic using the following command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You will get the following output:

app1

Kafka provides two APIs: producer and consumer. The producer is responsible for creating events and the consumer uses them to display it on the screen:

Now, run the producer and then type a few messages into the console to send to the server.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic app1

Type some text that you want to stream and display on the consumer.

>Hi, this is my first kafka server

Open another terminal and run the following command to display the generated event data in real-time:

cd /usr/local/kafka/
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic app1 --from-beginning

You will get the following output:

Hi, this is my first kafka server

Conclusion

In the above guide, we explained how to install Apache Kafka on Oracle Linux 8. You can now integrate Apache Kafka with your application to collect and analyze a large amount of data. For more information, you can visit the Apache Kafka documentation page. Try it on dedicated hosting from Atlantic.Net!