Atlantic.Net Blog

How to Install Apache Kafka on Fedora

Apache Kafka is an open-source distributed event streaming platform developed by the Apache Software Foundation. It is written in Java and Scala and used for high-performance data pipelines and streaming analytics. Apache Kafka is a distributed message broker designed to handle large volumes of data. It is highly scalable and can run on one or more servers.

This tutorial will show you how to install Apache Kafka on Fedora.


Step 1 – Install Java OpenJDK

First, install the Java JDK using the following commands.

dnf update -y
dnf install java-11-openjdk -y

After installing Java, verify the Java installation with the following command.

java --version

Output.

openjdk 11.0.15 2022-04-19
OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode, sharing)

Step 2 – Install Apache Kafka

First, visit the Kafka official download page, pick the latest Kafka version, and run the following command to download it to your server.

wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Next, extract the downloaded file using the following command.

tar xzf kafka_2.13-3.4.0.tgz

Next, move the extracted directory to /usr/local directory.

mv kafka_2.13-3.4.0 /usr/local/kafka 

Step 3 – Create a Systemd Service File for Kafka

Next, you must create a systemd service file to manage the Kafka service via systemd.

First, create a Zookeeper service file.

nano /etc/systemd/system/zookeeper.service

Add the following configurations.

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save the file, then create a Kafka service file.

nano /etc/systemd/system/kafka.service

Add the following configurations.

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon to apply the changes.

systemctl daemon-reload

Step 4 – Start Apache Kafka

Now, start the Apache Kafka and Zookeeper services and enable them to start at system reboot.

systemctl start zookeeper
systemctl start kafka
systemctl enable zookeeper
systemctl enable kafka

You can now check the status of Kafka using the following command.

systemctl status kafka

Output.

● kafka.service - Apache Kafka Server
     Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
     Active: active (running) since Sat 2023-04-22 23:21:35 EDT; 4s ago
       Docs: http://kafka.apache.org/documentation.html
   Main PID: 4830 (java)
      Tasks: 54 (limit: 4666)
     Memory: 306.7M
        CPU: 6.974s
     CGroup: /system.slice/kafka.service
             └─4830 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+Explicit>

Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,176] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReap>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,196] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReap>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,200] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,223] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,244] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,245] INFO Feature ZK node created at path: /feature (kafka.server.FinalizedFeatureChangeListener)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,300] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,309] INFO [MetadataCache brokerId=0] Updated cache from existing  to latest FinalizedFeaturesAndEpoch(fe>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,327] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,334] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChan>

Step 5 – Create a Topic in Kafka

At this point, Kafka is installed on your server. To test Kafka, create a topic named FedoraTopic.

cd /usr/local/kafka
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic FedoraTopic

Next, verify your created topic using the following command.

./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --list

Output:

FedoraTopic

Now, execute the producer and type a few messages into the console to send to the server.

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic FedoraTopic
>Hi
>How are you

Next, open another terminal and run the consumer to read data from the Kafka cluster and display messages to standard output.

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FedoraTopic --from-beginning
Hi
How are you

Conclusion

In this post, we explained how to install Apache Kafka on Fedora. We also create a sample topic and show you how producer and consume work. You can now install Kafka on dedicated server hosting from Atlantic.Net!

Get a $250 Credit and Access to Our Free Tier!

Free Tier includes:
G3.2GB Cloud VPS a Free to Use for One Year
50 GB of Block Storage Free to Use for One Year
50 GB of Snapshots Free to Use for One Year