Atlantic.Net Blog

How to install Apache Kafka on Oracle Linux 8

Hitesh Jethva
by Atlantic.Net (414 posts) under Dedicated Server Hosting, Tutorials
0 Comments

Apache Kafka is a free, open-source, distributed event streaming platform that is used by thousands of companies for high-performance data pipelines. It is a message broker software application primarily used to build real-time streaming data pipelines and applications. Apache Kafka allows you to process data streams via a distributed streaming platform. It is based on a distributed architecture, so it provides high fault tolerance and scalability capabilities.

In this post, we will explain how to install Apache Kafka on Oracle Linux 8.

Prerequisites

  • A server running Oracle Linux 8 on the Atlantic.Net Cloud Platform
  • A root password configured on your server

Step 1 – Create Atlantic.Net Cloud Server

First, log in to your Atlantic.Net Cloud Server. Create a new server, choosing Oracle Linux 8 as the operating system with at least 2GB RAM. Connect to your Cloud Server via SSH and log in using the credentials highlighted at the top of the page.

Once you are logged in to your server, run the following command to update your base system with the latest available packages.

dnf update -y

Step 2 – Install Java

Apache Kafka is based on Java, so you will need to install Java on your server. You can install it by running the following command:

dnf install java-11-openjdk-devel -y

Once Java is installed, verify the Java installation using the following command:

java --version

You will get the Java version in the following output:

openjdk 11.0.15 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10-LTS, mixed mode, sharing)

Step 3 – Install Apache Kafka on Oracle Linux 8

First, go to the Apache official website and download the latest version of Apache Kafka using the wget command:

wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz

Once the download is completed, extract the downloaded file using the following command:

tar -xvzf kafka_2.13-3.2.0.tgz

Next, move the extracted directory to /usr/local directory:

mv kafka_2.13-3.2.0/ /usr/local/kafka

Once you are finished, you can proceed to the next step.

Step 4 – Create Systemd Service File for Zookeeper and Kafka

It is a good idea to create a systemd service file to run both Zookeeper and Kafka in the background.

First, create a systemd service file for Zookeeper with the following command:

nano /etc/systemd/system/zookeeper.service

Add the following lines:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file, then create a systemd service file for Kafka using the following command:

nano /etc/systemd/system/kafka.service

Add the following lines:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon with the following command:

systemctl daemon-reload

Next, start both the Zookeeper and Kafka services and enable them to start at system reboot:

systemctl start zookeeper
systemctl start kafka
systemctl enable zookeeper
systemctl enable kafka

To check the Zookeeper service, run the following command:

systemctl status zookeeper

You will get the following output:

● zookeeper.service - Apache Zookeeper server
   Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-11 01:06:50 EDT; 28s ago
     Docs: http://zookeeper.apache.org
 Main PID: 29634 (java)
    Tasks: 26 (limit: 11409)
   Memory: 67.9M
   CGroup: /system.slice/zookeeper.service
           └─29634 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCI>

Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,302] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,311] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeepe>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,312] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshot loaded in 18 ms, highest zxid is 0x0, digest is 1371985504 >
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,324] INFO Snapshot taken in 2 ms (org.apache.zookeeper.server.ZooKeeperServer)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,378] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zook>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,388] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apa>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,446] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalM>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,447] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProv>
lines 1-20/20 (END)

To check the Kafka service, run the following command:

systemctl status kafka

You should see the following output:

● kafka.service - Apache Kafka Server
   Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-06-11 01:07:17 EDT; 55s ago
     Docs: http://kafka.apache.org/documentation.html
 Main PID: 29994 (java)
    Tasks: 69 (limit: 11409)
   Memory: 327.3M
   CGroup: /system.slice/kafka.service
           └─29994 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupanc>

Jun 11 01:07:23 oraclelinux8 bash[29994]: [2022-06-11 01:07:23,975] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNod>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,001] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Starting socket serv>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane a>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started socket serve>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka version: 3.2.0 (org.apache.kafka.common.utils.AppInfoParser)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka commitId: 38103ffaa962ef50 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka startTimeMs: 1654924044055 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,158] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,377] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorde>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,378] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Rec>

Step 5 – Create Your First Topic on Kafka

Next, you will need to create at least one topic on the server to test Apache Kafka.

Change the directory to Apache Kafka and create a test topic named app1 with the following command:

cd /usr/local/kafka/
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic app1

You can now verify your created topic using the following command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

You will get the following output:

app1

Kafka provides two APIs: producer and consumer. The producer is responsible for creating events and the consumer uses them to display it on the screen:

Now, run the producer and then type a few messages into the console to send to the server.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic app1

Type some text that you want to stream and display on the consumer.

>Hi, this is my first kafka server

Open another terminal and run the following command to display the generated event data in real-time:

cd /usr/local/kafka/
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic app1 --from-beginning

You will get the following output:

Hi, this is my first kafka server

Conclusion

In the above guide, we explained how to install Apache Kafka on Oracle Linux 8. You can now integrate Apache Kafka with your application to collect and analyze a large amount of data. For more information, you can visit the Apache Kafka documentation page. Try it on dedicated hosting from Atlantic.Net!

Get A Free To Use Cloud VPS

Free Tier Includes:
G3.2GB Cloud VPS Free to Use for One Year
50 GB of Block Storage Free to Use for One Year
50 GB of Snapshots Free to Use for One Year


Looking for a Hosting Solution?

We Provide Cloud, Dedicated, & Colocation.

  • Seven Global Data Center Locations.
  • Flexible Private, Public, & Hybrid Hosting.
  • 24x7x365 Security, Support, & Monitoring.
Contact Us Now! Med Tech Award FTC
SOC Audit HIPAA Audit HITECH Audit

Recent Posts

Top 10 Best Cybersecurity Training Services
How to Install Ansible on Oracle Linux 8
How to Install LEMP Server on Oracle Linux 8
How to Install and Configure Apache Webserver on Oracle Linux 8
How to Install and Configure Nginx Webserver on Oracle Linux 8

Get started with 12 months of free cloud VPS hosting

Free Tier includes:
G3.2GB Cloud VPS Server Free to Use for One Year
50 GB of Block Storage Free to Use for One Year
50 GB of Snapshots Free to Use for One Year


New York, NY

100 Delawanna Ave, Suite 1

Clifton, NJ 07014

United States

San Francisco, CA

2820 Northwestern Pkwy,

Santa Clara, CA 95051

United States

Dallas, TX

2008 Lookout Dr,

Dallas, Texas 75044

United States

Ashburn, VA

1807 Michael Faraday Ct,

Reston, VA 20190

United States

Orlando, FL

440 W Kennedy Blvd, Suite 3

Orlando, FL 32810

United States

Toronto, Canada

20 Pullman Ct, Scarborough,

Ontario M1X 1E4

Canada

London, UK

14 Liverpool Road, Slough,

Berkshire SL1 4QZ

United Kingdom

Resources