How to Install Apache Spark on CentOS 8

Apache Spark is a data processing framework that performs processing tasks over large data sets quickly. Apache Spark can also distribute data processing tasks across several computers. It does so either on its own or in tandem with other distributed computing tools. These two qualities make it particularly useful in the world of machine learning and big data. Spark also features an easy-to-use API, reducing the programming burden associated with data crunching. It undertakes most of the work associated with big data processing and distributed computing.

Step 1 – Install Java

Spark is a Java-based application, so you will need to install Java in your system. You can install the Java using the following command:

dnf install java-11-openjdk-devel -y

Once Java is installed, you can verify the Java version with the following command:

java --version

You should get the Java version in the following output:

openjdk 11.0.8 2020-07-14 LTSOpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)

Step 2 – Install Spark

First, you will need to download the latest version of Spark from its official website. You can download it with the following command:

wget https://mirrors.estointernet.in/apache/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz

Once the download is completed, extract the downloaded file with the following command:

tar -xvf spark-3.0.1-bin-hadoop2.7.tgz

Next, move the extracted directory to /opt with the following command:

mv spark-3.0.1-bin-hadoop2.7 /opt/spark

Next, create a separate user to run Spark with the following command:

useradd spark

Next, change the ownership of the /opt/spark directory to the spark user with the following command:

chown -R spark:spark /opt/spark

Step 3 – Create a Systemd Service File for Spark

Next, you will need to create a systemd service file for the Spark master and slave.

First, create a master service file with the following command:

nano /etc/systemd/system/spark-master.service

Add the following lines:

[Unit]
Description=Apache Spark Master
After=network.target

[Service]
Type=forking
User=spark
Group=spark
ExecStart=/opt/spark/sbin/start-master.sh
ExecStop=/opt/spark/sbin/stop-master.sh

[Install]
WantedBy=multi-user.target

Save and close the file when you are finished, then create a Spark slave service with the following command:

nano /etc/systemd/system/spark-slave.service.service

Add the following lines:

[Unit]

Description=Apache Spark Slave

After=network.target

[Service]
Type=forking
User=spark
Group=spark
ExecStart=/opt/spark/sbin/start-slave.sh spark://your-server-ip:7077
ExecStop=/opt/spark/sbin/stop-slave.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon with the following command:

systemctl daemon-reload

Step 4 – Start the Master Service

Now, you can start the Spark master service and enable it to start at boot with the following command:

systemctl start spark-master
systemctl enable spark-master

You can verify the status of the Master service with the following command:

systemctl status spark-master

You should get the following output:

• spark-master.service - Apache Spark Master

Loaded: loaded (/etc/systemd/system/spark-master.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-02 10:40:40 EDT; 18s ago
Process: 2554 ExecStart=/opt/spark/sbin/start-master.sh (code=exited, status=0/SUCCESS)
Main PID: 2572 (java)
Tasks: 32 (limit: 12523)
Memory: 174.5M
CGroup: /system.slice/spark-master.service
└─2572 /usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el8_2.x86_64/bin/java -cp 
/opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spar>

Oct 02 10:40:37 centos8 systemd[1]: Starting Apache Spark Master...
Oct 02 10:40:37 centos8 start-master.sh[2554]: starting org.apache.spark.deploy.master.Master, 
logging to /opt/spark/logs/spark-spark-org.apac>
Oct 02 10:40:40 centos8 systemd[1]: Started Apache Spark Master.

You can also check the Spark log file to check the Master server.

tail -f /opt/spark/logs/spark-spark-org.apache.spark.deploy.master.Master-1-centos8.out

You should get the following output:

20/10/02 10:40:40 INFO Utils: Successfully started service 'MasterUI' on port 8080.
20/10/02 10:40:40 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at 
http://centos8:8080
20/10/02 10:40:40 INFO Master: I have been elected leader! New state: ALIVE

At this point, the Spark master server is started and listening on port 8080.

Step 5 – Access Spark Dashboard

Now, open your web browser and access the Spark dashboard using the URL http://your-server-ip:8080. You should see the Spark dashboard in the following page:

In the above page, there are no workers attached to the master.

Step 6 – Start Slave Service

Now, start the Slave service and enable it to start at boot with the following command:

systemctl start spark-slave
systemctl enable spark-slave

Next, check the status of the Slave with the following command:

systemctl status spark-slave

You should get the following output:

• spark-slave.service - Apache Spark Slave

Loaded: loaded (/etc/systemd/system/spark-slave.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-02 10:45:12 EDT; 10s ago
Process: 2671 ExecStart=/opt/spark/sbin/start-slave.sh spark://45.58.32.165:7077 (code=exited, 
status=0/SUCCESS)
Main PID: 2680 (java)
Tasks: 35 (limit: 12523)
Memory: 197.9M
CGroup: /system.slice/spark-slave.service
└─2680 /usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el8_2.x86_64/bin/java -cp 
/opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spar>

Oct 02 10:45:09 centos8 systemd[1]: Starting Apache Spark Slave...
Oct 02 10:45:09 centos8 start-slave.sh[2671]: starting org.apache.spark.deploy.worker.Worker, 
logging to /opt/spark/logs/spark-spark-org.apach>
Oct 02 10:45:12 centos8 systemd[1]: Started Apache Spark Slave.

You can also check the Spark slave log file for confirmation:

tail -f /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-centos8.out

You should get the following output:

20/10/02 10:45:12 INFO Worker: Spark home: /opt/spark
20/10/02 10:45:12 INFO ResourceUtils: ==============================================================
20/10/02 10:45:12 INFO ResourceUtils: Resources for spark.worker:

20/10/02 10:45:12 INFO ResourceUtils: ==============================================================
20/10/02 10:45:12 INFO Utils: Successfully started service ‘WorkerUI’ on port 8081.
20/10/02 10:45:12 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://centos8:8081
20/10/02 10:45:12 INFO Worker: Connecting to master 45.58.32.165:7077…
20/10/02 10:45:12 INFO TransportClientFactory: Successfully created connection to /45.58.32.165:7077 after 66 ms (0 ms spent in bootstraps)
20/10/02 10:45:13 INFO Worker: Successfully registered with master spark://centos8:7077

Now, go to the Spark dashboard and reload the page. You should see the worker in the following page.

You can also access the worker directly using the URL http://your-server-ip:8081.

Conclusion

In this guide, you learned how to set up a single-node Spark cluster on CentOS 8. You can now configure Spark multinode cluster easily and use it for big data and machine learning processing. Set up Apache Spark on your VPS hosting account with Atlantic.Net!