How to Install Apache Spark on Arch Linux

Apache Spark is a data processing framework used for executing data engineering, data science, and machine learning on single-node machines or clusters. It is designed for fast computation and used in big data workloads to quickly perform processing tasks. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis.

In this tutorial, we will explain how to install Apache Spark on Arch Linux.

Step 1 – Configure Repository

By default, the default repository is outdated in Arch Linux, so you will need to modify the default mirror list. You can do it by editing the mirror list configuration file:

nano  /etc/pacman.d/mirrorlist

Remove all lines and add the following lines:

## Score: 0.7, United States
Server = http://mirror.us.leaseweb.net/archlinux/$repo/os/$arch
## Score: 0.8, United States
Server = http://lug.mtu.edu/archlinux/$repo/os/$arch
Server = http://mirror.nl.leaseweb.net/archlinux/$repo/os/$arch
## Score: 0.9, United Kingdom
Server = http://mirror.bytemark.co.uk/archlinux/$repo/os/$arch
## Score: 1.5, United Kingdom
Server = http://mirrors.manchester.m247.com/arch-linux/$repo/os/$arch
Server = http://archlinux.dcc.fc.up.pt/$repo/os/$arch
## Score: 6.6, United States
Server = http://mirror.cs.pitt.edu/archlinux/$repo/os/$arch
## Score: 6.7, United States
Server = http://mirrors.acm.wpi.edu/archlinux/$repo/os/$arch
## Score: 6.8, United States
Server = http://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
## Score: 7.1, India
Server = http://mirror.cse.iitk.ac.in/archlinux/$repo/os/$arch
## Score: 10.1, United States
Server = http://mirrors.xmission.com/archlinux/$repo/os/$arch

Save and close the file then update all the package indexes with the following command:

pacman -Syu

Step 2 – Install Java

Apache Spark is based on Java, so you will need to install Java JDK on your server. Run the following command to install Java:

pacman -S jre17-openjdk

Once installed, you can verify the Java installation using the following command.

java --version

You will get the following output.

openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment (build 17.0.6+10)
OpenJDK 64-Bit Server VM (build 17.0.6+10, mixed mode)

Step 3 – Install Scala

You will also need to install Scala on your server. You can install it using the following command.

pacman -S scala

After the installation, verify the Scala version:

scala --version

Sample output.

Scala code runner version 2.13.8-20220325-005602-unknown -- Copyright 2002-2021, LAMP/EPFL and Lightbend, Inc.

To connect to the Scala console, run the following command.

scala

You will get the following console.

Welcome to Scala 2.13.8-20220325-005602-unknown (OpenJDK 64-Bit Server VM, Java 17.0.6).
Type in expressions for evaluation. Or try :help.

Verify Scala using the following command.

scala> println("Testing")

Sample output.

Testing

Step 4 – Install Apache Spark

First, visit the Apache Spark download page, pick the download URL, and download it with the following command.

wget https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz

Once the download is completed, extract the downloaded file using the following command.

tar -xzf spark-3.3.1-bin-hadoop3.tgz

Next, move the extracted directory to /mnt.

mv spark-3.3.1-bin-hadoop3 /mnt/spark

Next, create a ~/.bashrc file and define your Apache Spark path.

nano ~/.bashrc

Add the following lines:

export SPARK_HOME=/mnt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Next, activate the environment variable using the following command.

source ~/.bashrc

Step 5 – Start Spark Master and Worker Node

At this point, Apache Spark is installed on your server. You can now start the Spark master with the following command.

start-master.sh

Sample output.

starting org.apache.spark.deploy.master.Master, logging to /mnt/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-archlinux.out

By default, Spark master listens on port 8080. You can check it with the following command.

ss -tpln | grep 8080

Sample output.

LISTEN 0      0                           *:8080             *:*    users:(("java",pid=57901,fd=265))

Now, open your web browser and access the Spark master using the URL http://your-server-ip:8080. You should see the following screen.

Next, start the Spark Worker using the following command.

start-worker.sh spark://your-server-ip:7077

Now, go back to the Spark master web interface and reload the page. You should see the added worker on the following screen.

Step 6 – Create a Systemd Service File for Apache Spark

Before creating systemd service file, stop both the worker and master service using the following command.

stop-worker.sh
stop-master.sh

Next, create a Spark master service using the following command.

nano /etc/systemd/system/spark-master.service

Add the following configuration.

[Unit]
Description=Apache Spark Master
After=network.target

[Service]
Type=forking
User=root
Group=root
ExecStart=/mnt/spark/sbin/start-master.sh
ExecStop=/mnt/spark/sbin/stop-master.sh

[Install]
WantedBy=multi-user.target

Next, create a Spark worker service file.

nano /etc/systemd/system/spark-worker.service

Add the following configuration.

[Unit]

Description=Apache Spark Worker

After=network.target

[Service]
Type=forking
User=root
Group=root
ExecStart=/mnt/spark/sbin/start-slave.sh spark://your-server-ip:7077
ExecStop=/mnt/spark/sbin/stop-slave.sh

[Install]
WantedBy=multi-user.target

Save and close the file, then reload the systemd daemon to apply the changes.

systemctl daemon-reload

Next, start both the Master and Worker services using the following command.

systemctl start spark-master spark-worker

You can check the status of both services using the following command.

systemctl status spark-master spark-worker

You should see the following output.

● spark-master.service - Apache Spark Master
     Loaded: loaded (/etc/systemd/system/spark-master.service; disabled; preset: disabled)
     Active: active (running) since Mon 2023-02-06 05:28:09 UTC; 19s ago
    Process: 58118 ExecStart=/mnt/spark/sbin/start-master.sh (code=exited, status=0/SUCCESS)
   Main PID: 58151 (java)
      Tasks: 34 (limit: 4700)
     Memory: 159.2M
     CGroup: /system.slice/spark-master.service
             └─58151 /usr/lib/jvm/java-17-openjdk/bin/java -cp "/mnt/spark/conf/:/mnt/spark/jars/*" -Xmx1g org.apache.spark.deploy.master.Mas>

Feb 06 05:28:06 archlinux systemd[1]: Starting Apache Spark Master...
Feb 06 05:28:06 archlinux start-master.sh[58131]: starting org.apache.spark.deploy.master.Master, logging to /mnt/spark/logs/spark-root-org.a>
Feb 06 05:28:09 archlinux systemd[1]: Started Apache Spark Master.

● spark-worker.service - Apache Spark Worker
     Loaded: loaded (/etc/systemd/system/spark-worker.service; disabled; preset: disabled)
     Active: active (running) since Mon 2023-02-06 05:28:09 UTC; 19s ago
    Process: 58119 ExecStart=/mnt/spark/sbin/start-slave.sh spark://your-server-ip:7077 (code=exited, status=0/SUCCESS)
   Main PID: 58157 (java)
      Tasks: 40 (limit: 4700)
     Memory: 170.5M
     CGroup: /system.slice/spark-worker.service
             └─58157 /usr/lib/jvm/java-17-openjdk/bin/java -cp "/mnt/spark/conf/:/mnt/spark/jars/*" -Xmx1g org.apache.spark.deploy.worker.Wor>

Feb 06 05:28:06 archlinux systemd[1]: Starting Apache Spark Worker...
Feb 06 05:28:06 archlinux start-slave.sh[58119]: This script is deprecated, use start-worker.sh
Feb 06 05:28:06 archlinux start-slave.sh[58133]: starting org.apache.spark.deploy.worker.Worker, logging to /mnt/spark/logs/spark-root-org.ap>
Feb 06 05:28:09 archlinux systemd[1]: Started Apache Spark Worker.

Conclusion

In this post, you learned how to install Apache Spark on Arch Linux. You can now use Apache Spark in data science, and machine learning project to handle high workloads. You can try to install the Apache Spark server on dedicated server hosting from Atlantic.Net!