Installing Apache Spark on Mac OS

Installing Apache Spark on Mac OS

Pre-requisites

Install Brew

Go to Brew Website https://brew.sh/

Image for post

Copy the url from Home page on mac os terminal to install Home brew

Image for post

Run below command to update home brew

brew upgrade && brew update

Java Installation

Check Java Version

Java -versionImage for post

Run below command to install Java8

brew cask install java8Image for post

For Latest Java use

brew cask install java

Check Java Version

Image for post

Install xcode-select

xcode-select –install

Install Scala

brew install scalaImage for post

Use scala -version to get the scala version

Image for post

Install Apache Spark

brew install apache-sparkImage for post

To start spark shell execute below command

Spark-shellImage for post

Run below command to check the execution which will return the string ?Hello World?

val s = “hello world”Image for post

Run pyspark to start pyspark shell

Image for post

Add Spark path to bash profile

Run below command and then add the path to the profile

nano ~/.profileexport SPARK_HOME=/usr/local/Cellar/apache-spark/2.4.4/libexecexport PYTHONPATH=/usr/local/Cellar/apache-spark/2.4.4/libexec/python/:$PYTHONP$source ~/.bash_profileImage for postcd /usr/local/Cellar/apache-spark/2.4.4/libexec/sbin

And execute below command to start all services

sbin/start-all.shImage for post

Spark Master UI : http://localhost:8080/

Image for post

Spark Application UI : http://localhost:4040/

Image for post

13

No Responses

Write a response