Category: database

Spark 2.1.0 on CentOS 6.8 Installation ….procedure

Posted on by lechuck park

This page is for installation Setup Spark on CentOS 6.8 . the final goal is all removing WARN/ERROR after starting pyspark on my machie. so. you cant get enough information. but I hope that it will help you.

below is my WARN message when I started pyspark ( in spark-2.1.0-bin-hadoop2.7.tgz ).

17/03/02 06:50:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
17/03/02 06:50:58 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 211.58.253.190 instead (on interface eth2)
17/03/02 06:50:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
/home/lechuck/SPARK/spark-2.1.0-bin-hadoop2.7/python/pyspark/context.py:194: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn(“Support for Python 2.6 is deprecated as of Spark 2.0.0”)
17/03/02 06:51:02 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

# Java update.

17/03/02 06:42:29 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0

# java -version
java version “1.7.0_131”

# yum list | grep jdk

…………….

# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

………………

# java -version
openjdk version “1.8.0_121”
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

# Set Envinroment SET_LOCAL_IP

17/03/02 06:50:58 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 211.58.253.190 instead (on interface eth2)

17/03/02 06:50:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

# mv spark-env.sh.template spark-env.sh ( on spark conf directory)

# vi spark-env.sh

# – SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
export SPARK_LOCAL_IP=127.0.0.1              <<< add this

# ./spark-env.sh

# For Python 2.7  ( Skipped installation Python 2.7. It will be easy with configure/make/make install with tarball )

/home/lechuck/SPARK/spark-2.1.0-bin-hadoop2.7/python/pyspark/context.py:194: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn(“Support for Python 2.6 is deprecated as of Spark 2.0.0”)

# python -V
Python 2.6.6

# vi ~/.bashrc

alias python=/usr/local/bin/python2.7

# . ~/.bashrc                     << no need to do it from next login 

# Hadoop ( NOT YET )

17/03/02 07:24:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

# global temporary view ( NOT YET )

17/03/02 06:51:02 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException