Category: database

Spark 2.1.0 on CentOS 6.8 Installation ….procedure

Posted on by lechuck park

This page is for installation Setup Spark on CentOS 6.8 . the final goal is all removing WARN/ERROR after starting pyspark on my machie. so. you cant get enough information. but I hope that it will help you.

below is my WARN message when I started pyspark ( in spark-2.1.0-bin-hadoop2.7.tgz ).

17/03/02 06:50:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
17/03/02 06:50:58 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 211.58.253.190 instead (on interface eth2)
17/03/02 06:50:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
/home/lechuck/SPARK/spark-2.1.0-bin-hadoop2.7/python/pyspark/context.py:194: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn(“Support for Python 2.6 is deprecated as of Spark 2.0.0”)
17/03/02 06:51:02 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

# Java update.

17/03/02 06:42:29 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0

# java -version
java version “1.7.0_131”

# yum list | grep jdk

…………….

# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

………………

# java -version
openjdk version “1.8.0_121”
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

# Set Envinroment SET_LOCAL_IP

17/03/02 06:50:58 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 211.58.253.190 instead (on interface eth2)

17/03/02 06:50:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

# mv spark-env.sh.template spark-env.sh ( on spark conf directory)

# vi spark-env.sh

# – SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
export SPARK_LOCAL_IP=127.0.0.1              <<< add this

# ./spark-env.sh

# For Python 2.7  ( Skipped installation Python 2.7. It will be easy with configure/make/make install with tarball )

/home/lechuck/SPARK/spark-2.1.0-bin-hadoop2.7/python/pyspark/context.py:194: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn(“Support for Python 2.6 is deprecated as of Spark 2.0.0”)

# python -V
Python 2.6.6

# vi ~/.bashrc

alias python=/usr/local/bin/python2.7

# . ~/.bashrc                     << no need to do it from next login 

# Hadoop ( NOT YET )

17/03/02 07:24:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

# global temporary view ( NOT YET )

17/03/02 06:51:02 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

How about remove old data from mysql.

Posted on by lechuck park
  1. check Event ON state.

    show variables where Variable_name like ‘%event%’;

  2. if not set ON

    SET GLOBAL event_scheduler = ON;

  3. Example for Event Job – delete data which was inserted over a week ago.

    delimiter |
    CREATE EVENT IF NOT EXISTS <#event_name>
    ON SCHEDULE
    EVERY 1 DAY
    DO
    BEGIN
    delete from <#table_name> where stime < DATE_SUB(NOW(), INTERVAL 7 Day );
    END |
    delimiter ;

  4. check Event

    select EVENT_DEFINITION from information_schema.events where event_name='<#event_name>’;

  5. check Event state

    show events from <#database name>

  6. How to backup to csv. ( how about adding this event before delete.. with where phase) – this example is not concerned with  event example on this page.

  7. SELECT order_id,product_name,qty
    FROM orders
    INTO OUTFILE ‘/tmp/orders.csv’
    FIELDS TERMINATED BY ‘,’
    ENCLOSED BY ‘”‘
    LINES TERMINATED BY ‘\n’;