PySpark - Read & Write files from Hive

GitHub Page : exemple-pyspark-read-and-write

Common part

Libraries dependency

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession, HiveContext

Set Hive metastore uri

sparkSession = (SparkSession
 .builder
 .appName('example-pyspark-read-and-write-from-hive')
 .enableHiveSupport()
 .getOrCreate())
data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)]
df = sparkSession.createDataFrame(data)

Creating Spark Session

sparkSession = SparkSession.builder.appName("example-pyspark-read-and-write").getOrCreate()

How to write a table into Hive?

Code example

# Write into Hive
df.write.saveAsTable('example')

How to read a table from Hive?

Code example

This Code only shows the first 20 records of the file.

# Read from Hive
df_load = sparkSession.sql('SELECT * FROM example')
df_load.show()

Spark 3.1 with Hive 1.1.0

Starting from Spark 3.1, you must update your command line if you want to connect to a Hive Metastore V1.1.0.
To help you do so, Hive jars for version 1.1.0 have been already downloaded inside the Saagie Spark docker images under the /opt/spark/hive_1.1.0_jars directory.
To load these jars in your classpath, simply add the following parameters in your Spark job command line :

spark-submit \
...
--conf spark.sql.hive.metastore.jars=path \
--conf spark.sql.hive.metastore.jars.path=file:///opt/spark/hive_1.1.0_jars/*.jar \
--conf spark.sql.hive.metastore.version=1.1.0 \
--conf spark.sql.catalogImplementation=hive
... 

How to use on Saagie?

Please refer to the Python application packaging guidelines

How to use on Saagie's Jupyter Notebooks?

Prior to spark session creation, you must add the following snippet:

import os
os.environ["HADOOP_USER_NAME"] = "hdfs"
os.environ["PYTHON_VERSION"] = "3.5.2"