Gist Page: example-python-write-figure-to-hdfs
Common part
Libraries dependency
from hdfs import InsecureClient
import os
import io
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
WEBHDFS URI
A WEBHDFS URI follows this format:
http://namenodedns:port/user/hdfs/folder/file.csv
The default port is 50070
Connection
# Connecting to Webhdfs by providing hdfs host ip and webhdfs port (50070 by default)
client_hdfs = InsecureClient('http://' + os.environ['IP_HDFS'] + ':50070')
How to create a Matplotlib figure with Python?
# ====== Figure Creation ======
# Creating Matplotlib figure but not printing it
plt.ioff()
plt.figure(figsize=(15,9))
plt.scatter(range(10),range(10),c=range(10),marker='o',s=500)
plt.title('Example of figure')
plt.tight_layout()
How to write a Matplotlib figure to HDFS with Python?
# ====== Writing to hdfs ======
# Writing figure to hdfs through a BytesIO python object
buf = io.BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
with client_hdfs.write('/user/hdfs/data/figure.png',overwrite=True) as writer :
writer.write(buf.getvalue())
buf.close()
Comments
0 comments
Article is closed for comments.