Manual upload of big files to HDFS

Your browser might prevent you from manually uploading files over a few gigabytes to HDFS via HUE.

 A workaround is:

• Split file as a zip archive on your computer (use 7-Zip on Windows or Split tool on Linux) into multiples parts less than 1 gigabyte (resulting in files myfile.zip.001, myfile.zip.002, myfile.zip.003...)
• Upload files to HDFS throught HUE
• Create and run a SQOOP job as follows (replace the values of mypath & myzip variables accordingly):

mypath="/hdfspath/to/data/"
myzip="name of myfile" # without the .zip extension
 
hadoop fs -chmod 777 "$mypath"
 
hadoop fs -ls "$mypath$myzip.zip".*
 
hadoop fs -cat "$mypath$myzip.zip".* > file.zip
ls -la
unzip file.zip -d "$myzip"
ls -la "$myzip/"
 
hadoop fs -put -f $( echo "$myzip/" | sed s/\ /\%20/g ) "$mypath"