List files from HDFS with Kerberos

Github Project : kerberos-talend-client 

To use a job displayed in this article, you have to complete a repository with saagie's platform information (value).

values example :

• kerberos_login=$username
• kerberos_pwd=$userpassword
• kerberos_principal_name="hdfs/_HOST@<YOUR REALM>"
  • example : "impala/[e-mail]"
• uri_path=<full hostname>
  • example : "hdfs://192.168.55.10:8020"

Read a file from HDFS (In console)

• Create a new job
• Add the component tSystem: Allows the creation of a Kerberos connection.
• Add the component tHDFSConnection --> Allows the creation of a HDFS connection.
• Add the component tHDFSList --> List the different files contents in the hdfs folder.
• Add the component tRowGenerator--> Assign the filenames to rows
• Add the component tLogRow --> Display the result.
• Create links:
  • tSystem is connected with tHDFSConnection (through "OnSubjobOk")
  • tHDFSConnection is connected with tHDFSList (through "OnSubjobOk")
  • tHDFSList is connected with tHDFSProperties (through "Iterate") 
  • tRowGenerator is connected with tLogRun (through "Main")

Component tSystem

 Component tHDFSConnection

 Component tHDFSList

 Component tRowGenerator

• Run the job