Query & Insert from Hive with Kerberos

Github Project : kerberos-talend-client

To use the jobs displayed in this article, you have to complete a repository with saagie's platform information (value).

values example :

• kerberos_login=$username
• kerberos_pwd=$userpassword
• kerberos_principal_name="hdfs/_HOST@<YOUR REALM>"
  • example : "hdfs/[e-mail]"
• kerberos_principal_hive="hive/_HOST@<YOUR REALM>"
  • example : "hive/[e-mail]"
• connexion_hote=<full hostname>
  • example : "dn1.p6.saagie.prod.saagie.io"
• connexion_port=10000
• uri_path=<full hostname>
  • example : "hdfs://192.168.55.10:8020"

Write from Hive

• Create a new job
• Add the component tSystem: Allows the creation of a Kerberos connection.
• Add the component tHiveConnection : Allows the creation of a Hive connection.
• Add the component tHDFSConnection : Allows the creation of a HDFS connection.
• Add the component tHiveCreateTable: Allows the creation of a table
• Add the component tRowGenerator: Allows the generation of data
• Add the component tHDFSOutput: Writes data to HDFS.
• Add the component tHiveLoad: Allows the data loading in Hive
• Create links:
  • tSystem is connected with tHiveConnection (through "OnSubjobOk")
  • tHiveConnection is connected with tHDFSConnection (through "OnComponentOk")
  • tHDFSConnection is connected with tHiveCreateTable (through "OnComponentOk")
  • tHiveCreateTable is connected with tRowGenerator (through "OnComponentOk")
  • tRowGenerator is connected with tHDFSOutput (through "Main")
  • tRowGenerator is connected with tHiveLoad (through "OnSubjobOk")

Component tSystem

Component tHiveConnection

 Component tHDFSConnection

 Component tHiveCreateTable

 Component tRowGenerator

Component tHDFSOutput

Component tHiveLoad

• Run the job

Read from Hive (In console)

• Create a new job
• Add the component tSystem: Allows the creation of a Kerberos connection.
• Add the component tHiveConnection : Allows the creation of a Hive connection.
• Add the component tHiveInput: Read a file in the Hive .
• Add the component tLogRow: Display the result.
• Create links:
  • tSystem is connected with tHDFSConnection (through "OnSubjobOk")
  • tHDFSConnection is connected with tHiveInput (through "OnComponentOk")
  • tHiveInput is connected with tLogRun (through "Main")

Component tSystem

 Component tHiveConnection

 Component tHiveInput

• Run the job