Github Project : kerberos-talend-client
To use the jobs displayed in this article, you have to complete a repository with saagie's platform information (value).
values example :
- kerberos_login=$username
- kerberos_pwd=$userpassword
- kerberos_principal_name="hdfs/_HOST@<YOUR REALM>"
- example : "hdfs/_HOST@SAAGIE.PROD.SAAGIE.IO"
- kerberos_principal_hive="hive/_HOST@<YOUR REALM>"
- example : "hive/_HOST@SAAGIE.PROD.SAAGIE.IO"
- connexion_hote=<full hostname>
- example : "dn1.p6.saagie.prod.saagie.io"
- connexion_port=10000
- uri_path=<full hostname>
- example : "hdfs://192.168.55.10:8020"
Write from Hive
- Create a new job
- Add the component tSystem: Allows the creation of a Kerberos connection.
- Add the component tHiveConnection : Allows the creation of a Hive connection.
- Add the component tHDFSConnection : Allows the creation of a HDFS connection.
- Add the component tHiveCreateTable: Allows the creation of a table
- Add the component tRowGenerator: Allows the generation of data
- Add the component tHDFSOutput: Writes data to HDFS.
- Add the component tHiveLoad: Allows the data loading in Hive
- Create links:
- tSystem is connected with tHiveConnection (through "OnSubjobOk")
- tHiveConnection is connected with tHDFSConnection (through "OnComponentOk")
- tHDFSConnection is connected with tHiveCreateTable (through "OnComponentOk")
- tHiveCreateTable is connected with tRowGenerator (through "OnComponentOk")
- tRowGenerator is connected with tHDFSOutput (through "Main")
- tRowGenerator is connected with tHiveLoad (through "OnSubjobOk")
Component tSystem
Component tHiveConnection
Component tHDFSConnection
Component tHiveCreateTable
Component tRowGenerator
Component tHDFSOutput
Component tHiveLoad
- Run the job
Read from Hive (In console)
- Create a new job
- Add the component tSystem: Allows the creation of a Kerberos connection.
- Add the component tHiveConnection : Allows the creation of a Hive connection.
- Add the component tHiveInput: Read a file in the Hive .
- Add the component tLogRow: Display the result.
- Create links:
- tSystem is connected with tHDFSConnection (through "OnSubjobOk")
- tHDFSConnection is connected with tHiveInput (through "OnComponentOk")
- tHiveInput is connected with tLogRun (through "Main")
Component tSystem
Component tHiveConnection
Component tHiveInput
- Run the job
Comments
0 comments
Please sign in to leave a comment.