Github Project : example-talend-download-file-opendata-and-write-file-with-HDFS
Preamble
Configuration: Group of Context
To create the different jobs displayed in this article, you have to create a repository with saagie's platform information (value).
Api Rest request and write file with HDFS
- Create a new job
- Add the component HDFSConnection : Allows the creation of a HDFS connection.
- Add the component tREST : Send a message for RESTful webservice and retrieve the response.
- Add the component tHDFSOutput : Writes data to HDFS.
- Create links:
- HDFSConnection is connected with tREST (through "OnSubjobOk")
- tREST is connected with tHDFSOutput (through "Main")
- Double click on tHDFSConnection and set its properties:
- Add a "Cloudera" distribution and select the latest version of Cloudera
- Enter the NameNode URL.
The URL has to respect this format :hdfs://ip_hdfs:port_hdfs/
Use context variables if possible : "hdfs://"+context.IP_HDFS+":"+context.Port_HDFS+"/" - Add the user
- Untick Use Datanode Hostname
- Double click on tREST and enter URL
- Double click on tHDFSOutput :
- Tick Use an existing connection
- Enter a path file
- Run the job
Comments
0 comments
Article is closed for comments.