API Rest request and write file with HDFS

Github Project : example-talend-download-file-opendata-and-write-file-with-HDFS

Preamble

Configuration: Group of Context

To create the different jobs displayed in this article, you have to create a repository with saagie's platform information (value).

Api Rest request and write file with HDFS 

• Create a new job
• Add the component HDFSConnection : Allows the creation of a HDFS connection.
• Add the component tREST : Send a message for RESTful webservice and retrieve the response.
• Add the component tHDFSOutput : Writes data to HDFS.
• Create links:
  • HDFSConnection is connected with tREST (through "OnSubjobOk")
  • tREST is connected with tHDFSOutput (through "Main")

• Double click on tHDFSConnection and set its properties:
  • Add a "Cloudera" distribution and select the latest version of Cloudera
  • Enter the NameNode URL. 
    The URL has to respect this format :

    hdfs://ip_hdfs:port_hdfs/

    Use context variables if possible : "hdfs://"+context.IP_HDFS+":"+context.Port_HDFS+"/" 
  • Add the user
  • Untick Use Datanode Hostname

• Double click on tREST and enter URL

• Double click on tHDFSOutput :
  • Tick Use an existing connection
  • Enter a path file

• Run the job