Using a remote file as input

Dear all,First of all, I'm already very sorry, as this might to be a very basic question... But I am starting to dig into Talend and still feel quite lost :-)What I would like to do is (I guess) pretty simple: download an xml or csv file through http, map it against an existing PostgreSql table and populate the Postrgre table with the records of the input file.My problem is that the component tFileInputDelimited seems to accept only local files and don't recognize my http string:http://earthquake.usgs.gov/earthquakes/catalogs/eqs1day-M1.txtIs there a particular syntax? or did I forget to do something? a component that will download the file to feed the tFileInputDelimited component?Thanks a lot in advance for your help!Stéphane

Re: Using a remote file as input

Well... I have one already!How to link the tFileFetch component to the tFileInputDelimited?Should I "hard-code" in the parameters of tFileFetch the output path and name and then use this as an input for tFileInputDelimited? Or is there a way to link directly the 2 components? (so that, e.g., if I change that output path of tFileFetch, the input path of tFileInputDelimited is automatically updated).And, in any case, will the output file of tFileFetch replace the previous output file everytime I run the job? Or is there a specific setting to setup to reach this behavior?Thanks again!Stéphane

Re: Using a remote file as input

so that, e.g., if I change that output path of tFileFetch, the input path of tFileInputDelimited is automatically updated).

Define a context variable for the output path of tFileFetch and the input path of tFileInputDelimited, so that both the two component always use the same path.

And, in any case, will the output file of tFileFetch replace the previous output file everytime I run the job?

If you set a fixed file name, it will replace the previous output file everytime, so you need to set a dynamical file name, eg: "test"+TalendDate.formatDate("yyyy/MM/dd_HH:mm",TalendDate.getCurrentDate())+".txt"when you run the job, it generate a file called:test_2011/01/20_13:27.txtBest regardsShong

----------------------------------------------------------Talend | Data Agility for Modern Business

Re: Using a remote file as input

It works perfectly!Now just to improve a little bit, I would like to use what you wrote in your first post in order to keep tracks of all fetched files:"test"+TalendDate.formatDate("yyyy/MM/dd_HH:mm",TalendDate.getCurrentDate())+".txt"This used to work if I hard-code the path and file name but not if I enter it as default value in the context variable. Any advice on this?Thanks!

Re: Using a remote file as input

Hello Add a tJava at the begining of job, for example:tJava |onsubjobok |tFileFetch |onsubjobok |tFileInputDelimited--main-->tLogRowand type in the following code on tJava:context.filename="test"+TalendDate.formatDate("yyyy/MM/dd_HH:mm",TalendDate.getCurrentDate())+".txt";Best regardsShong

----------------------------------------------------------Talend | Data Agility for Modern Business

Re: Using a remote file as input

Hi Shong,I followed this example as explained by you. I am getting exception as below:Exception in component tFileFetch_1java.lang.Exception: Method failed: HTTP/1.1 405 Method Not Allowed at talenddemosjava.filefetchfromhttp_0_1.FileFetchFromHTTP.tFileFetch_1Process(FileFetchFromHTTP.java:427) at talenddemosjava.filefetchfromhttp_0_1.FileFetchFromHTTP.runJobInTOS(FileFetchFromHTTP.java:1263) at talenddemosjava.filefetchfromhttp_0_1.FileFetchFromHTTP.main(FileFetchFromHTTP.java:1125)The settings are exactly similar as given here, only difference is, my url is https so I have selected the protocol as https. (tried with http too!)Screenshots are attached.Appreciate your help!!

Re: Using a remote file as input

Hi

Method failed: HTTP/1.1 405 Method Not Allowed

The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource. Try to uncheck the box 'post method'.Best regardsShong

----------------------------------------------------------Talend | Data Agility for Modern Business