Ask StreamSets - RSS feedhttps://ask.streamsets.com/questions/Question & Answer site for StreamSets big data integration productsenCopyright StreamSets, Inc., 2017Fri, 14 Dec 2018 00:22:43 -0600How to read latest file (always) automatically in sftp/ftp?https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/ I have ftp where I want to take the file that is last dropped. I tried First File field with file pattern ABC*.csv.
But I got below exception:
org.apache.commons.vfs2.FileSystemException: Could not determine the last modified timestamp of "ftp://****/**/**/****/***/ABC*.csv" because it does not exist.
Thanks
RajashreeWed, 12 Dec 2018 02:47:18 -0600https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/Answer by iamontheinet for <p>I have ftp where I want to take the file that is last dropped. I tried First File field with file pattern ABC*.csv.</p>
<p>But I got below exception:
org.apache.commons.vfs2.FileSystemException: Could not determine the last modified timestamp of "ftp://<strong><em>*/</em></strong><em>/<strong>/</strong></em><strong><em></em></strong><em>/</em><em>*/ABC</em>.csv" because it does not exist. </p>
<p>Thanks
Rajashree</p>
https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?answer=3970#post-id-3970Hi!
It appears that the **Resource URL** might not be evaluating correctly. For reference, see below:
![image description](/upfiles/15446291001640005.png)
In the above config, all files relative to ***$USER_HOME/datafiles*** folder will be processed, including new files (with later timestamp) that are uploaded to that folder after pipeline has started.
Hope this helps.
Cheers,
Dash
Wed, 12 Dec 2018 09:44:25 -0600https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?answer=3970#post-id-3970Comment by Tutgirl for <p>Hi!</p>
<p>It appears that the <strong>Resource URL</strong> might not be evaluating correctly. For reference, see below:</p>
<p><img alt="image description" src="/upfiles/15446291001640005.png"></p>
<p>In the above config, all files relative to <strong><em>$USER_HOME/datafiles</em></strong> folder will be processed, including new files (with later timestamp) that are uploaded to that folder after pipeline has started.</p>
<p>Hope this helps.</p>
<p>Cheers,
Dash</p>
https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3992#post-id-3992Hi
I am not able to upload the screenshot image. The files are already uploaded and then I'm starting the pipeline.
But when it go to production the files will be dropped when the pipeline is running, so I would like to know both the scenarios.
Thanks in advanceFri, 14 Dec 2018 00:22:43 -0600https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3992#post-id-3992Comment by iamontheinet for <p>Hi!</p>
<p>It appears that the <strong>Resource URL</strong> might not be evaluating correctly. For reference, see below:</p>
<p><img alt="image description" src="/upfiles/15446291001640005.png"></p>
<p>In the above config, all files relative to <strong><em>$USER_HOME/datafiles</em></strong> folder will be processed, including new files (with later timestamp) that are uploaded to that folder after pipeline has started.</p>
<p>Hope this helps.</p>
<p>Cheers,
Dash</p>
https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3986#post-id-3986It will process files in earliest to latest timestamp order. Can you update your original question and add screenshot of your config like I did in my answer? Also, are you starting and stopping the pipeline before uploading new files or are you uploading new files while the pipeline is running?Thu, 13 Dec 2018 11:36:04 -0600https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3986#post-id-3986Comment by Tutgirl for <p>Hi!</p>
<p>It appears that the <strong>Resource URL</strong> might not be evaluating correctly. For reference, see below:</p>
<p><img alt="image description" src="/upfiles/15446291001640005.png"></p>
<p>In the above config, all files relative to <strong><em>$USER_HOME/datafiles</em></strong> folder will be processed, including new files (with later timestamp) that are uploaded to that folder after pipeline has started.</p>
<p>Hope this helps.</p>
<p>Cheers,
Dash</p>
https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3978#post-id-3978I have tried this, its not giving any error but it is picking the earliest file not the latest.Thu, 13 Dec 2018 04:34:30 -0600https://ask.streamsets.com/question/3966/how-to-read-latest-file-always-automatically-in-sftpftp/?comment=3978#post-id-3978