1 Answer

If you don't need to process the file content, you can use Whole File data format to copy the files from SFTP to HDFS.

If you do need to process the records in each file, it's a little more complicated. You would need to use an Expression Evaluator to set the roll header attribute when record:id() ends in ::0, then configure the HDFS destination to use the roll attribute.

Comments

Hi metadady,
Thanks for an update. Now I can move the files from SFTP to HDFS. Actually the files are compressed. So please tell me how can I uncompress the Whole File when it is residing on before in the HDFS?

Hi metadaddy,
I tried to copy the files from SFTP to HDFS by using the Whole File data format.The files are moved to hdfs but the moved files do not containing all the contents some contents are missing in hdfs file.Please hell me to configure.

@Sam Sambath: Can you please share the configuration screenshots how you configure for sftp to hdfs. Please help me i try to configure from past 3 days but i'm not able to pull data from sftp. Connections and configurations all are correct.