Hi…
I want to import tables from MSSQL Server to HDFS on regular basis (twice a day). How do I do it through sqoop incremental import. Many of my table do not have primary key defined on it. Even tables do not have update date column. Is it possible to import all the tables in one go, coz I have 300 tables that needs to be imported on daily basis.. please help me with this

Hi Patrick,
Since you are looking to append the table, you would need to use ‘–-incremental append’ with the –check-column.
Refer: http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports.
Please give this a try and see if you are still seeing the issue.
-Mahesh
...

I used the following, but every time I run this it loads the entire table and not the incremental
sqoop import –connect ‘jdbc:sqlserver://10.10.100.62:4101;database=mycollection;username=hadoop;password=hadoop’ –table PortalInformation –hive-import — –schema dbo –-incremental -–check-column LineID –last-value 100
what am I missing ?
Thanks in advance,
Patrick
...

For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. ...

For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job –exec someIncrementalJob will continue to import only newer rows than those previously imported.

For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work

Each table must have a single-column primary key.
You must intend to import all columns of each table.
You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.