Reply To: Hortonworks Teradata Connector Incremental import : Teradata table to Hive table

Hi Mahesh,

Thanks a lot for pointing out the mistake.

From http://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_free_form_query_imports :
Note:
If you are issuing the query wrapped with double quotes (“), you will have to use \$CONDITIONS instead of just $CONDITIONS to disallow your shell from treating it as a shell variable. For example, a double quoted query may look like: “SELECT * FROM x WHERE a=’foo’ AND \$CONDITIONS”

$SQOOP_HOME/bin/sqoop import -libjars $LIB_JARS –connection-manager org.apache.sqoop.teradata.TeradataConnManager –connect jdbc:teradata://xxx.xxx.xxx.xxx/terdatadb –username testuser –password testpwd -query “select * from bookinfo where last_modified_col > ‘2014-12-17 13:49:20.460000′ AND \$CONDITIONS” –target-dir /user/hive/warehouse/bookinfo –split-by bookid

14/12/18 12:04:55 ERROR teradata.TeradataSqoopImportHelper: Exception running Teradata import job
com.teradata.connector.common.exception.ConnectorException: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/hive/warehouse/bookinfo already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at com.teradata.connector.common.ConnectorOutputFormat.checkOutputSpecs(ConnectorOutputFormat.java:42)

Next if I mention a new directory then it works: –target-dir /user/hive/warehouse/bookinfo_new_dir
$SQOOP_HOME/bin/sqoop import -libjars $LIB_JARS –connection-manager org.apache.sqoop.teradata.TeradataConnManager –connect jdbc:teradata://xxx.xxx.xxx.xxx/terdatadb –username testuser –password testpwd -query “select * from bookinfo where last_modified_col > ‘2014-12-17 13:49:20.460000′ AND \$CONDITIONS” –target-dir /user/hive/warehouse/bookinfo_new_dir –split-by bookid

This imports the data in HDFS ONLY and does not create a new hive table. I can see the data in HDFS:
/user/hive/warehouse/bookinfo_new_dir
/user/hive/warehouse/bookinfo_new_dir/_SUCCESS
/user/hive/warehouse/bookinfo_new_dir/part-m-00000
/user/hive/warehouse/bookinfo_new_dir/part-m-00001
/user/hive/warehouse/bookinfo_new_dir/part-m-00002
/user/hive/warehouse/bookinfo_new_dir/part-m-00003

But I need to do an incremental import in the existing Hive ‘bookinfo’ table.

Also from the documentation bk_HortonworksConnectorForTeradata.pdf I see below options are NOT Supported for import:
Unsupported Sqoop Import Options
Incremental Options
–check-column
–incremental
–last-value

Hive Support Options
–hive-delims-replacement
–hive-drop-import-delims
–hive-home
–hive-overwrite
–hive-partition-key
–hive-partition-value

–target-dir seems to be mandatory when using ‘Free-form Query Imports’.
When I am using the existing one it gives me error:Output directory /user/hive/warehouse/bookinfo already exists
If I CANNOT use the same –target-dir then how can I do an incremental import to the same Hive table?

Please help me how can i do incremental import in an existing table.

Thanks,
-Nirmal