Hi All,
I am able to import the Teradata Table in Hive table using:
$SQOOP_HOME/bin/sqoop import -libjars $LIB_JARS -Dteradata.db.input.target.database=hive_database -Dteradata.db.input.target.table.schema=”Id int, PaperTitle string,PaperYear string,ConferenceId int, JournalId int, Keyword string, last_modified_col timestamp” -Dteradata.db.input.split.by.column=Id –connection-manager org.apache.sqoop.teradata.TeradataConnManager –connect jdbc:teradata://192.168.199.137/testtd –username testtd –password etloffload –table Paper_STAGE –hive-import
The Hive table is created and loaded with data.
However, I see that
-Dteradata.db.input.target.table.schema=”Id int, PaperTitle string,PaperYear string,ConferenceId int, JournalId int, Keyword string, last_modified_col timestamp”
is always required for the import.
If I remove this property I get the below exception:
14/10/09 13:01:21 INFO processor.HiveOutputProcessor: hive table hive_database.Paper_STAGE does not exist
14/10/09 13:01:21 WARN tool.ConnectorJobRunner: com.teradata.connector.common.exception.ConnectorException: The output post processor returns 1
14/10/09 13:01:21 INFO processor.TeradataInputProcessor: input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByHashProcessor starts at: 1412839881857
14/10/09 13:01:26 INFO processor.TeradataInputProcessor: input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByHashProcessor ends at: 1412839881857
14/10/09 13:01:26 INFO processor.TeradataInputProcessor: the total elapsed time of input postprocessor com.teradata.connector.teradata.processor.TeradataSplitByHashProcessor is: 5s
14/10/09 13:01:26 ERROR teradata.TeradataSqoopImportHelper: Exception running Teradata import job
com.teradata.connector.common.exception.ConnectorException: Import Hive table’s column schema is missing
at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:104)
at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:48)
at org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)
at org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Is -Dteradata.db.input.target.table.schema always required for first time import ?
Do I need to map the Teradata Data Types to Hive Data Types manually?
And how about handling Teradata specific data types like Period, Interval and also CLOB, BLOB?
Can these be mapped with ‘string’ in Hive?
Thanks,
-Nirmal