HDP crew:
I’m trying to run Sqoop on Windows Server 2012 using the latest HDP release. I have Hadoop on Windows configured as a remote client to a cluster running on EMR and can execute hdfs commands, etc. However, when I try to import a database using Sqoop, it appears that Sqoop tries to move some jar files into HDFS (presumably to support the map-reduce job), but since those jar file paths are in Windows form (eg, “e:\hdp\blah\whatever”), it fails when HDFS rejects the path as invalid:
15/05/19 17:47:34 DEBUG mapreduce.DataDrivenImportJob: Using InputFormat: class com.cloudera.sqoop.mapreduce.db.DataDriv
enDBInputFormat
15/05/19 17:47:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/E:/hdp/sqoop-1.4.5.2.2.4.2-0002/sqoop-1.4.5.2.
2.4.2-0002.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/E:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/db2jcc4.ja
r
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/E:/hdp/sqoop-1.4.5.2.2.4.2-0002/sqoop-1.4.5.2.
2.4.2-0002.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/E:/hdp/sqoop-1.4.5.2.2.4.2-0002/sqoop-1.4.5.2.
2.4.2-0002.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/ant-contri
b-1.0b3.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/ant-eclips
e-1.0-jvm1.2.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/avro-1.7.5
.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/avro-ipc-1
.7.5-tests.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/avro-mapre
d-1.7.5.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/commons-co
mpress-1.4.1.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/commons-io
-1.4.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/db2jcc4.ja
r
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/hsqldb-1.8
.0.10.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/jackson-co
re-asl-1.9.13.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/jackson-ma
pper-asl-1.9.13.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/jetty-util
-6.1.26.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/netty-3.4.
0.Final.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/paranamer-
2.3.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/servlet-ap
i-2.5-20081211.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/snappy-jav
a-1.0.5.jar
15/05/19 17:47:34 DEBUG mapreduce.JobBase: Adding to job classpath: file:/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/xz-1.0.jar
15/05/19 17:47:34 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/05/19 17:47:34 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/05/19 17:47:38 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/schtableau755789086/.staging/job_l
ocal755789086_0001
15/05/19 17:47:38 ERROR tool.ImportTool: Imported Failed: Pathname /e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/xz-1.0.jar from
hdfs://10.1.115.231:9000/e:/hdp/sqoop-1.4.5.2.2.4.2-0002/lib/xz-1.0.jar is not a valid DFS filename.
PS C:\>
Does anybody have any idea how I might solve this?