Hi All
I find that HDP2.x (2.1,2.2.,2.3 ) use a commons-compress-1.4.1.jar ,which has some error with my program of mapreduce2.
The error is like this:
Caused by: java.io.IOException:
at org.apache.tika.parser.ParsingReader.read(ParsingReader.java:260)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.LineNumberReader.readLine(LineNumberReader.java:199)
at com.spss.ae.taparser.TAParser.readLine(TAParser.java:212)
at com.spss.ae.taparser.TAParser.readRecord(TAParser.java:269)
… 33 more
Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.compressors.CompressorStreamFactory.setDecompressConcatenated(Z)V
at org.apache.tika.parser.pkg.CompressorParser.parse(CompressorParser.java:102)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120)
at org.apache.tika.parser.ParsingReader$ParsingTask.run(ParsingReader.java:221)
at java.lang.Thread.run(Thread.java:745)
I found this is because the HDP use a version of commons-compress-1.4.1.jar , which conflicts with other jars .
If replace it with commons-compress-1.9.jar then my program is ok. On Windows HDP2.1 this is easy to do, just replace the {Hadoop-Install}/share/hadoop/common/lib/commons-compress-1.4.1.jar with the commons-compress-1.9.jar , then restart HDP .
But the replacing jar seems not work in HDP2.2 Linux even if replace every commons-compress-1.4.1.jar in hadoop install folders ( {hadoop-install}/hadoop/lib , {hadoop-install}/hadoop-yarn/lib, {hadoop-install}/hadoop-mapreduce,{hadoop-install}/hadoop-mapreduce/lib )
Do you know how to upgrade a single jar, such as commons-compress-1.4.1.jar to higher version in HDP2.x?
Thanks