I have an Oozie workflow running a Hive action. The hive action is pretty simple, it just reads from one table copies to another table. The job ran with the following properties:
user.name=yarn
mapreduce.job.user.name=cloudfeeds
The job fails in HDP 2.2.4.2 cluster with the following error:
15/07/16 18:45:25 INFO mapreduce.Job: Job job_1435680786374_0060 failed with state FAILED due to: Application application_1435680786374_0060 failed 2 times due to AM Container for appattempt_1435680786374_0060_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://master-1.local:8088/proxy/application_1435680786374_0060/Then, click on links to logs of each attempt.
Diagnostics: Permission denied: user=cloudfeeds, access=EXECUTE, inode=”/user/yarn/.staging”:yarn:hdfs:drwx——
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:208)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:171)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6795)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4387)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:855)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:835)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
I don’t understand why user cloudfeeds needs to access the /user/yarn/.staging directory. If the job runs as user cloudfeeds, shouldn’t it be trying to access /user/cloudfeeds/.staging instead?
How do I fix this?
The same job ran fine in an older cluster running HDP 2.2.0.0.
thanks,
-shinta