Hi.
We have also encountered this issue with HDP2.2 and with security enabled. Since this issue was very hard to locate i thought I should share this. The issue was https://issues.apache.org/jira/browse/YARN-2964
What we could be observe:
* In oozie, running a hive action with several statements taking longer than 10minutes in total would trigger this issue. Altohugh it can be triggered by any Oozie action that spawns sub-jobs, i.e Hive, Pig etc
Some logs
Hive logs:
2015-03-27 13:13:19,602 WARN [LeaseRenewer:123.123.123.123::8020] ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 1234 for foobar) can't be found in cache
In the logs of resource manager:
2015-03-30 10:45:15,173 WARN security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(785)) - Unable to add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: 123.123.123.123:8020, Ident: (HDFS_DELEGATION_TOKEN token 1234 for foobar)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Renewal request for unknown token
at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:502)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Renewal request for unknown token
at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:502)