Hi.
I’m using HDP-2.1.5 with kerberization.
When I use hive metastore (through hiveserver2, ultimately from CDH 5.1.3 hue), I see the following error in hive metastore:
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: owner=helloAhn, renewer=helloAhn, realUser=hive/master-5.amber.gbcl.net@AMBER.GBCLUSTER.NET, issueDate=1415065084288, maxDate=1415669884288, sequenceNumber=66, masterKeyId=2]
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:594)
at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:509)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:676)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:673)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1574)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:673)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
So it seems that security token has been expired or removed.
The current settings are following:
– kerberos ticket max lifetime: 1 day
– kerberos ticket max renew lifetime: 7 days
– hive.cluster.delegation.token.max-lifetime: 7days
– hive.cluster.delegation.key.update-interval: 1day
– hive.cluster.delegation.token.renew-interval: 1day
I noticed that this error is user-specific, meaning that some user faces this problem and the other do not.
I suspect security token renewal or expiration doesn’t honor its nominal value (1day and 7days, respectively)
There’s some pattern in this error. When user A tries to access metastore about 24hours later than the first access (which was successful), this occurs.
Can anybody please help me?