I’m getting errors in hive every ~2-3 days now with:
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:165)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:153)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:198)
Once that happens, it appears that hive basically dies. It starts reporting errors like:
java.sql.SQLException: Error while compiling statement: FAILED: RuntimeException org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzPluginException: Failed to retrieve roles for {username}: javax.jdo.JDOException: Exception thrown when executing query
… java.sql.SQLException: Connection is closed!
Restarting the hive tasks fixes it.
I’m not seeing anything strange in the hdfs logs or in the hive logs, other than the “connection closed” messages. Given that restarting the sql server “fixes” it, it seems like a hive issue. There is a strange-ish message in hivemestastore which sounds like it’s closing a connection (possibly due to some other error?):
—
2014-10-19 08:11:57,154 INFO [pool-3-thread-187]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(661)) – 15: source:/192.168.32.2 alter_partition : db=default tbl=mytable
2014-10-19 08:11:57,155 INFO [pool-3-thread-187]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(312)) – ugi=hive ip=/192.168.32.2 cmd=source:/192.168.32.2 alter_partition : db=defa
ult tbl=mytable
2014-10-19 08:11:57,155 INFO [pool-3-thread-187]: metastore.HiveMetaStore (HiveMetaStore.java:rename_partition(2644)) – New partition values:[20141019]
2014-10-19 08:11:57,183 WARN [pool-3-thread-187]: hive.log (MetaStoreUtils.java:updatePartitionStatsFast(297)) – Updating partition stats fast for: mytable
2014-10-19 08:11:57,189 WARN [pool-3-thread-187]: hive.log (MetaStoreUtils.java:updatePartitionStatsFast(300)) – Updated size to 2693681334
2014-10-19 08:11:57,249 INFO [pool-3-thread-187]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(661)) – 15: source:/192.168.32.2 get_partition_with_auth : db=default tbl=mytable[20141019]
2014-10-19 08:11:57,249 INFO [pool-3-thread-187]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(312)) – ugi=hive ip=/192.168.32.2 cmd=source:/192.168.32.2 get_partition_with_auth : db=default tbl=mytable[20141019]
2014-10-19 08:11:57,268 WARN [com.google.common.base.internal.Finalizer]: bonecp.ConnectionPartition (ConnectionPartition.java:finalizeReferent(162)) – BoneCP detected an unclosed connection and will now attempt to close it for you. You should be closing this connection in your application – enable connectionWatch for additional debugging assistance or set disableConnectionTracking to true to disable this feature entirely.
—
Any tips on debugging this one?