OK, some more information.
I realised that although I could not edit the workflow.xml directly in Hive I could find its location in hdfs, download it, edit it, and upload it again. This has resulted in some progress.
The job fails with
117291 [main] INFO hive.metastore – Trying to connect to metastore with URI thrift://bruathdp003.MYDOMAIN.local:9083
2015-07-01 17:54:27,343 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(319)) – Trying to connect to metastore with URI thrift://bruathdp003.MYDOMAIN.local:9083
2015-07-01 17:54:27,346 ERROR [main] transport.TSaslTransport (TSaslTransport.java:open(296)) – SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
Now – I don’t know how to switch on Kerberos debugging here (it is a command line parameter for the JVM) but I am suspicious….
I added in this to the top of my XML
<credentials>
<credential name=’hive_credentials’ type=’hcat’>
<property>
<name>hcat.metastore.uri</name>
<value>thrift://bruathdp003.MYDOMAIN.local:9083</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/_HOST@MYDOMAIN.LOCAL</value>
</property>
</credential>
</credentials>
This suggests to me that it is actually trying to connect to the thrift server as use hive – BUT user hive does not have any kerberos keytab files on the worker (YARN) nodes – so that wont be found.
Does the file listed in hive.server2.authentication.kerberos.keytab need to exist on ALL my boxes or just the HiveServer2/Metastore one?
Thanks!