I’m tried to do the hdp 2.2 to 2.3 rolling upgrade through Ambari 2.1.1. It had problems due to the second namenode being stuck in maintenance mode. So I rolled back. Ever since I did that, mapreduce2 history server is having problems One of them is an ArrayOutOfBounds exception. Does anyone see what I might be missing?
STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 7d56f02902b436d46efba030651a2fbe7c1cf1e9; compiled by 'jenkins' on 2014-11-19T19:42Z
STARTUP_MSG: java = 1.7.0_67
************************************************************/
2015-09-06 10:35:05,198 INFO hs.JobHistoryServer (SignalLogger.java:register(91)) - registered UNIX signal handlers for [TERM, HUP, INT]
2015-09-06 10:35:06,363 INFO hs.JobHistory (JobHistory.java:serviceInit(83)) - JobHistory Init
2015-09-06 10:35:07,104 INFO jobhistory.JobHistoryUtils (JobHistoryUtils.java:getDefaultFileContext(248)) - Default file system [hdfs://hdp:8020]
2015-09-06 10:35:07,278 INFO jobhistory.JobHistoryUtils (JobHistoryUtils.java:getDefaultFileContext(248)) - Default file system [hdfs://hdp:8020]
2015-09-06 10:35:17,297 INFO hs.HistoryFileManager (HistoryFileManager.java:initExisting(664)) - Initializing Existing Jobs...
2015-09-06 10:35:17,309 INFO hs.CachedHistoryStorage (CachedHistoryStorage.java:serviceInit(66)) - CachedHistoryStorage Init
2015-09-06 10:35:17,343 INFO ipc.CallQueueManager (CallQueueManager.java:(53)) - Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-09-06 10:35:17,354 INFO ipc.Server (Server.java:run(605)) - Starting Socket Reader #1 for port 10033
2015-09-06 10:35:17,435 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(111)) - loaded properties from hadoop-metrics2.properties
2015-09-06 10:35:17,507 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(376)) - Scheduled snapshot period at 60 second(s).
2015-09-06 10:35:17,507 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobHistoryServer metrics system started
2015-09-06 10:35:17,516 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:updateCurrentKey(325)) - Updating the current master key for generating delegation tokens
2015-09-06 10:35:17,518 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(638)) - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2015-09-06 10:35:17,518 INFO delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:updateCurrentKey(325)) - Updating the current master key for generating delegation tokens
2015-09-06 10:35:17,600 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-09-06 10:35:17,607 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.jobhistory is not defined
2015-09-06 10:35:17,621 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(699)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-09-06 10:35:17,624 INFO http.HttpServer2 (HttpServer2.java:addFilter(677)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context jobhistory
2015-09-06 10:35:17,624 INFO http.HttpServer2 (HttpServer2.java:addFilter(684)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2015-09-06 10:35:17,624 INFO http.HttpServer2 (HttpServer2.java:addFilter(684)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2015-09-06 10:35:17,628 INFO http.HttpServer2 (HttpServer2.java:initializeWebServer(402)) - adding path spec: /jobhistory/*
2015-09-06 10:35:17,628 INFO http.HttpServer2 (HttpServer2.java:initializeWebServer(402)) - adding path spec: /ws/*
2015-09-06 10:35:17,639 INFO http.HttpServer2 (HttpServer2.java:openListeners(887)) - Jetty bound to port 19888
2015-09-06 10:35:17,639 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx
2015-09-06 10:35:17,672 INFO mortbay.log (Slf4jLog.java:info(67)) - Extract jar:file:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar!/webapps/jobhistory to /tmp/Jetty_0_0_0_0_19888_jobhistory____.djq1tw/webapp
2015-09-06 10:35:17,999 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:19888
2015-09-06 10:35:18,000 INFO webapp.WebApps (WebApps.java:start(274)) - Web app /jobhistory started at 19888
2015-09-06 10:35:18,423 INFO webapp.WebApps (WebApps.java:start(288)) - Registered webapp guice modules
2015-09-06 10:35:18,441 INFO ipc.CallQueueManager (CallQueueManager.java:(53)) - Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-09-06 10:35:18,448 INFO ipc.Server (Server.java:run(605)) - Starting Socket Reader #1 for port 10020
2015-09-06 10:35:18,462 INFO pb.RpcServerFactoryPBImpl (RpcServerFactoryPBImpl.java:createServer(174)) - Adding protocol org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB to the server
2015-09-06 10:35:18,462 INFO ipc.Server (Server.java:run(827)) - IPC Server Responder: starting
2015-09-06 10:35:18,464 INFO ipc.Server (Server.java:run(674)) - IPC Server listener on 10020: starting
2015-09-06 10:35:18,467 INFO hs.HistoryClientService (HistoryClientService.java:serviceStart(146)) - Instantiated HistoryClientService at labnamenode2.collabraspace.app/192.168.3.72:10020
2015-09-06 10:35:18,475 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService failed in state STARTED; cause: java.lang.ArrayIndexOutOfBoundsException: 0
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62)
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.creatRMClient(AggregatedLogDeletionService.java:267)
at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.scheduleLogDeletionTask(AggregatedLogDeletionService.java:243)
at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService.serviceStart(AggregatedLogDeletionService.java:192)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStart(JobHistoryServer.java:194)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:220)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:229)