After more research, I found the active NN shut down by itself after failed to write to all the JNs with the following log:
2015-04-24 14:30:09,836 WARN client.QuorumJournalManager (IPCLoggerChannel.java:call(360)) – Remote journal 10.1.10.7:8485 failed to write txns 26843227-26843227. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): IPC’s epoch 39 is less than the last promised epoch 40
(stack trace too long to post)
2015-04-24 14:30:09,837 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(364)) – Error: flush failed for required journal (JournalAndStream(mgr=QJM to [10.1.10.7:8485, 10.1.10.8:8485, 10.1.10.9:8485], stream=QuorumOutputStream starting at txid 26843227))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.1.10.7:8485: IPC’s epoch 39 is less than the last promised epoch 40
… (stack trace too long to post)
2015-04-24 14:30:09,838 WARN client.QuorumJournalManager (QuorumOutputStream.java:abort(72)) – Aborting QuorumOutputStream starting at txid 26843227
2015-04-24 14:30:09,873 INFO util.ExitUtil (ExitUtil.java:terminate(124)) – Exiting with status 1
2015-04-24 14:30:09,891 INFO namenode.NameNode (StringUtils.java:run(640)) – SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at name1/10.1.10.6
************************************************************/
Need to find out why it failed to write to JNs.