Hi,
I have installed a cluster with Ambari 2.1.1 and Ranger 0.5.0.2.3 with a postgresql database. There are two databases in this postgresql named ranger and ranger_audit. I have also enabled the Ranger HDFS plugin through the Ambari web console and I’m able to apply policies to the HDFS service.
The problem comes when I activate the HDFS Audit to DB. I can see this messages into the HDFS log that repeats forever:
2015-09-29 14:24:56,983 ERROR utils.RangerCredentialProvider (RangerCredentialProvider.java:getCredentialProviders(80)) – Unable to get the Credential Provider from the Configuration
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:698)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:106)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialProviders(RangerCredentialProvider.java:78)
at org.apache.ranger.authorization.hadoop.utils.RangerCredentialProvider.getCredentialString(RangerCredentialProvider.java:54)
at org.apache.ranger.audit.provider.MiscUtil.getCredentialString(MiscUtil.java:410)
at org.apache.ranger.audit.destination.DBAuditDestination.connect(DBAuditDestination.java:152)
at org.apache.ranger.audit.destination.DBAuditDestination.getTransaction(DBAuditDestination.java:252)
at org.apache.ranger.audit.destination.DBAuditDestination.beginTransaction(DBAuditDestination.java:261)
at org.apache.ranger.audit.destination.DBAuditDestination.log(DBAuditDestination.java:84)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
at java.lang.Thread.run(Thread.java:745)
2015-09-29 14:24:56,984 INFO destination.DBAuditDestination (DBAuditDestination.java:connect(164)) – JDBC Driver=org.postgresql.Driver, JDBC URL=jdbc:postgresql://sv01pronode01.datalab1.sp.ingdirect.intranet/ranger_audit, dbUser=rangerlogger, passwordAlias=auditDBCred, credFile=jceks://file/etc/ranger/datalab1_hadoop/cred.jceks, usingPassword=yes
2015-09-29 14:24:56,987 INFO destination.DBAuditDestination (DBAuditDestination.java:connect(180)) – entityManagerFactory=org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl@5e16dfa6
2015-09-29 14:24:56,990 ERROR destination.DBAuditDestination (DBAuditDestination.java:connect(195)) – Error connecting audit database. dbURL=jdbc:postgresql://sv01pronode01.datalab1.sp.ingdirect.intranet/ranger_audit, dbUser=rangerlogger
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services – 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
Error Code: 0
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:766)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:204)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:304)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:336)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:302)
at org.apache.ranger.audit.dao.DaoManager.getEntityManager(DaoManager.java:48)
at org.apache.ranger.audit.destination.DBAuditDestination.connect(DBAuditDestination.java:186)
at org.apache.ranger.audit.destination.DBAuditDestination.getTransaction(DBAuditDestination.java:252)
at org.apache.ranger.audit.destination.DBAuditDestination.beginTransaction(DBAuditDestination.java:261)
at org.apache.ranger.audit.destination.DBAuditDestination.log(DBAuditDestination.java:84)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services – 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
Error Code: 0
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:326)
at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:138)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:204)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:741)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:239)
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:685)
… 19 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:400)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:410)
at org.postgresql.Driver.connect(Driver.java:280)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:98)
… 24 more
2015-09-29 14:24:56,992 ERROR destination.DBAuditDestination (DBAuditDestination.java:getEntityManager(223)) – DBAuditDestination.getEntityManager(): failed
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services – 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
Error Code: 0
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:766)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:204)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:304)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:336)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:302)
at org.apache.ranger.audit.dao.DaoManager.getEntityManager(DaoManager.java:48)
at org.apache.ranger.audit.destination.DBAuditDestination.getEntityManager(DBAuditDestination.java:221)
at org.apache.ranger.audit.destination.DBAuditDestination.getTransaction(DBAuditDestination.java:255)
at org.apache.ranger.audit.destination.DBAuditDestination.beginTransaction(DBAuditDestination.java:261)
at org.apache.ranger.audit.destination.DBAuditDestination.log(DBAuditDestination.java:84)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services – 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
Error Code: 0
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:331)
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:326)
at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:138)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:204)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:741)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:239)
at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:685)
… 19 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user “rangerlogger”
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:400)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:410)
at org.postgresql.Driver.connect(Driver.java:280)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:98)
… 24 more
2015-09-29 14:24:56,992 INFO destination.DBAuditDestination (DBAuditDestination.java:cleanUp(201)) – DBAuditDestination: cleanUp()
2015-09-29 14:24:56,992 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(301)) – Audit Status Log: name=hdfs.async.batch.db, interval=30.029 seconds, events=1, deferredCount=1, totalEvents=224, totalDeferredCount=223
2015-09-29 14:24:56,992 WARN destination.DBAuditDestination (DBAuditDestination.java:beginTransaction(268)) – DBAuditDestination.beginTransaction(): trx is null
2015-09-29 14:24:56,992 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) – Error sending logs to consumer. provider=hdfs.async.batch, consumer=hdfs.async.batch.db
I checked the file /etc/ranger/datalab1_hadoop/cred.jceks and it has zero size. Seems that the namenode cannot get the credentials to access the audit database from the jceks. I can see other files with information:
1044 4 -rwx—— 1 ranger ranger 960 Sep 29 11:54 /etc/ranger/admin/rangeradmin.jceks
1067 0 -rw-r–r– 1 hdfs hdfs 0 Sep 29 11:59 /etc/ranger/datalab1_hadoop/cred.jceks
1037 4 -rwx—— 1 ranger ranger 995 Sep 29 11:47 /etc/ranger-usersync/2.3.0.0-2557/0/ugsync.jceks
Can you help me?
Thank you