I have six nodes, 5 cluster nodes and a dedicated Ambari node. All boxes are clean installs of Ubuntu Server 12.04 LTS running on OpenStack in our private cloud. Ambari installed with a few hiccups and after some effort I was able to use it to deploy HDP 2.2 to my 5 nodes in the cluster. However, many of the services fail to start on the nodes. However, many of the failures are likely related to the failure of the NameNode and sNameNode to start. The failure log for the name node is below, but I suspect the issue revolves around the bit:
Fail: Execution of ‘ulimit -c unlimited; su -s /bin/bash – hdfs -c ‘export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /etc/hadoop/conf start namenode” returned 1. stdin: is not a tty
I have seen a couple of posts about various bugs relating to the tty problem. Some allude to a python issue, while other point to linux(ubuntu) when ssh to a box is from another box that has been ssh’d into. But I have not found anything that specifically addresses this issue.
I have followed Hortonworks node setup for HDP 2.2. The stack is:
Ubuntu Server 12.04 LTS deployed in OpenStack
Ambari 1.7.0
HDP 2.2
(Worth noting is that we are trying Ubuntu servers as we had issues with the Ambari install on the Satellite distribution of CentOS – lots of packages missing out of the box.)
Installed with the defaults across the board. Please advise if you have encountered this error or are aware of what may be the cause. Also, let me know if there are additional details that would help with analysis.
Thank you.
stderr:
2015-03-04 21:13:23,193 – Error while executing command ‘start':
Traceback (most recent call last):
File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 123, in execute
method(env)
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/namenode.py”, line 46, in start
namenode(action=”start”)
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/hdfs_namenode.py”, line 45, in namenode
create_log_dir=True
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/utils.py”, line 102, in service
not_if=service_is_up
File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 148, in __init__
self.env.run()
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 149, in run
self.run_action(resource, action)
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 115, in run_action
provider_action()
File “/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py”, line 241, in action_run
raise ex
Fail: Execution of ‘ulimit -c unlimited; su -s /bin/bash – hdfs -c ‘export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /etc/hadoop/conf start namenode” returned 1. stdin: is not a tty
starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-dev-hdp-0.stp.rbc.out
stdout:
2015-03-04 21:13:18,792 – Execute[‘mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x “” –retry 10 http://dev-ambari-1.stp.rbc:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip’] {‘environment': …, ‘not_if': ‘test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip’, ‘ignore_failures': True, ‘path': [‘/bin’, ‘/usr/bin/’]}
2015-03-04 21:13:18,796 – Skipping Execute[‘mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x “” –retry 10 http://dev-ambari-1.stp.rbc:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip’] due to not_if
2015-03-04 21:13:18,797 – Group[‘hadoop’] {‘ignore_failures': False}
2015-03-04 21:13:18,797 – Modifying group hadoop
2015-03-04 21:13:18,805 – Group[‘nobody’] {‘ignore_failures': False}
2015-03-04 21:13:18,805 – Modifying group nobody
2015-03-04 21:13:18,812 – Group[‘users’] {‘ignore_failures': False}
2015-03-04 21:13:18,812 – Modifying group users
2015-03-04 21:13:18,820 – Group[‘nagios’] {‘ignore_failures': False}
2015-03-04 21:13:18,820 – Modifying group nagios
2015-03-04 21:13:18,827 – Group[‘knox’] {‘ignore_failures': False}
2015-03-04 21:13:18,827 – Modifying group knox
2015-03-04 21:13:18,834 – User[‘nobody’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’nobody’]}
2015-03-04 21:13:18,834 – Modifying user nobody
2015-03-04 21:13:18,840 – User[‘oozie’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’users’]}
2015-03-04 21:13:18,841 – Modifying user oozie
2015-03-04 21:13:18,847 – User[‘mapred’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,847 – Modifying user mapred
2015-03-04 21:13:18,853 – User[‘nagios’] {‘gid': ‘nagios’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,853 – Modifying user nagios
2015-03-04 21:13:18,859 – User[‘ambari-qa’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’users’]}
2015-03-04 21:13:18,859 – Modifying user ambari-qa
2015-03-04 21:13:18,865 – User[‘flume’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,866 – Modifying user flume
2015-03-04 21:13:18,872 – User[‘zookeeper’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,872 – Modifying user zookeeper
2015-03-04 21:13:18,878 – User[‘hdfs’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,878 – Modifying user hdfs
2015-03-04 21:13:18,884 – User[‘kafka’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,884 – Modifying user kafka
2015-03-04 21:13:18,890 – User[‘knox’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,890 – Modifying user knox
2015-03-04 21:13:18,896 – User[‘storm’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,896 – Modifying user storm
2015-03-04 21:13:18,902 – User[‘yarn’] {‘gid': ‘hadoop’, ‘ignore_failures': False, ‘groups': [u’hadoop’]}
2015-03-04 21:13:18,902 – Modifying user yarn
2015-03-04 21:13:18,908 – File[‘/var/lib/ambari-agent/data/tmp/changeUid.sh’] {‘content': StaticFile(‘changeToSecureUid.sh’), ‘mode': 0555}
2015-03-04 21:13:18,909 – Execute[‘/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null’] {‘not_if': ‘test $(id -u ambari-qa) -gt 1000′}
2015-03-04 21:13:18,915 – Skipping Execute[‘/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null’] due to not_if
2015-03-04 21:13:18,915 – Directory[‘/etc/hadoop/conf.empty’] {‘owner': ‘root’, ‘group': ‘root’, ‘recursive': True}
2015-03-04 21:13:18,915 – Link[‘/etc/hadoop/conf’] {‘not_if': ‘ls /etc/hadoop/conf’, ‘to': ‘/etc/hadoop/conf.empty’}
2015-03-04 21:13:18,920 – Skipping Link[‘/etc/hadoop/conf’] due to not_if
2015-03-04 21:13:18,928 – File[‘/etc/hadoop/conf/hadoop-env.sh’] {‘content': InlineTemplate(…), ‘owner': ‘hdfs’}
2015-03-04 21:13:18,935 – Execute[‘/bin/echo 0 > /selinux/enforce’] {‘only_if': ‘test -f /selinux/enforce’}
2015-03-04 21:13:18,940 – Skipping Execute[‘/bin/echo 0 > /selinux/enforce’] due to only_if
2015-03-04 21:13:18,941 – Execute[‘mkdir -p /usr/hdp/current/hadoop-client/lib/native/Linux-i386-32; ln -sf /usr/hdp/current/hadoop-client/lib/libsnappy.so /usr/hdp/current/hadoop-client/lib/native/Linux-i386-32/libsnappy.so’] {}
2015-03-04 21:13:18,948 – Execute[‘mkdir -p /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64; ln -sf /usr/hdp/current/hadoop-client/lib64/libsnappy.so /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libsnappy.so’] {}
2015-03-04 21:13:18,954 – Directory[‘/var/log/hadoop’] {‘owner': ‘root’, ‘group': ‘hadoop’, ‘mode': 0775, ‘recursive': True}
2015-03-04 21:13:18,954 – Directory[‘/var/run/hadoop’] {‘owner': ‘root’, ‘group': ‘root’, ‘recursive': True}
2015-03-04 21:13:18,955 – Directory[‘/tmp/hadoop-hdfs’] {‘owner': ‘hdfs’, ‘recursive': True}
2015-03-04 21:13:18,958 – File[‘/etc/hadoop/conf/commons-logging.properties’] {‘content': Template(‘commons-logging.properties.j2′), ‘owner': ‘hdfs’}
2015-03-04 21:13:18,959 – File[‘/etc/hadoop/conf/health_check’] {‘content': Template(‘health_check-v2.j2′), ‘owner': ‘hdfs’}
2015-03-04 21:13:18,960 – File[‘/etc/hadoop/conf/log4j.properties’] {‘content': ‘…’, ‘owner': ‘hdfs’, ‘group': ‘hadoop’, ‘mode': 0644}
2015-03-04 21:13:18,963 – File[‘/etc/hadoop/conf/hadoop-metrics2.properties’] {‘content': Template(‘hadoop-metrics2.properties.j2′), ‘owner': ‘hdfs’}
2015-03-04 21:13:18,963 – File[‘/etc/hadoop/conf/task-log4j.properties’] {‘content': StaticFile(‘task-log4j.properties’), ‘mode': 0755}
2015-03-04 21:13:19,091 – Directory[‘/etc/security/limits.d’] {‘owner': ‘root’, ‘group': ‘root’, ‘recursive': True}
2015-03-04 21:13:19,095 – File[‘/etc/security/limits.d/hdfs.conf’] {‘content': Template(‘hdfs.conf.j2′), ‘owner': ‘root’, ‘group': ‘root’, ‘mode': 0644}
2015-03-04 21:13:19,095 – XmlConfig[‘hadoop-policy.xml’] {‘owner': ‘hdfs’, ‘group': ‘hadoop’, ‘conf_dir': ‘/etc/hadoop/conf’, ‘configuration_attributes': …, ‘configurations': …}
2015-03-04 21:13:19,102 – Generating config: /etc/hadoop/conf/hadoop-policy.xml
2015-03-04 21:13:19,103 – File[‘/etc/hadoop/conf/hadoop-policy.xml’] {‘owner': ‘hdfs’, ‘content': InlineTemplate(…), ‘group': ‘hadoop’, ‘mode': None, ‘encoding': ‘UTF-8′}
2015-03-04 21:13:19,103 – Writing File[‘/etc/hadoop/conf/hadoop-policy.xml’] because contents don’t match
2015-03-04 21:13:19,103 – XmlConfig[‘hdfs-site.xml’] {‘owner': ‘hdfs’, ‘group': ‘hadoop’, ‘conf_dir': ‘/etc/hadoop/conf’, ‘configuration_attributes': …, ‘configurations': …}
2015-03-04 21:13:19,109 – Generating config: /etc/hadoop/conf/hdfs-site.xml
2015-03-04 21:13:19,110 – File[‘/etc/hadoop/conf/hdfs-site.xml’] {‘owner': ‘hdfs’, ‘content': InlineTemplate(…), ‘group': ‘hadoop’, ‘mode': None, ‘encoding': ‘UTF-8′}
2015-03-04 21:13:19,111 – Writing File[‘/etc/hadoop/conf/hdfs-site.xml’] because contents don’t match
2015-03-04 21:13:19,111 – XmlConfig[‘core-site.xml’] {‘group': ‘hadoop’, ‘conf_dir': ‘/etc/hadoop/conf’, ‘mode': 0644, ‘configuration_attributes': …, ‘owner': ‘hdfs’, ‘configurations': …}
2015-03-04 21:13:19,117 – Generating config: /etc/hadoop/conf/core-site.xml
2015-03-04 21:13:19,117 – File[‘/etc/hadoop/conf/core-site.xml’] {‘owner': ‘hdfs’, ‘content': InlineTemplate(…), ‘group': ‘hadoop’, ‘mode': 0644, ‘encoding': ‘UTF-8′}
2015-03-04 21:13:19,118 – Writing File[‘/etc/hadoop/conf/core-site.xml’] because contents don’t match
2015-03-04 21:13:19,119 – File[‘/etc/hadoop/conf/slaves’] {‘content': Template(‘slaves.j2′), ‘owner': ‘hdfs’}
2015-03-04 21:13:19,120 – Directory[‘/hadoop/hdfs/namenode’] {‘owner': ‘hdfs’, ‘group': ‘hadoop’, ‘recursive': True, ‘mode': 0755}
2015-03-04 21:13:19,120 – File[‘/var/lib/ambari-agent/data/tmp/checkForFormat.sh’] {‘content': StaticFile(‘checkForFormat.sh’), ‘mode': 0755}
2015-03-04 21:13:19,121 – Execute[‘/var/lib/ambari-agent/data/tmp/checkForFormat.sh hdfs /etc/hadoop/conf /usr/hdp/current/hadoop-client/bin /var/run/hadoop/hdfs/namenode/formatted/ /var/lib/hdfs/namenode/formatted/ /hadoop/hdfs/namenode’] {‘path': [‘/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin’], ‘not_if': ‘test -d /var/run/hadoop/hdfs/namenode/formatted/ || test -d /var/lib/hdfs/namenode/formatted/’}
2015-03-04 21:13:19,126 – Skipping Execute[‘/var/lib/ambari-agent/data/tmp/checkForFormat.sh hdfs /etc/hadoop/conf /usr/hdp/current/hadoop-client/bin /var/run/hadoop/hdfs/namenode/formatted/ /var/lib/hdfs/namenode/formatted/ /hadoop/hdfs/namenode’] due to not_if
2015-03-04 21:13:19,126 – Directory[‘/var/lib/hdfs/namenode/formatted/’] {‘recursive': True}
2015-03-04 21:13:19,127 – File[‘/etc/hadoop/conf/dfs.exclude’] {‘owner': ‘hdfs’, ‘content': Template(‘exclude_hosts_list.j2′), ‘group': ‘hadoop’}
2015-03-04 21:13:19,128 – Directory[‘/var/run/hadoop/hdfs’] {‘owner': ‘hdfs’, ‘recursive': True}
2015-03-04 21:13:19,128 – Directory[‘/var/log/hadoop/hdfs’] {‘owner': ‘hdfs’, ‘recursive': True}
2015-03-04 21:13:19,129 – File[‘/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid’] {‘action': [‘delete’], ‘not_if': ‘ls /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1 && ps cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1′}
2015-03-04 21:13:19,140 – Deleting File[‘/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid’]
2015-03-04 21:13:19,140 – Execute[‘ulimit -c unlimited; su -s /bin/bash – hdfs -c ‘export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /etc/hadoop/conf start namenode”] {‘not_if': ‘ls /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1 && ps cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid >/dev/null 2>&1′}
2015-03-04 21:13:23,193 – Error while executing command ‘start':
Traceback (most recent call last):
File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 123, in execute
method(env)
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/namenode.py”, line 46, in start
namenode(action=”start”)
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/hdfs_namenode.py”, line 45, in namenode
create_log_dir=True
File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS/package/scripts/utils.py”, line 102, in service
not_if=service_is_up
File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 148, in __init__
self.env.run()
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 149, in run
self.run_action(resource, action)
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 115, in run_action
provider_action()
File “/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py”, line 241, in action_run
raise ex
Fail: Execution of ‘ulimit -c unlimited; su -s /bin/bash – hdfs -c ‘export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh –config /etc/hadoop/conf start namenode” returned 1. stdin: is not a tty
starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-dev-hdp-0.stp.rbc.out