hello-
I am doing my first hadoop install using the RPMs. I am installing on an ec2 instance.
When I try to format the name node, I run this command:
/usr/lib/hadoop/bin/hadoop namenode -format
I get this error:
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bae00000, 671088640, 0) failed; error=’Cannot allocate memory’ (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 671088640 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid26789.log
While I could send the output of /tmp/hs_err_pid26789.log, I’m pretty sure the issue is that the JVM is failing to allocate 671 GB.
I have reviewed the document that I am following, and I can’t see where I went wrong. The heap sizes in /etc/hadoop/conf/hadoop-env.sh look fine (~512-1024).
Here is my output from hdp-configuration-utils.py:
[ec2-user@ip-172-31-3-170 scripts]$ ./hdp-configuration-utils.py -c1 -m1 -d1 -k True
Using cores=1 memory=1GB disks=1 hbase=True
Profile: cores=1 memory=2048MB reserved=0GB usableMem=-1GB disks=1
Num Container=3
Container Ram=640MB
Used Ram=1GB
Unused Ram=0GB
yarn.scheduler.minimum-allocation-mb=640
yarn.scheduler.maximum-allocation-mb=1920
yarn.nodemanager.resource.memory-mb=1920
mapreduce.map.memory.mb=640
mapreduce.map.java.opts=-Xmx512m
mapreduce.reduce.memory.mb=1280
mapreduce.reduce.java.opts=-Xmx1024m
yarn.app.mapreduce.am.resource.mb=640
yarn.app.mapreduce.am.command-opts=-Xmx512m
mapreduce.task.io.sort.mb=256
tez.am.resource.memory.mb=1280
tez.am.java.opts=-Xmx1024m
hive.tez.container.size=640
hive.tez.java.opts=-Xmx512m
hive.auto.convert.join.noconditionaltask.size=134217000
I can see that there are java memory settings for mapreduce, but I’m not getting anywhere near that. I’m just trying to format the namenode.
Any ideas?
Let me know your thoughts,
-DM
