We were loading 900 million rows of data into hive with pig… it was failing with “no space on device” errors because it was writing temporary jar files to the local file system under “/tmp” … which has limited disk space in our environment
we resolved this issue by setting the java tmp dir to somewhere other than /tmp
export _JAVA_OPTIONS=-Djava.io.tmpdir=/new/tmp/dir
We then tried to load 1.8 billion rows of data, and now its failing again with “no space on device” errors because it appears to be writing “unjars” to the local “/tmp” directory
for example…
hadoop-unjar4911731471312030980
does anyone know how to change the location where hadoop puts these “unjars”
Thanks in advance !