I ran into a very weird problem within a hadoop cluster (HDP 2.2) I setup in Amazon EC2 (3 data nodes + one name node + one secondary name node). Hue server runs on the main name node and hive server runs on the secondary name node. I was using Hue web interface to create table “mytable” in HCatalog using a CSV file loaded into HDFS. The table creation returned successfully without error. The table was created and displayed in the Hue web interface. However, when I tried to query the table, it returned 0 record. I went to the /app/hive/warehouse folder, I could see the table folder “mytable” was created, but the CSV file was never copied into that folder. I reproduced the same behavior using hive shell. If I manually uploaded the csv file to /app/hive/warehouse/mytable directory (/app/hive/warehouse is set by hive.metastore.warehouse.dir in the hive-site.xml), I can use hive query to retrieve data without issue. It kind of ruled out permission issue since the same hue user can upload file to that hdfs directory.
If I does the same operation in the HDP sandbox VM, everything works as expected. After the table creation, the /app/hive/warehouse/mytable folder contains the CSV file I imported into the table.
Any help is highly appreciated.