I have this table: (executed in Hive, using the default schema.)
CREATE TABLE logs_parquet
(
unixTimestamp String,
one String,
two String,
three String
)
PARTITIONED BY (create_date date)
STORED AS PARQUET;
How do I insert a row there via Spark, if the source is RDD (lines) from Spark streaming:
case class LogFile(unixTimestamp: String, one: String, two: String, three: String)
val lines = kafkaStream.map(_._2)
lines.foreachRDD(
rdd => {
// get particular columns only
val logs = rdd.map(_.split(“,”)).map(log => LogFile(log(0), log(2), log(3), log(4)))
logs.registerTempTable(“logFile”)
// store in parquet via hive
/*if (rdd.count() > 0) {
hc.sql(“insert overwrite table logs_parquet PARTITION(create_Date=’2015-02-24′) select * from logFile”)
}*/
}
)
I tried to register the RDD as temp table (logFile) and do a select, but it throws an exception when I run the Spark job.
I can do a sqlContext.sql(“select * from logFile”) but not hiveContext.sql
It throws this error:
ERROR metadata.Hive: NoSuchObjectException(message:default.logfile table not found) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29338) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29306) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:29237) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1036) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1022) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89) at com.sun.proxy.$Proxy12.getTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)