Hi there,
Is Spark Streaming integrated in HDP 2.2 ? Do you think that Spark Streaming is production ready ?
Spark looks good, but Spark Streaming is quite young.
Do you know production cluster using Spark Streaming ?
Cheers
Romain
Spark Streaming production ready ?
↧
↧
Reply To: Cannot connect to 127.0.0.1:8888
Hello,
I have imported VM but web UI is not working even
in sandbox I am not able to connect
ssh root@127.0.0.1 -p 2222;
getting connection refused error please suggest.
I checked IP of VM its correct from machine I pinged to 127.0.0.1 and I am getting response.
I am able to connect to 127.0.0.1:8888 but 127.0.0.1:8000 for web UI is showing not found
I read almost all forums topic but those solution r not working for me
↧
Upgrade Ambari to 1.7
Is there a document for upgrading Amabri from 1.6.1 to 1.7 on Linux
↧
HDP 2.2 Preview Kafka Tutorial
Hi all,
I’m following the Kafka tutorial and failing at the step:
java -cp target/Tutorial-1.0-SNAPSHOT.jar com.hortonworks.tutorials.tutorial1.TruckEventsProducer localhost:9092 localhost:2181 &
I’ve made sure that zookeeper is running, the truckevent topic is active, and maven compiled a clean package. Console producer works just fine, but the TruckEventsProducer doesn’t work and I can’t figure out why.
↧
Reply To: Hadoop installation path
Well, I am also a newbie here. But I guess its installed at /usr/hdp/current
↧
↧
HDP2.2- Mapreduce pointing to local file system instead of HDFS
Env – HDP2.2 installed from ambari-server.
On HDP2.2 Sanbox it is working fine and not on hdp2.2 cluster which is installed from ambari-server which is bizarre
.
When trying to read a hdfs file inside a mapreduce program it is pointing to local file system(file:///) though value of fs.defaultFS in /etc/hadoop/conf/core-site.xml is pointing to hdfs.
Also when tried to read hdfs file just before the mapper is created (means inside main method itself), for this scenario it is pointing to hdfs and as execution starts it points to local file system and so getting file not found exception.
Container exited with a non-zero exit code 143
14/12/11 11:34:40 INFO mapreduce.Job: Task Id : attempt_1418323359240_0008_m_000000_1, Status : FAILED
Error: java.io.FileNotFoundException: File /user/root/vsconfig.properties does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
↧
Reply To: HDP2.2- Mapreduce pointing to local file system instead of HDFS
Hi.. facing similar issue… Need a fix on this.
↧
Man pages in Sandbox
Please do not remove man pages from Hortonworks Sandbox in future.
How much space does that really save vs the inconvenience it causes?
↧
Reply To: How do you get Spark running in HDP 2.2 Preview?
The Spark 1.1 Tech Preview won’t work on HDP 2.2 Sandbox. We plan to release Spark 1.2 based Tech preview when Spark 1.2 become available shortly.
↧
↧
Reply To: Error: HDP 2.1 Installation nagios-plugins-1.4.9-1.x86_64 failed
Can anyone help on this issue ? thanks in adv.
↧
Sandbox Hive Connectivity Issue
Hello, I have painfully looked through several threads about issues connecting to Hive in the sandbox (Virtual Box) and still cannot get anywhere. Trying to connect to Hive from both the command line via beeline or client Java application hangs and the process never stops. I am aware that a port forwarding rule needs to be made in virtual box (which I’ve done) and have totally run out of things to troubleshoot and appreciate some help. At the very least I should be able to connect via beeline but it simply freezes on this:
!connect jdbc:hive2://localhost:10000 root hadoop org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://localhost:10000
Does anyone know what else I need to do to connect to Hive via JDBC out of the box besides creating a port forwarding rule? Is there anything else I might be missing? Thanks for any help.
↧
Reply To: Spark 1.2 in HDP 2.2
Spark 1.1 is available as a Tech preview and will work with HDP 2.1. Apache Spark community is voting on releasing Spark 1.2 right now and we plan to revise our tech preview to Spark 1.2 on HDP 2.2 shortly after that.
↧
Unable to import incremental data from Oracle using SQOOP with timestamp
Hi ,
We have been trying to import the data from Oracle to HDFS using SQOOP. The column that we are trying to use for incremental import is a timestamp.
SQOOP is throwing a Java exception as follows:
Error: java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.sql.SQLDataException: ORA-01841: (full) year must be between -4713 and +9999, and not be 0
Here is the SQOOP command that is used to perform incremental import:
sqoop import –connect ‘jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=xxxxxxxx)(port=1521))(connect_data=(SID=YYYY)))’ –username AAAA –password ZZZZ –table TABLE123 –target-dir /user/hive/incremental_table1 -m 1 –check-column xyz_TS –incremental lastmodified –last-value {last_import_date}
Your response is highly appreciated.
↧
↧
spark 1.2 + Avro does not work
have recently installed HDP2.2(Depends on hadoop 2.6).
My spark 1.2 is built with hadoop 2.3 profile.
( mvn -Pyarn -Dhadoop.version=2.6.0 -Dyarn.version=2.6.0 -Phadoop-2.3 -Phive -DskipTests clean package)
My program has following dependencies
val avro = “org.apache.avro” % “avro-mapred” %”1.7.7″
val spark = “org.apache.spark” % “spark-core_2.10″ % “1.2.0” % “provided”
My program to read avro files fails with the following error. What am I doing wrong?
Thanks
Manas
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:107)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Manas Kar
↧
Reply To: Confirming and registering hosts fails
I am having a problem with Ambari.
I am trying to use Ambari to install Hadoop to a three node cluster. the name node is where the Ambari server is located. I am getting this error:
ERROR 2014-12-12 17:39:56,963 main.py:137 – Ambari agent machine hostname (localhost.localdomain) does not match expected ambari server hostname (namenode). Aborting registration. Please check hostname, hostname -f and /etc/hosts file to confirm your hostname is setup correctly
‘, None)
Here is the contents of my hosts file:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 localhost.namenode namenode
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
xxx.xxx.200.144 localhost.datanode10
xxx.xxx.200.107 localhost.datanode01
xxx.xxx.200.143 localhost.namenode namenode
I am not sure what the problem is. Since there are only four steps to run ambari there is not a lot of background to determine the cause of this problem.
↧
Reply To: Installation issue sandbox 2.1
I have exactly the same issue. Where you able to fix it?
↧
HDP 2.2 Preview Error "Device already attached to Port "1
I am using an Oracle VM VirtualBox (Version 4.3.20) and I am trying to Import the HDP 2.2 Preview VirtualBox from an ova file downloaded from this site.
It runs until the last few seconds then gives me an error that the Port is already in use: “Device is already attached to port 1, device 0….. ”
I have successfully installed other HortonWorks Appliances (Sandbox 2.0) and Ubunto to this same VM with no issues.
I have deleted the existing VMDK in the Virtual Media Manager but that doesn’t seem to help.
I am very new to VirtualBoxes and trying to learn about Hadoop and Big Data, so any help would be useful.
↧
↧
Reply To: Sandbox – Pig Basic Tutorial example is nbot working
Hello, I’m new in this helldoop error 1070 using pig.
1. I’ve downloaded the HDP 2.2
2. I’ve modified the file /usr/bin/pig as Larry says (no luck)
3. I’ve tried to run the sed command sed -i ’49s/.*/includeHCatalog=true;/’ /usr/lib/pig/bin/pig (Error: can’t read /usr/lib/pig/bin/pig: No such file or directory)
4. I added to the pig arguments section the sentence “-useHCatalog” (same horrible 1070 error)
5. I’ve googled with no luck.
I know that this should be super easy but It is not working.
Does anybody know how to solve this error? Your help will be much appreciated!!!
↧
How to connect to Internet inside the sandbox?
Hi Experts
i currently need to access to Internet inside cluster – but it seems that current i don’t have access to Internet in HDP sandbox.
I am using HDP 2.2 in virtualbox. My host environment is Windows. i just imported the appliance, without any changes..
how could i change the configuration to access internet from virtualbox? for example, use wget/yum to download packages?
↧
Host Tab displays wrong IP addresses
Hi,
We have an 4 node development cluster configured with Ambari and HDP 2.2. All our servers have two network interfaces eth0 the backup network and eth1 the service network. Unfortunately is the service interface eth1 and Ambari displays on the Host tab the IPs from eth0 interface.
Is it possible to decide that Ambari should ignore the eth0 interface?
We tried to update the columns IPv4 and IPv6 in Ambari repository, but the displayed IP addresses were not modified. After the next reboot our changes were overwritten.
Regards
↧
More Pages to Explore .....
