[转载]基于activemq+zookeeper+kafka+spark&nbsp

SEVERE: ID:localhost.localdomain-43416-1404369427202-0:0:1:1
Exception while processing message:
ID:localhost.localdomain-53089-1404363180895-0:7315:1:1:1
kafka.common.FailedToSendMessageException: Failed to send messages
after 3 tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:38)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQMessageConsumer.iterate(ActiveMQMessageConsumer.java:1362)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:194)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--KAFKA的服务器已经宕机了，导致这个错，需要将kafka的master拉起来。

[ocdc@oc-datanode46 bin]$ Exception in thread "ActiveMQ Session
Task" java.lang.NoClassDefFoundError:
kafka/producer/ProducerConfig
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:28)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:134)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:205)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--缺失apache-activemq的jar包

六月 24, 2014 6:20:55 下午 org.apache.activemq.ActiveMQMessageConsumer
dispatch
严重: ID:oc-datanode46-56108-1403605203024-0:0:1:1 Exception while
processing message: ID:mq-1-43282-1403517046176-3:1:1:1:1
java.lang.IllegalArgumentException: requirement failed: Missing
required property 'metadata.broker.list'
at scala.Predef$.require(Predef.scala:233)
at
kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
at kafka.producer.ProducerConfig.(ProducerConfig.scala:66)
at kafka.producer.ProducerConfig.(ProducerConfig.scala:56)
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:28)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQMessageConsumer.iterate(ActiveMQMessageConsumer.java:1362)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:194)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--activemq服务没有起起来。无法获取到接口数据

Exception in thread "ActiveMQ Session Task"
java.lang.NoClassDefFoundError: com/yammer/metrics/Metrics
at
kafka.metrics.KafkaMetricsGroup$class.newMeter(KafkaMetricsGroup.scala:46)
at
kafka.producer.ProducerStats.newMeter(ProducerStats.scala:23)
at kafka.producer.ProducerStats.(ProducerStats.scala:24)
at
kafka.producer.ProducerStatsRegistry$$anonfun$1.apply(ProducerStats.scala:33)
at
kafka.producer.ProducerStatsRegistry$$anonfun$1.apply(ProducerStats.scala:33)
at kafka.utils.Pool.getAndMaybePut(Pool.scala:61)
at
kafka.producer.ProducerStatsRegistry$.getProducerStats(ProducerStats.scala:37)
at
kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:48)
at kafka.producer.Producer.(Producer.scala:59)
at kafka.javaapi.producer.Producer.(Producer.scala:26)
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:30)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:134)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:205)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException:
com.yammer.metrics.Metrics
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
--------------------------------------------------------------------------------------

Exception in thread "ActiveMQ Session Task"
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at
com.yammer.metrics.reporting.JmxReporter.(JmxReporter.java:19)
at com.yammer.metrics.Metrics.(Metrics.java:20)
at
kafka.metrics.KafkaMetricsGroup$class.newMeter(KafkaMetricsGroup.scala:46)
at
kafka.producer.ProducerStats.newMeter(ProducerStats.scala:23)
at kafka.producer.ProducerStats.(ProducerStats.scala:24)
at
kafka.producer.ProducerStatsRegistry$$anonfun$1.apply(ProducerStats.scala:33)
at
kafka.producer.ProducerStatsRegistry$$anonfun$1.apply(ProducerStats.scala:33)
at kafka.utils.Pool.getAndMaybePut(Pool.scala:61)
at
kafka.producer.ProducerStatsRegistry$.getProducerStats(ProducerStats.scala:37)
at
kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:48)
at kafka.producer.Producer.(Producer.scala:59)
at kafka.javaapi.producer.Producer.(Producer.scala:26)
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:32)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:134)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:205)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException:
org.slf4j.LoggerFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 22 more

需要配置kafka的server.配置文件
修改hostname=必须是本地IP，使用名称有问题。
----------------------------------------------------------------------------------
六月 26, 2014 10:39:31 上午 org.apache.activemq.ActiveMQMessageConsumer
dispatch
严重: ID:oc-datanode46-40825-1403750326305-0:0:1:1 Exception while
processing message: ID:mq-1-54214-1403662139893-3:1:1:1:1
kafka.common.FailedToSendMessageException: Failed to send messages
after 3 tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at
com.asiainfo.ssyx.service.MsgExcuteService.handTopicMessage(MsgExcuteService.java:35)
at
com.asiainfo.ssyx.jms.TopicMsgListener$1.onMessage(TopicMsgListener.java:45)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1229)
at
org.apache.activemq.ActiveMQMessageConsumer.iterate(ActiveMQMessageConsumer.java:1362)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:194)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.streaming.kafka.KafkaUtils$.createStream(Lorg/apache/spark/streaming/StreamingContext;Ljava/lang/String;Ljava/lang/String;Lscala/collection/immutable/Map;Lorg/apache/spark/storage/StorageLevel;)Lorg/apache/spark/streaming/dstream/ReceiverInputDStream;
at
org.apache.spark.streaming.ssyx.OnlineMarket$.main(OnlineMarket.scala:38)
at
org.apache.spark.streaming.ssyx.OnlineMarket.main(OnlineMarket.scala)

Exception in thread "Thread-33" org.apache.spark.SparkException:
Job aborted: Spark cluster looks down
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

spark服务启动有错误导致这样的问题

14/07/04 11:31:02 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:50439:
java.net.BindException: 地址已在使用
java.net.BindException: 地址已在使用
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:286)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at
org.apache.spark.ui.JettyUtils$$anonfun$1.apply$mcV$sp(JettyUtils.scala:118)
at
org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
at
org.apache.spark.ui.JettyUtils$$anonfun$1.apply(JettyUtils.scala:118)
at scala.util.Try$.apply(Try.scala:161)
at
org.apache.spark.ui.JettyUtils$.connect$1(JettyUtils.scala:118)
at
org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:129)
at
org.apache.spark.deploy.master.ui.MasterWebUI.start(MasterWebUI.scala:48)
at
org.apache.spark.deploy.master.Master.preStart(Master.scala:110)
at akka.actor.ActorCell.create(ActorCell.scala:562)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
at
akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
at akka.dispatch.Mailbox.run(Mailbox.scala:218)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/07/04 11:31:02 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@22bbe161:
java.net.BindException: 地址已在使用

----------------------------------------------------------------------
Column family f1 does not exist in

[2014-07-09 15:26:44,681] ERROR [KafkaApi-0] Error while fetching
metadata for partition [com.ailk.flowmanagement.mcd.flux_action,1]
(kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for
partition [com.ailk.flowmanagement.mcd.flux_action,1]
at
kafka.server.KafkaApis$$anonfun$20$$anonfun$23.apply(KafkaApis.scala:586)
at
kafka.server.KafkaApis$$anonfun$20$$anonfun$23.apply(KafkaApis.scala:574)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractTraversable.map(Traversable.scala:105)
at
kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:574)
at
kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:569)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:569)
at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:662)
[2014-07-09 15:26:44,683] INFO Closing socket connection to
/10.95.66.88. (kafka.network.Processor)

at
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:44)
at com.asiainfo.ssyx.service.SendSMS$.main(SendSMS.scala:30)
at com.asiainfo.ssyx.service.SendSMS.main(SendSMS.scala)
调用zookeeper的配置有错误，需要检查zookeeper配置是否正确。