2

运行 kafka consumer 时出现以下错误：

    ERROR receiver.BlockGenerator: Error in block pushing thread
    java.io.NotSerializableException: org.jnetpcap.packet.PcapPacket
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
at org.apache.spark.serializer.SerializationStream$class.writeAll(Serializer.scala:102)
at org.apache.spark.serializer.JavaSerializationStream.writeAll(JavaSerializer.scala:30)
at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:996)
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1005)
at org.apache.spark.storage.MemoryStore.putValues(MemoryStore.scala:79)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:663)
at org.apache.spark.storage.BlockManager.put(BlockManager.scala:574)

build.sbt 文件：

name := "testpacket"
version := "1.0"
scalaVersion := "2.10.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.0.2
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.0.2"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.0.2"
libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.0.1"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

错误的原因可能是什么？

4

1 回答 1

0

我之前在两种情况下遇到过这个问题，所以在没有看到你的代码的情况下，我无法确定到底是什么问题。

  1. 您在 RDD 数据集中包含不可序列化的类。
  2. 您正在仅在驱动程序类/函数/代码之外导入不可序列化的类。

我的猜测是您正在经历 #1 并且将 PcapPacket 作为您的 RDD 的一部分。如果是这种情况，那么您将需要创建 PcapPacket 的可序列化版本，这应该不难，因为底层字节数组支持 PcapPacket。

于 2014-12-28T15:58:29.797 回答