scala - 将 scala 字符串转换为 RDD[seq[string]]

Question

 // 4 workers
  val sc = new SparkContext("local[4]", "naivebayes")

  // Load documents (one per line).
  val documents: RDD[Seq[String]] = sc.textFile("/tmp/test.txt").map(_.split(" ").toSeq)

  documents.zipWithIndex.foreach{
  case (e, i) =>
  val collectedResult = Tokenizer.tokenize(e.mkString)
  }

  val hashingTF = new HashingTF()
  //pass collectedResult instead of document
  val tf: RDD[Vector] = hashingTF.transform(documents)

  tf.cache()
  val idf = new IDF().fit(tf)
  val tfidf: RDD[Vector] = idf.transform(tf)

在上面的代码片段中，我想提取collectedResult以将其重用于hashingTF.transform，如何在tokenize函数的签名处实现

 def tokenize(content: String): Seq[String] = {
...
}

Answer 1

看起来你想要map而不是foreach. 我不明白你在用什么zipWithIndex，也不明白为什么你打电话split只是为了再次加入他们的线路mkString。

val lines: Rdd[String] = sc.textFile("/tmp/test.txt")
val tokenizedLines = lines.map(tokenize)
val hashes = tokenizedLines.map(hashingTF)
hashes.cache()
...