0

嗨，我正在尝试在 SBT 中设置一个小型 Spark 应用程序，

我的 build.sbt 是

import Dependencies._

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)

libraryDependencies += scalaTest % Test

一切正常，我得到了 SBT 解决的所有依赖项，但是当我尝试在我的 hello.scala 项目文件中导入 spark 时，我得到了这个错误 ：value spark

我的 hello.scala 文件是

package example
import org.apache.spark._
import org.apache.spark.SparkContext._

object Hello extends fileImport with App {
  println(greeting)
  anime.select("*").orderBy($"rating".desc).limit(10).show()
}

trait fileImport {
  lazy val greeting: String = "hello"
  var anime = spark.read.option("header", true).csv("C:/anime.csv")
  var ratings = spark.read.option("header", true).csv("C:/rating.csv")
}

这是我得到的错误文件

[info] Compiling 1 Scala source to C:\Users\haftab\Downloads\sbt-0.13.16\sbt\alfutaim\target\scala-2.11\classes...
[error] C:\Users\haftab\Downloads\sbt-0.13.16\sbt\alfutaim\src\main\scala\example\Hello.scala:12: not found: value spark
[error]   var anime = spark.read.option("header", true).csv("C:/anime.csv")
[error]               ^
[error] C:\Users\haftab\Downloads\sbt-0.13.16\sbt\alfutaim\src\main\scala\example\Hello.scala:13: not found: value spark
[error]   var ratings = spark.read.option("header", true).csv("C:/rating.csv")
[error]                 ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 3 s, completed Sep 10, 2017 1:44:47 PM
4

1 回答 1

2

sparkspark-shell仅在初始化

但是对于代码你需要自己初始化spark变量

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("testings").master("local").getOrCreate

您可以将testings名称更改为您想要的名称.master选项是可选的，如果您想使用运行代码spark-submit

于 2017-09-10T09:12:38.073 回答