2

我正在尝试将一个 126 Gb 表从 HAWQ（PostgreSQL，在本例中为 8.2）中提取到 Spark 中，但它无法正常工作。我可以拉小桌子没问题。对于这个我不断收到错误：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): ExecutorLostFailure (executor driver lost)
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

我的集群规格如下： 64 核，512 Gb RAM，2 个节点
这是 2 个节点上的 Spark 独立集群（相信我，我想要更多节点，但这就是我得到的全部）。所以我有一个节点作为纯从属节点，另一个节点包含主节点和另一个从节点。

我已经使用 spark-submit 作业尝试了许多内存分配配置，我将在这里列出一些，但没有一个有效：

    // CONFIG_5: FAIL (96 Gb driver  144 Gb executor)
    --driver-memory 96g --executor-memory 6g --num-executors 24 --executor-cores 24 

    // CONFIG_4: FAIL (48 Gb driver  196 Gb executor)
    --driver-memory 48g --executor-memory 8g --num-executors 24 --executor-cores 24  

    //CONFIG_3: FAIL (120 Gb driver  128 Gb executor)
    --driver-memory 120g --executor-memory 4g --num-executors 32 --executor-cores 32   

    // CONFIG_2: FAIL (156 driver  96 executor)
    --driver-memory 156g --executor-memory 4g --num-executors 24 --executor-cores 24   

    // CONFIG_1: FAIL (224 Gb driver  48 Gb executor)
    --driver-memory 224g --executor-memory 1g --num-executors 1 --executor-cores 48

每次的错误都是一样的——ExecutorLostFailure(executor driver lost)

4

0 回答 0