0

我已经为由于版本问题而失败的亚马逊 Deequ 编写了代码。在我的系统中，Spark 2.4.0 可用，任何人都可以建议使用哪个版本的 Deequ 和 Scala、fastxml 等兼容？我收到信息，例如检测到多个 Scala 版本。我正在使用 Deequ LATEST 或尝试使用其他一些最新版本。我得到了多个版本的 Scala，这可能是问题的主要原因。如果您可以提供正确的 pom 文件和所有兼容版本，请提供帮助。

以下是信息、错误和使用的 POM.xml ：

[INFO] --- scala-maven-plugin:3.1.3:compile (scala-compile-first) @ test-dq-framework ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.12
[WARNING]  com.twitter:chill_2.11:0.9.3 requires scala version: 2.11.12
[WARNING]  org.apache.spark:spark-core_2.11:2.4.0 requires scala version: 2.11.12
[WARNING]  org.json4s:json4s-jackson_2.11:3.5.3 requires scala version: 2.11.11

错误 ：

Caused by: java.lang.NoClassDefFoundError: scala/runtime/java8/JFunction1$mcZD$sp
... 13 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.java8.JFunction1$mcZD$sp
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

POM.xml：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>dq-test</artifactId>
  <groupId>com.package.name</groupId>
  <version>${revision}</version>
  <properties>
    <revision>0.1</revision>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.0</version>
      <exclusions>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.4.0</version>
      <exclusions>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.11.0</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.11.0</version>
    </dependency>
    <dependency>
      <groupId>com.amazon.deequ</groupId>
      <artifactId>deequ</artifactId>
      <version>1.1.0_spark-2.4-scala-2.11</version>
      <exclusions>
        <exclusion>
          <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.typesafe</groupId>
      <artifactId>config</artifactId>
      <version>1.2.1</version>
    </dependency>
    <dependency>
      <groupId>com.sun.mail</groupId>
      <artifactId>javax.mail</artifactId>
      <version>1.6.2</version>
    </dependency>
    <dependency>
      <groupId>com.sun.activation</groupId>
      <artifactId>javax.activation</artifactId>
      <version>1.2.0</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.1.3</version>
        <executions>
          <execution>
            <id>scala-compile-first</id>
            <phase>process-resources</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.3</version>
        <configuration>
          <skipMain>true</skipMain> <!-- skip compile -->
          <skip>true</skip> <!-- skip testCompile -->
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.7</version>
        <configuration>
          <skipTests>true</skipTests>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <mainClass>className</mainClass>
            </manifest>
          </archive>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>
4

1 回答 1

0

在 Maven 存储库中，您可以使用此版本的 Amazon Deequ 的编译依赖项：https ://mvnrepository.com/artifact/com.amazon.deequ/deequ/1.1.0_spark-2.4-scala-2.11

它显示 spark-core_2.11 版本 2.4.2、spark-sql_2.11 版本 2.4.2 和 scala-library 版本 2.11.10。

也许在您的 pom.xml 中调整 spark-core 版本并添加以下 scala-library 依赖项：

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.10</version>
    </dependency>
于 2021-08-28T03:49:12.090 回答