0

My problem is somewhat as described here. Part of Code (actually took from apache site) is as below

val httpHosts = new java.util.ArrayList[HttpHost]
httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"))
httpHosts.add(new HttpHost("10.2.3.1", 9200, "http"))

val esSinkBuilder = new ElasticsearchSink.Builder[String](
  httpHosts,
  new ElasticsearchSinkFunction[String] {
    def createIndexRequest(element: String): IndexRequest = {
      val json = new java.util.HashMap[String, String]
      json.put("data", element)


      return Requests.indexRequest()
              .index("my-index")
              .`type`("my-type")
              .source(json)

If I add these three statements, I am getting error as below

import org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction
import org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer
import org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink

Error I am getting

object elasticsearch is not a member of package org.apache.flink.streaming.connectors
object elasticsearch6 is not a member of package org.apache.flink.streaming.connectors

If I do not add those import statements, I get error as below

Compiling 1 Scala source to E:\sar\scala\practice\readstbdata\target\scala-2.11\classes ...
[error] E:\sar\scala\practice\readstbdata\src\main\scala\example\readcsv.scala:35:25: not found: value ElasticsearchSink
[error] val esSinkBuilder = new ElasticsearchSink.Builder[String](
[error]                         ^
[error] E:\sar\scala\practice\readstbdata\src\main\scala\example\readcsv.scala:37:7: not found: type ElasticsearchSinkFunction
[error]   new ElasticsearchSinkFunction[String] {
[error]       ^
[error] two errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 1 s, completed 10 Feb, 2020 2:15:04 PM

Stackflow question I referred above, some function has been extended. My understanding is, flink.streaming.connectors.elasticsearch have to be extended into REST libraries. 1) Is my understanding correct 2) if Yes, can I have complete extensions 3)If my understanding is wrong, please give me a solution.

Note: I added the following statements in build.sbt

libraryDependencies += "org.elasticsearch.client" % "elasticsearch-rest-high-level-client" % "7.5.2" ,
    libraryDependencies += "org.elasticsearch" % "elasticsearch" % "7.5.2",
4

2 回答 2

0

流连接器不是 flink 二进制分发的一部分。您必须将它们与您的应用程序打包在一起。

因为elasticsearch6你需要添加flink-connector-elasticsearch6_2.11，你可以这样做

libraryDependencies += "org.apache.flink" %% "flink-connector-elasticsearch6" % "1.6.0"

一旦这个 jar 成为您构建的一部分，编译器就会找到缺少的组件。但是，我不知道这个 ES6 客户端是否适用于 7.5.2 版本。

于 2020-02-10T09:07:29.710 回答
0

Flink Elasticsearch 连接器 7

请查看我在 此处提供的工作和详细答案，它是用 Scala 编写的。

于 2020-05-08T16:41:12.817 回答