2

是否可以使用 Nifi 将 json 文件加载到结构化表中?

我调用了以下天气预报数据(来自 6000 个气象站),我目前正在将其加载到 HDFS 中。这一切都出现在一行中:

{"SiteRep":{"Wx":{"Param":[{"name":"F","units":"C","$":"Feels Like Temperature"},{"name":"G","units":"mph","$":"Wind Gust"},{"name":"H","units":"%","$":"Screen Relative Humidity"},{"name":"T","units":"C","$":"Temperature"},{"name":"V","units":"","$":"Visibility"},{"name":"D","units":"compass","$":"Wind Direction"},{"name":"S","units":"mph","$":"Wind Speed"},{"name":"U","units":"","$":"Max UV Index"},{"name":"W","units":"","$":"Weather Type"},{"name":"Pp","units":"%","$":"Precipitation Probability"}]},"DV":{"dataDate":"2017-01-12T22:00:00Z","type":"Forecast","Location":[{"i":"14","lat":"54.9375","lon":"-2.8092","name":"CARLISLE AIRPORT","country":"ENGLAND","continent":"EUROPE","elevation":"50.0","Period":{"type":"Day","value":"2017-01-13Z","Rep":{"D":"WNW","F":"-3","G":"25","H":"67","Pp":"0","S":"13","T":"2","V":"EX","W":"1","U":"1","$":"720"}}},{"i":"22","lat":"53.5797","lon":"-0.3472","name":"HUMBERSIDE AIRPORT","country":"ENGLAND","continent":"EUROPE","elevation":"24.0","Period":{"type":"Day","value":"2017-01-13Z","Rep":{"D":"NW","F":"-2","G":"43","H":"63","Pp":"3","S":"25","T":"4","V":"EX","W":"3","U":"1","$":"720"}}}, .....

理想情况下,我希望将架构构造成一个 6000 行的表。

我尝试编写一个模式将上述内容传递给 Pig,但没有成功,可能是因为我对 json 不够熟悉,无法正确翻译。

寻找一种向数据添加一些结构的简单方法,我发现 Nifi 中有一个 PutHBaseJson 处理器。

谁能建议这个 PutHBaseJson 处理器是否适用于上述数据结构?如果是这样,任何人都可以向我指出一个体面的教程,给我一个配置的起点吗?

非常感谢任何指导。

4

1 回答 1

3

您可能希望使用SplitJson处理器将 6000 条记录 JSON 结构拆分为 6000 个单独的流文件。如果您需要从顶级响应“注入”参数定义,您可以执行ReplaceTextorJoltTransformJSON操作来操作各个 JSON 记录。这是Yolanda Davis 的一篇好文章,描述了如何在 NiFi 中执行 Jolt 转换 (JSON -> JSON)。

拥有包含单个 JSON 记录的单个流文件后,将它们放入 HBase 非常容易。Bryan Bende 写了一篇文章,描述了处理器的必要配置。PutHBaseJson

于 2017-01-13T01:31:26.270 回答