数据湖之Hudi：基于Spark引擎使用Hudi的DeltaStreamer组件工具

主要记录操作，操作内容参考尚硅谷Hudi公开资料以及Hudi官方文档

具体参看官方文档：https://hudi.apache.org/docs/hoodie_deltastreamer/

文章目录

DeltaStreamer工具介绍
命令
测试

DeltaStreamer工具介绍

HoodieDeltaStreamer工具 (hudi-utilities-bundle中的一部分) 提供了从DFS或Kafka等不同来源进行摄取的方式，并具有以下功能：

精准一次从Kafka采集新数据，从Sqoop、HiveIncrementalPuller的输出或DFS文件夹下的文件增量导入
导入的数据支持json、avro或自定义数据类型
管理检查点，回滚和恢复
利用 DFS 或 Confluent schema registry的 Avro Schema
支持自定义转换操作

命令

利用spark提交的命令如下

spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer /opt/software/hudi-0.12.1/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.12.1.jar --help

Schema Provider和Source配置项：https://hudi.apache.org/docs/hoodie_deltastreamer

如下以File Based Schema Provider和JsonKafkaSource为例演示

测试

启动kafka集群，准备数据

启动kafka集群，创建topic

/opt/module/kafka/bin/kafka-topics.sh --bootstrap-server m2:9092 --create --topic hudi_test

java生产者代码往topic发送测试数据

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.7.2</version>
        </dependency>

        <!--fastjson <= 1.2.80 存在安全漏洞,-->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.83</version>
        </dependency>

import com.alibaba.fastjson.JSONObject;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

import java.util.Properties;
import java.util.Random;

public class TestKafkaProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "192.168.2.102:9092");
        props.put("acks", "-1");
        props.put("batch.size", "1048576");
        props.put("linger.ms", "5");
        props.put("compression.type", "snappy");
        props.put("buffer.memory", "33554432");
        props.put("key.serializer",
                "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer",
                "org.apache.kafka.common.serialization.StringSerializer");
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
        Random random = new Random();
        for (int i = 0; i < 1000; i++) {
            JSONObject model = new JSONObject();
            model.put("userid", i);
            model.put("username", "name" + i);
            model.put("age", 18);
            model.put("partition", random.nextInt(100));
            producer.send(new ProducerRecord<String, String>("hudi_test", model.toJSONString()));
        }
        producer.flush();
        producer.close();
    }
}

准备配置文件

定义arvo所需schema文件（包括source和target）

mkdir /opt/test/hudi-props/
vim /opt/test/hudi-props/source-schema-json.avsc

{        
  "type": "record",
  "name": "Profiles",   
  "fields": [
    {
      "name": "userid",
      "type": [ "null", "string" ],
      "default": null
    },
    {
      "name": "username",
      "type": [ "null", "string" ],
      "default": null
    },
    {
      "name": "age",
      "type": [ "null", "string" ],
      "default": null
    },
    {
      "name": "partition",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

cp source-schema-json.avsc target-schema-json.avsc

拷贝hudi配置base.properties

cp /opt/software/hudi-0.12.1/hudi-utilities/src/test/resources/delta-streamer-config/base.properties /opt/test/hudi-props/

根据源码里提供的模板，编写自己的kafka source的配置文件

cp /opt/software/hudi-0.12.1/hudi-utilities/src/test/resources/delta-streamer-config/kafka-source.properties /opt/test/hudi-props/

vim /opt/test/hudi-props/kafka-source.properties

我这边编写好的kafka source 配置文件如下

如下配置文件我都是在本地，实际上一般都是放在hdfs上

如果放到HDFS上，可以把hudi-props目录下文件都put上去：hadoop fs -put /opt/module/hudi-props/ /

然后例如/opt/test/hudi-props/source-schema-json.avsc改成

hdfs://m1:8020/hudi-props/source-schema-json.avsc

###
include=/opt/test/hudi-props/base.properties
# Key fields, for kafka example
hoodie.datasource.write.recordkey.field=userid
hoodie.datasource.write.partitionpath.field=partition
# schema provider configs
# hoodie.deltastreamer.schemaprovider.registry.url=http://localhost:8081/subjects/impressions-value/versions/latest
hoodie.deltastreamer.schemaprovider.source.schema.file=/opt/test/hudi-props/source-schema-json.avsc
hoodie.deltastreamer.schemaprovider.target.schema.file=/opt/test/hudi-props/target-schema-json.avsc
# Kafka Source
#hoodie.deltastreamer.source.kafka.topic=uber_trips
hoodie.deltastreamer.source.kafka.topic=hudi_test
#Kafka props
bootstrap.servers=m2:9092
auto.offset.reset=earliest
group.id=test-group
schema.registry.url=http://localhost:8081

拷贝所需hudi的jar包到Spark

需要把之前打包打好的hudi-utilities-bundle_2.12-0.12.1.jar放入spark的jars路径下，否则报错找不到一些类和方法。

cp /opt/software/hudi-0.12.1/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.12.1.jar /opt/module/spark-3.2.2/jars/

运行导入命令

spark-submit \
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer  \
/opt/module/spark-3.2.2/jars/hudi-utilities-bundle_2.12-0.12.1.jar \
--props /opt/test/hudi-props/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider  \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource  \
--source-ordering-field userid \
--target-base-path hdfs://m1:8020/tmp/hudi/hudi_test  \
--target-table hudi_test \
--op BULK_INSERT \
--table-type MERGE_ON_READ

查看导入结果

启动spark-sql

spark-sql \
  --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
  --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
  --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'

指定location创建hudi表

use spark_hudi;

create table hudi_test using hudi
location 'hdfs://m1:8020/tmp/hudi/hudi_test'

查询hudi表
```
select * from hudi_test;
```