在分布式消息系统中，Kafka以其高吞吐量和可靠性而广受欢迎。本文将深入探讨Kafka消费者如何使用commitSync()方法进行同步消息提交，并结合实际代码示例，展示这一过程的详细步骤。
Kafka消费者同步提交机制
Kafka的KafkaConsumer类提供了多种提交偏移量的方法，其中commitSync()方法用于同步提交。该方法会阻塞直到以下情况之一发生：
- 提交成功；
- 遇到不可恢复的错误（此时会抛出异常）；
- 超时（由
default.api.timeout.ms指定，超时会抛出TimeoutException）。
同步提交的不同形式
commitSync()方法有以下几种形式：
- 无参数的
commitSync()，提交当前分区的偏移量。 - 带超时参数的
commitSync(Duration timeout)，允许指定超时时间。 - 带偏移量参数的
commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets)，提交指定的偏移量。 - 带偏移量和超时参数的
commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets, final Duration timeout)。
实例代码分析
Kafka配置类
首先，我们定义一个配置类ExampleConfig，用于配置Kafka生产者和消费者的属性。
package com.logicbig.example;
import java.util.Properties;
public class ExampleConfig {
public static final String BROKERS = "localhost:9092";
public static Properties getProducerProps() {
Properties props = new Properties();
props.put("bootstrap.servers", BROKERS);
props.put("acks", "all");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
return props;
}
public static Properties getConsumerProps() {
Properties props = new Properties();
props.setProperty("bootstrap.servers", BROKERS);
props.setProperty("group.id", "testGroup");
props.setProperty("enable.auto.commit", "false");
props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
return props;
}
}
创建Kafka主题
接下来，我们创建一个工具类TopicCreator用于创建Kafka主题。
package com.logicbig.example;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import java.util.Collections;
import java.util.Properties;
import java.util.stream.Collectors;
public class TopicCreator {
public static void main(String[] args) throws Exception {
createTopic("example-topic-2020-5-29", 1);
}
private static void createTopic(String topicName, int numPartitions) throws Exception {
Properties config = new Properties();
config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, ExampleConfig.BROKERS);
AdminClient admin = AdminClient.create(config);
boolean alreadyExists = admin.listTopics().names().get().stream()
.anyMatch(existingTopicName -> existingTopicName.equals(topicName));
if (alreadyExists) {
System.out.printf("topic already exits: %s%n", topicName);
} else {
System.out.printf("creating topic: %s%n", topicName);
NewTopic newTopic = new NewTopic(topicName, numPartitions, (short) 1);
admin.createTopics(Collections.singleton(newTopic)).all().get();
}
admin.describeTopics(Collections.singleton(topicName)).all().get()
.forEach((topic, desc) -> {
System.out.println("Topic: " + topic);
System.out.printf("Partitions: %s, partition ids: %s%n", desc.partitions().size(),
desc.partitions()
.stream()
.map(p -> Integer.toString(p.partition()))
.collect(Collectors.joining(",")));
});
admin.close();
}
}
使用commitSync()方法
最后，我们通过CommitSyncExample类展示如何使用commitSync()方法同步提交消息。
package com.logicbig.example;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import java.time.Duration;
import java.util.*;
public class CommitSyncExample {
private static String TOPIC_NAME = "example-topic-2020-5-29";
private static KafkaConsumer<String, String> consumer;
private static TopicPartition topicPartition;
public static void main(String[] args) throws Exception {
Properties consumerProps = ExampleConfig.getConsumerProps();
consumer = new KafkaConsumer<>(consumerProps);
topicPartition = new TopicPartition(TOPIC_NAME, 0);
consumer.assign(Collections.singleton(topicPartition));
printOffsets("before consumer loop", consumer, topicPartition);
sendMessages();
startConsumer();
}
private static void startConsumer() {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("consumed: key = %s, value = %s, partition id= %s, offset = %s%n",
record.key(), record.value(), record.partition(), record.offset());
}
if (records.isEmpty()) {
System.out.println("-- terminating consumer --");
break;
}
printOffsets("before commitAsync() call", consumer, topicPartition);
consumer.commitSync();
printOffsets("after commitAsync() call", consumer, topicPartition);
}
printOffsets("after consumer loop", consumer, topicPartition);
}
private static void printOffsets(String message, KafkaConsumer<String, String> consumer,
TopicPartition topicPartition) {
Map<TopicPartition, OffsetAndMetadata> committed = consumer
.committed(new HashSet<>(Arrays.asList(topicPartition)));
OffsetAndMetadata offsetAndMetadata = committed.get(topicPartition);
long position = consumer.position(topicPartition);
System.out
.printf("Offset info %s, Committed: %s, current position %s%n", message, offsetAndMetadata == null ? null :
offsetAndMetadata
.offset(), position);
}
private static void sendMessages() {
Properties producerProps = ExampleConfig.getProducerProps();
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);
for (int i = 0; i < 4; i++) {
String value = "message-" + i;
System.out.printf("Sending message topic: %s, value: %s%n", TOPIC_NAME, value);
producer.send(new ProducerRecord<>(TOPIC_NAME, value));
}
producer.flush();
producer.close();
}
}
项目依赖与技术栈
- kafka_2.12 2.5.0: Apache Kafka
- JDK 8
- Maven 3.5.4
通过本文的详细步骤和代码示例，您应该能够理解如何在Kafka中使用commitSync()方法进行同步消息提交，并能够将其应用到实际的项目中。希望这能帮助您更好地管理和处理Kafka消息。