【Kafka2.0.0新特性】

Kafka 2.0.0 includes a number of significant new features. Here is a summary of some notable changes:

• KIP-290 adds support for prefixed ACLs, simplifying access control management in large secure deployments. Bulk access to topics, consumer groups or transactional ids with a prefix can now be granted using a single rule. Access control for topic creation has also been improved to enable access to be granted to create specific topics or topics with a prefix.
• KIP-255 adds a framework for authenticating to Kafka brokers using OAuth2 bearer tokens. The SASL/OAUTHBEARER implementation is customizable using callbacks for token retrieval and validation.
• Host name verification is now enabled by default for SSL connections to ensure that the default SSL configuration is not susceptible to man-in-the-middle attacks. You can disable this verification if required.
• You can now dynamically update SSL truststores without broker restart. You can also configure security for broker listeners in ZooKeeper before starting brokers, including SSL keystore and truststore passwords and JAAS configuration for SASL. With this new feature, you can store sensitive password configs in encrypted form in ZooKeeper rather than in cleartext in the broker properties file.
• The replication protocol has been improved to avoid log divergence between leader and follower during fast leader failover. We have also improved resilience of brokers by reducing the memory footprint of message down-conversions. By using message chunking, both memory usage and memory reference time have been reduced to avoid OutOfMemory errors in brokers.
• Kafka clients are now notified of throttling before any throttling is applied when quotas are enabled. This enables clients to distinguish between network errors and large throttle times when quotas are exceeded.
• We have added a configuration option for Kafka consumer to avoid indefinite blocking in the consumer.
• We have dropped support for Java 7 and removed the previously deprecated Scala producer and consumer.
• Kafka Connect includes a number of improvements and features. KIP-298 enables you to control how errors in connectors, transformations and converters are handled by enabling automatic retries and controlling the number of errors that are tolerated before the connector is stopped. More contextual information can be included in the logs to help diagnose problems and problematic messages consumed by sink connectors can be sent to a dead letter queue rather than forcing the connector to stop.
• KIP-297 adds a new extension point to move secrets out of connector configurations and integrate with any external key management system. The placeholders in connector configurations are only resolved before sending the configuration to the connector, ensuring that secrets are stored and managed securely in your preferred key management system and not exposed over the REST APIs or in log files.
• We have added a thin Scala wrapper API for our Kafka Streams DSL, which provides better type inference and better type safety during compile time. Scala users can have less boilerplate in their code, notably regarding Serdes with new implicit Serdes.
• Message headers are now supported in the Kafka Streams Processor API, allowing users to add and manipulate headers read from the source topics and propagate them to the sink topics.
• Windowed aggregations performance in Kafka Streams has been largely improved (sometimes by an order of magnitude) thanks to the new single-key-fetch API.
• We have further improved unit testibility of Kafka Streams with the kafka-streams-testutil artifact.

 

• 增加了前缀通配符访问控制(ACL)的支持，详见 KIP-290，这样我们可以更加细粒度的进行访问控制；
• 更全面的数据安全支持，KIP-255 里面添加了一个框架，我们可以使用OAuth2 bearer tokens 来对访问 KafkaBrokers 进行权限控制。
• 现在，SSL连接默认启用主机名验证（Host name verification），以确保默认 SSL 配置不受中间人攻击的影响。 如果需要，您可以禁用此验证。
• 现在，我们可以在不重启 Broker 的情况下动态更新 SSL 信任库（ SSL truststores）。我们还可以在启动 Broker 之前在 ZooKeeper 中为 Broker 侦听器（broker listeners）配置安全性，包括 SSL 密钥库和信任库密码以及 SASL的JAAS配置。 使用此新功能，您可以在 ZooKeeper 中以加密形式存储敏感密码配置，而不是在 Broker 属性文件中以明文形式存储。
• 复制协议已得到改进，以避免在 fast leader failover 期间 leader 和 follower 之间的日志分歧（log divergence）。
• 保证在线升级的方便性，在这一次的 2.0.0 版本中，更多相关的属性被加了进来，详情请参见 KIP-268、KIP-279、KIP-283 等等
• 简化了 Kafka Streams 升级过程，详情参见 KIP-268
• 进一步加强了 Kafka 的可监控性，包括添加了很多系统静态属性以及动态健康指标，请参见 KIP-223、KIP-237、KIP-272 等等。
• 在即将发布的 2.0 版本中，加入了另一个“领先”指标（lead metrics），定义为分区首端（log-start-offset）与消费者在分区上的位置距离，当此指标趋近于零时，代表消费者有跌出可消费范围因而丢失数据的危险。

 

KIP-290 adds support for prefixed ACLs, simplifying access control management in large secure deployments. Bulk access to topics, consumer groups or transactional ids with a prefix can now be granted using a single rule. Access control for topic creation has also been improved to enable access to be granted to create specific topics or topics with a prefix. 

KIP-290增加了对前缀ACL的支持，简化了在大型安全部署中的访问控制管理。现在可以使用单个规则来对主题、消费群体或具有前缀的事务ID进行批量访问。主题创建的访问控制也得到了改进，以便允许访问以创建具有前缀的特定主题或主题。

 

KIP-255 adds a framework for authenticating to Kafka brokers using OAuth2 bearer tokens. The SASL/OAUTHBEARER implementation is customizable using callbacks for token retrieval and validation. 

KIP-255增加了一个使用OAuth2承载令牌对KafkaBroker进行认证的框架。SASL/OAuthBurER实现可使用回调进行令牌检索和验证。

 

Host name verification is now enabled by default for SSL connections to ensure that the default SSL configuration is not susceptible to man-in-the-middle attacks. You can disable this verification if required. 

默认情况下，启用SSL连接的主机名验证，以确保默认SSL配置不受中间人攻击的影响。如果需要，可以禁用此验证。

 

You can now dynamically update SSL truststores without broker restart. You can also configure security for broker listeners in ZooKeeper before starting brokers, including SSL keystore and truststore passwords and JAAS configuration for SASL. With this new feature, you can store sensitive password configs in encrypted form in ZooKeeper rather than in cleartext in the broker properties file. 

您现在可以在不重新启动代理的情况下动态更新SSL信任库。 您还可以在启动代理之前在ZooKeeper中为代理侦听器配置安全性，包括SSL密钥库和信任库密码以及SASL的JAAS配置。 使用此新功能，您可以在ZooKeeper中以加密形式存储敏感密码配置，而不是在代理属性文件中以明文形式存储。

 

The replication protocol has been improved to avoid log divergence between leader and follower during fast leader failover. We have also improved resilience of brokers by reducing the memory footprint of message down-conversions. By using message chunking, both memory usage and memory reference time have been reduced to avoid OutOfMemory errors in brokers. 

复制协议已得到改进，以避免在快速领导者故障转移期间leader和follower之间的日志分歧。 我们还通过减少消息下转换的内存占用来提高代理的恢复能力。 通过使用消息分块，内存使用和内存引用时间都已减少，以避免代理中的OutOfMemory(内存不足)错误。

 

Kafka clients are now notified of throttling before any throttling is applied when quotas are enabled. This enables clients to distinguish between network errors and large throttle times when quotas are exceeded. 

现在，在启用配额之前应用任何限制之前，Kafka客户端会收到限制通知。 这使客户能够在超过配额时区分网络错误和大的节流时间。

 

We have added a configuration option for Kafka consumer to avoid indefinite blocking in the consumer. 

我们为Kafka消费者添加了一个配置选项，以避免消费者无限期阻止。

 

We have dropped support for Java 7 and removed the previously deprecated Scala producer and consumer. 

我们已经放弃了对Java 7的支持，并删除了之前弃用的Scala生产者和消费者。

 

Kafka Connect includes a number of improvements and features. KIP-298 enables you to control how errors in connectors, transformations and converters are handled by enabling automatic retries and controlling the number of errors that are tolerated before the connector is stopped. More contextual information can be included in the logs to help diagnose problems and problematic messages consumed by sink connectors can be sent to a dead letter queue rather than forcing the connector to stop. 

Kafka Connect包含许多改进和功能。 KIP-298使您能够通过启用自动重试和控制连接器停止前容许的错误数来控制连接器，转换和转换器中的错误处理方式。 日志中可以包含更多上下文信息，以帮助诊断问题，并且可以将接收器连接器消耗的有问题消息发送到死信队列，而不是强制连接器停止。

 

KIP-297 adds a new extension point to move secrets out of connector configurations and integrate with any external key management system. The placeholders in connector configurations are only resolved before sending the configuration to the connector, ensuring that secrets are stored and managed securely in your preferred key management system and not exposed over the REST APIs or in log files. 

KIP-297增加了一个新的扩展点，可以将密钥从连接器配置中移除，并与任何外部密钥管理系统集成。 连接器配置中的占位符仅在将配置发送到连接器之前解析，确保在首选密钥管理系统中安全地存储和管理机密，而不是通过REST API或日志文件公开。

 

We have added a thin Scala wrapper API for our Kafka Streams DSL, which provides better type inference and better type safety during compile time. Scala users can have less boilerplate in their code, notably regarding Serdes with new implicit Serdes. 

我们为Kafka Streams DSL添加了一个瘦Scala包装器API，它在编译期间提供了更好的类型推断和更好的类型安全性。 Scala用户可以在代码中使用更少的样板，特别是关于具有新隐式Serdes的Serdes。

 

Message headers are now supported in the Kafka Streams Processor API, allowing users to add and manipulate headers read from the source topics and propagate them to the sink topics. 

Kafka Streams Processor API现在支持消息头，允许用户添加和操作从源主题读取的头，并将它们传播到接收器主题。

 

Windowed aggregations performance in Kafka Streams has been largely improved (sometimes by an order of magnitude) thanks to the new single-key-fetch API. 

由于采用了新的单键获取API，Kafka Streams中的窗口聚合性能已大大提高（有时甚至达到一个数量级）。

 

We have further improved unit testibility of Kafka Streams with the kafka-streams-testutil artifact. 

我们使用kafka-streams-testutil工件进一步改进了Kafka Streams的单元可测性。