We introduce a recipe for creating resilient Kafka consumers using Reactor Kafka. This approach is one that we've developed over time and incorporates the learnings from our experience with running Reactor Kafka - and all the challenges that come with that. The consumer described in this article provides at-least-once delivery semantics using manual acknowledgments, which is mostly suited for applications where data loss is not acceptable. However, most of the concepts described here still apply to other types of delivery semantics supported by the framework.
Kafka Configuration
The configuration class defines the properties to be used by the Kafka consumer, after which it uses them to instantiate a KafkaReceiver that will be made available as a bean in the application context.
Separation of Reactive and Non-Reactive Code
One of the most important distinctions to keep in mind when working with Project Reactor or any other reactive streams implementation is the difference between assembly-time vs. subscription-time in code execution.
Regardless of your level of experience with reactive programming, odds are you’ve already encountered the famous: