Tips, Expertise, Articles and Advice from the Pro's for Your Website or Blog to Succeed
Non Blocking retries in Kafka are done via configuring retry topics for the main topic. An Additional Dead Letter Topic can also be configured if required. Events will be forwarded to DLT if all retries are exhausted. A lot of resources are available in the public domain to understand the technicalities.
It can be a challenging job when it comes to writing integration tests for the retry mechanism in your code.
Apache Kafka is the most widely used reliable and scalable eventing platform. It helps in providing the base for a fast, scalable, fault-tolerant, event-driven microservice architecture. Any type of data can be published into a Kafka topic and can be read from. Kafka does not provide an out-of-the-box schema validation system and hence is prone to junk data being fed into a topic.
Confluent has devised a concept of a schema registry that can be used to implement type checking of messages in any Kafka installation. The Schema Registry needs the message schema to be registered against a topic and it enforces that only messages conforming to the schema are sent to the topic.
The requirement was to hit an eternal web endpoint and fetch some data and apply some logic to it, but the external API was super slow and the response time grew linearly with the number of records fetched. This called for the need to parallelize the entire API call in chunks/pages and aggregate the data.
Our synchronous feign client: