This article outlines a solution for streaming events from Kafka, forwarding them to Redis using its Stream API, and reading individual streams from Redis via its streaming API. The added complexity in this scenario is the need to stream events from an HTTP endpoint using Server-Sent Events (SSE) while ensuring that only events relevant to a specific client ID are processed and sent.
Problem Statement
Many companies have an existing Kafka infrastructure where events are being produced. Our goal is to set up a system that subscribes to Kafka messages but only processes events relevant to a specific client ID. These filtered events should be forwarded to Redis using its Stream API. Additionally, we need to establish an HTTP endpoint for Server-Sent Events (SSE) that allows the specified client to receive real-time event updates.
Welcome to this series of blog posts that covers Redis Streams with the help of a practical example. We will use a sample application to make Twitter data available for search and query in real-time. RediSearch and Redis Streams serve as the backbone of this solution that consists of several co-operating components, each of which will be covered in a dedicated blog post.
The code is available in this GitHub repo - https://github.com/abhirockzz/redis-streams-in-action
Welcome to this series of blog posts that cover Redis Streams with the help of a practical example. We will use a sample application to make Twitter data available for search and query in real-time. RediSearch and Redis Streams serve as the backbone of this solution that consists of several co-operating components, each of which will be covered in a dedicated blog post.
The code is available in this GitHub repo - https://github.com/abhirockzz/redis-streams-in-action
The enterprise app software market is growing at an annual rate of 10.4% — so it’s crucial that these applications both deliver high-speed data processing and accommodate specific business requirements. Adopting a rapidly deployable microservices architecture can bring greater flexibility and agility, but this requires you to pay significant attention to how you share data between microservices and prepare for communication at scale. Redis offers open-source solutions with high availability and durability for building reliable event stores for trading microservice messages. This Refcard will teach you about types of data processing in microservices, using Redis Streams for interservices communication, optimizing microservices, and more.
