Tips, Expertise, Articles and Advice from the Pro's for Your Website or Blog to Succeed
Over the last few years, companies have used real-time updates to add new experiences and features and increase their market share.
It's now standard to expect a page within an app or browser to update parts of itself without forcing it to refresh. For example, a news page of live sports scores updates with the latest goal scored, or an app shows a change as you track your taxi on a map.
With the exponential growth of data and a lot of businesses moving online, it has become imperative to design systems that can act in real-time or near real-time to make any business decisions. So, after working on multiple backend projects through many years, I finally got to do build a real-time streaming platform. While working on the project, I did start experimenting with different tech stacks to deal with this. So, I am trying to share my learnings in a series of articles. Here is the first of them.
This post is aimed at engineers who are already familiar with microservices and Java and are looking to build their first real-time streaming pipeline. This POC is divided into 4 articles for the purpose of readability. They are as follows:
In the previous article, we learned how to install Resgate and use it to create a next-generation API with a simple Hello World service.
This time, we will look at how to update the data, and have all the clients synchronized in real-time, based on the Edit Text example found in the ResgateIO.Service library for .NET.
Made up of Elastisearch, "a search and analytics engine," Logstash, "a server-side data processing pipeline that "ingests data from multiple sources simultaneously, transforms it, and then sends it to a 'stash'," (according to Elastic's official site) and Kibana, a robust visualization tool, the ELK stack has quickly become one of the premier tools available to developers for data processing, management, and visualization.
Whether you're just starting out with any of the three technologies, or you're a seasoned veteran, we've compiled the best that our community has to offer for basic questions about getting started to complex tutorials for real-time data management.
This is the second part of the four parts series of articles. In the previous article, we introduced you to Apache Kafka. In this article, we will briefly discuss Kafka APIs with special attention given to Kafka's Streams API.
Before we have a deep dive in Kafka streams, here's a quick refresher on important concepts in Kafka.
Today, with the rise of IoT and Smart Devices, we are generating data at an unprecedented speed. With distributed computing, data is generated somewhere and processed somewhere else. Sensors or UI on devices capture some data (manual or automated) as an event and send it to some other unit for processing. This happens continuously.
These events may be processed at a fixed rate or in bursts, resulting in a stream of events. This process is known as an Event Stream. In most scenarios, these events are generated at a very high speed (seconds or even milliseconds). So, we need to process these event streams at the same or higher processing rate.