Author: Gautam Goswami

At first, data tiering was a tactic used by storage systems to reduce data storage costs. This involved grouping data that was not accessed as often into more affordable, if less effective, storage array choices. Data that has been idle for a year or more, for example, may be moved from an expensive Flash tier to a more affordable SATA disk tier. Even though they are quite costly, SSDs and flash can be categorized as high-performance storage classes. Smaller datasets that are actively used and require the maximum performance are usually stored in Flash.

Cloud data tiering has gained popularity as customers seek alternative options for tiering or archiving data to a public cloud. Public clouds presently offer a mix of object and file storage options. Object storage classes such as Amazon S3 and Azure Blob (Azure Storage) deliver significant cost efficiency and all the benefits of object storage without the complexities of setup and management. 

Over the past few years, Apache Kafka has emerged as the leading standard for streaming data. Fast-forward to the present day: Kafka has achieved ubiquity, being adopted by at least 80% of the Fortune 100. This widespread adoption is attributed to Kafka's architecture, which goes far beyond basic messaging. Kafka's architecture versatility makes it exceptionally suitable for streaming data at a vast "internet" scale, ensuring fault tolerance and data consistency crucial for supporting mission-critical applications. 

Flink is a high-throughput, unified batch and stream processing engine, renowned for its capability to handle continuous data streams at scale. It seamlessly integrates with Kafka and offers robust support for exactly-once semantics, ensuring each event is processed precisely once, even amidst system failures. Flink emerges as a natural choice as a stream processor for Kafka. While Apache Flink enjoys significant success and popularity as a tool for real-time data processing, accessing sufficient resources and current examples for learning Flink can be challenging. 

When data is analyzed and processed in real time, it can yield insights and actionable information either instantly or with very little delay from the time the data is collected. The capacity to collect, handle, and retain user-generated data in real time is crucial for many applications in today’s data-driven environment. 

There are various ways to emphasize the significance of real-time data analytics like timely decision-making, IoT and sensor data processing, enhanced customer experience, proactive problem resolution, fraud detection and security, etc. Rising to the demands of diverse real-time data processing scenarios, Apache Kafka has established itself as a dependable and scalable event streaming platform. 

In today's data-driven world, the capability to transport and circulate large amounts of data, especially video files, in real-time is crucial for news media companies. For example, an incident occurred in a specific location, and a news reporter promptly filmed the entire situation. Subsequently, the complete video was distributed for broadcasting across their multiple studios situated in geographically distant locations.

To construct or create a comprehensive solution for the given problem statement, we can utilize Apache Kafka in conjunction with external storage to upload large-sized video files. The external storage may take the form of a cloud store, such as Amazon S3, or an on-premise large file storage system, such as a network system or HDFS (Hadoop Distributed File System). Of course, Apache Kafka was not designed to handle large data files as direct messages for communication between publishers and subscribers/consumers. Instead, it serves as a modern event streaming platform where, by default, the maximum size of messages is limited to 1 MB.

The Apache Kafka, a distributed event streaming technology, can process trillions of events each day and eventually demonstrate its tremendous throughput and low latency. That’s building trust and over 80% of Fortune 100 businesses use and rely on Kafka. To develop high-performance data pipelines, streaming analytics, data integration, etc., thousands of companies presently use Kafka around the globe. By leveraging the zero-copy principle, Kafka improves efficiency in terms of data transfer. In short, when doing computer processes, the zero-copy technique is employed to prevent the CPU from being used for data copying across memory regions. Additionally, it removes pointless data copies, conserving memory bandwidth and CPU cycles.

Broadly, the zero-copy principle in Apache Kafka refers to a technique used to improve the efficiency of data transfer between producers and consumers by minimizing the amount of data copying performed by the operating system. By minimizing the CPU and memory overhead involved in copying data across buffers, the zero-copy technique can be very helpful for high-throughput, low-latency systems like Kafka.

Although both Apache Druid and Apache Kafka are potent open-source data processing tools, they have diverse uses. While Druid is a high-performance, column-store, real-time analytical database, Kafka is a distributed platform for event streaming. However, they can work together in a typical data pipeline scenario where Kafka is used as a messaging system to ingest and store data/events, and Druid is used to perform real-time analytics on that data. In short, indexing is the process of loading data in Druid, and Druid reads the data from a streaming source system like Kafka and eventually stores it in data files called segments. Druid includes an Apache Kafka Indexing Service that enables Druid to accept data streams from Apache Kafka, analyze the data in real time, and index the data for querying and analysis. 

A supervisor is a built-in part of Druid, making it easier to ingest, analyze, and monitor data in real-time. The data ingestion lifecycle is managed by druid supervisors. They handle jobs like reading information from a streaming source (like Kafka topics), indexing it into Druid segments, and keeping track of the ingestion procedure. The data ingestion for Kafka streaming ingestion is configured by the supervisor's specification. A JSON specification (often referred to as the supervisor spec) that specifies how the supervisor should consume data from Kafka and how it should process and index that data into Druid must be provided when configuring an Apache Kafka supervisor. Kafka indexing tasks read events using Kafka’s own partition and offset mechanism to guarantee exact-once ingestion. The Kafka supervisor in Druid reads the data in real time from the mentioned topic name and converts them into Druid events based on the submitted supervisor spec. The supervisor applies any necessary transformations or aggregations on the data before indexing it into Druid segments. These segments are essentially Druid’s way of storing and organizing data for efficient querying. 

The phrase “deep storage” refers to the long-term storage system used by Apache Druid, where past data segments are preserved for durability and retrieval in the future. Druid stores data in files called segments, and deep storage is the place where segments are stored. Even though Druid’s native integration with Apache Kafka (can read here how to integrate Druid with Kafka) and Amazon Kinesis, which allows query-on-arrival at millions of events per second, low latency ingestion, etc., and eventually enables us to fully exploit the potential of streaming data, the deep storage mechanism added another advantage of data durability. Impressively, it’s a type of storage that Apache Druid does not offer.

Druid’s Deep storage guarantees long-term data persistence even if data is deleted from the live cluster after compaction. This ensures data longevity and offers data loss protection. In short, Druid stores data in files called segments, and the compaction in Druid can be defined as the process where small segments are merged into larger segments to boost the query performance after data is ingested into Druid. Historical processes of Druid cache data segments on local disk and serve queries from that cache as well as from an in-memory cache. Thus, Druid never needs to access deep storage during a query and helps it offer the best query latencies possible. Additionally, it means that we need to have enough disc space for the data we intend to load across all of our historical servers.

A real-time analytics database called Apache Druid is developed for quick slice-and-dice analysis on massive data volumes. The best data for Apache Druid is event-oriented and frequently utilized as the database backend for analytical application GUIs and for highly concurrent APIs that require quick aggregations. Druid can be leveraged very effectively, where real-time ingestion, fast query performance, and high uptime are crucial.

At the other end, Apache Kafka is gaining outstanding momentum as a distributed event streaming platform with excellent performance, low latency, fault tolerance, and high throughput and having capable of handling thousands of messages per second.

To get more clarity about ISR in Apache Kafka, we should first carefully examine the replication process in the Kafka broker. In short, replication means having multiple copies of our data spread across multiple brokers. Maintaining the same copies of data in different brokers makes possible the high availability in case one or more brokers go down or are untraceable in a multi-node Kafka cluster to server the requests. Because of this reason, it is mandatory to mention how many copies of data we want to maintain in the multi-node Kafka cluster while creating a topic. It is termed a replication factor, and that’s why it can’t be more than one while creating a topic on a single-node Kafka cluster. The number of replicas specified while creating a topic can be changed in the future based on node availability in the cluster.

On a single-node Kafka cluster, however, we can have more than one partition in the broker because each topic can have one or more partitions. The Partitions are nothing but sub-divisions of the topic into multiple parts across all the brokers on the cluster, and each partition would hold the actual data(messages). Internally, each partition is a single log file upon which records are written in an append-only fashion. Based on the provided number, the topic internally split into the number of partitions at the time of creation. Thanks to partitioning, messages can be distributed in parallel among several brokers in the cluster. Kafka scales to accommodate several consumers and producers at once by employing this parallelism technique. This partitioning technique enables linear scaling for both consumers and providers. Even though more partitions in a Kafka cluster provide a higher throughput but with more partitions, there are pitfalls too. Briefly, more file handlers would be created if we increase the number of partitions as each partition maps to a directory in the file system in the broker.

Any trustworthy data streaming pipeline needs to be able to identify and handle faults. Exceptionally while IoT devices ingest endlessly critical data/events into permanent persistence storage like RDBMS for future analysis via multi-node Apache Kafka cluster. (Please click here to read how to set up a multi-node Apache Kafka Cluster). There could be scenarios where IoT devices might send fault/bad events due to various reasons at the source points, and henceforth appropriate actions can be executed to correct it further. The Apache  Kafka architecture does not include any filtering and error handling mechanism within the broker so that maximum performance/scale can be yielded.  Instead, it is included in Kafka Connect, which is an integration framework of Apache Kafka. As a default behavior, if a problem arises as a result of consuming an invalid message, the Kafka Connect task terminates, and the same applies to JDBC Sink Connector.

Kafka Connect has been classified into two categories, namely Source (to ingest data from various data generation sources and transport to the topic) and Sink (to consume data/messages from the topic and send them eventually to various destinations).  Without implementing a strict filtering mechanism or exception handling, we can ingest/publishes messages inclusive of wrong formatted to the Kafka topic because the Kafka topic accepts all messages or records as byte arrays in key-value pairs. But by default, the Kafka Connect task stops if an error occurs because of consuming an invalid message, and on top of that JDBC sink connector additionally won’t work if there is an ambiguity in the message schema.

As a bird’s eye view, Apache Zookeeper has been leveraged to get coordination services for managing distributed applications. It holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems. As an example, Apache Kafka uses ZooKeeper for choosing their leader node for the topic partitions. Please click here if you want to read how to set up the multi-node Apache Zookeeper cluster on Ubuntu/Linux.

ZNodes

The key concept of ZooKeeper is the ZNode, which can act as either files or directories. ZNodes can be replicated between servers as they are working in a distributed file system. A ZNode can be described by a data structure called stats, and it consolidates information about ZNode context like creation time, number of changes (as version), number of children, length of stored data or ZXID (ZooKeeper transaction ID) of creation, and last change. 

As a bird’s eye view, Apache Zookeeper has been leveraged to get coordination services for managing distributed applications. It holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems. To consider as an example, Apache Kafka uses Zookeeper for choosing their leader node for the topic partitions.

zNodes

The key concept of Zookeeper is the zNode, which can be acted either as files or directories. ZNodes can be replicated between servers as they are working in a distributed file system. Znode can be described by a data structure called stats and it consolidates information about zNode context like creation time, number of changes (as version), number of children, length of stored data or zxid (ZooKeeper transaction ID) of creation, and last change. For every modification of zNodes, its version increases. 

 This short article explains how to resolve the error: ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)kafka.common.InconsistentClusterIdException when we start the Apache Kafka installed and configured on a multi-node cluster. You can read the steps for setting up a multi-node Apache Kafka cluster here.

 Without integrating Apache Zookeeper, Kafka alone won’t be able to form the complete Kafka cluster. Because ZooKeeper handles the leadership election of Kafka brokers and manages service discovery as well as cluster topology. It also tracks when topics are created or deleted from the cluster and maintains a topic list. Overall, ZooKeeper provides an in-sync view of the Kafka cluster.

Data is quickly becoming the new currency of the digital economy, but it is useless if it can’t be processed. The processing of data is essential for subsequent decision-making or executable actions either by the human brain or various devices/applications, etc. There are two primary ways of processing data: namely, batch processing and stream processing. Typically batch processing has been adopted for very large data sets and projects where there is a necessity for deeper data analysis. On the other side, stream processing is used for speed and quickness as soon as data gets generated at the source. In stream processing, a data point or “micro-batch” is inserted directly into the analytical system bit-by-bit as soon as it is generated and processed subsequently to produce key insights in near real-time. By leveraging platforms/frameworks like Apache Kafka, Apache Flink, Apache Storm, or Apache Samza, we can make decisions quickly and efficiently from generated key insights after processing the streaming of data. (In my previous post "Crafting a Multi-Node Multi-Broker Kafka Cluster- A Weekend Project," read more on how to install Apache Kafka.)

Before developing a system or new infrastructure at the enterprise level for data processing, the adoption of efficient architecture is mandatory to ensure software/frameworks are flexible and scalable enough to handle the massive volume of data with an open design principle. In today’s Big Data landscape, the Lambda architecture is a new archetype for handling the vast amount of data. This architecture can be adopted for both batches as well as stream processing of data as it is a combination of three layers namely batch layer, speed or real-time layer, and service layer. Each layer in the Lambda Architecture relay on various software components.