Tips, Expertise, Articles and Advice from the Pro's for Your Website or Blog to Succeed
In this edition of "Best of DZone," we've chosen to take a look at Apache Kafka, the low-latency stream-processing platform that has become an industry-standard for real-time streaming and analytics, log aggregation, and Spark data ingestion since LinkedIn first released it to the open source community in early 2011.
With this collection, we hoped to provide readers with the resources and knowledge they need, regardless of their level of expertise with working the platform or big data in general, to master all things Kafka.
Hey all, today I will show one way to generate multiple consumer groups dynamically with Spring-Kafka. Before this approach, let's do it with annotations. We just create a configuration class which consist of a spring @Bean that generates our KafkaListenerContainerFactory.
You may also like: Spring for Apache Kafka Part 1: Error Handling, Message Conversion, and Transaction Support.
Example consumer config file:
Presto is a state of the art Distributed SQL Query Engine for big data, enabling efficient querying on cold data and various data sources. With extended SQL language and features like geospatial queries, joins between different data sources (SQL to join data from HDFS, Elasticsearch, and Kafka anyone?), and the ability to run on containers and cheap servers, Presto is slowly becoming the standard ad-hoc querying engine for big data.
In this talk, we will present Presto and how it can be used with Kafka. We will discuss data architectures, Presto features and why is it so good for your data, and finally see how it can be leveraged to querying data from Kafka as well as executing a single SQL statement that joins data from Kafka on data from SQL, Cassandra, Elastic, and more.
This article is a continuation of Part 1, Kafka Technical Overview and Part 2, Kafka Producer Overview articles. Let's look into different delivery semantics and how to achieve them using producer and broker properties.
Based on broker and producer configuration, all three delivery semantics— “at most once”, “at least once” and “exactly once” — are supported.
In this tutorial, we will quickly explore some basic to high-level approaches for testing microservices applications built using Kafka. Also, we will learn about the advantages of the declarative way of testing Kafka applications over the traditional/existing way of testing.
For everything explained here, you can find a running code example in the "Conclusion" section of this post.
In simple words, Apache Kafka is a hybrid of a distributed database and a message queue. In order to process terabytes of information, many large companies use it. Also, for its features, Kafka is widely popular. For example, a company like LinkedIn uses it to stream data about user activity, while the company like Netflix uses it for data collection and buffering for downstream systems like Elasticsearch, Amazon EMR, Mantis, and many more.
Moreover, let’s throw light on some features of Kafka that are important for Kafka load testing: