In terms of messaging technologies, we have dozens of popular options on the market, while each of them may only support certain use cases in the queuing and streaming space. Before I explain how to choose the right messaging system, I think it’s worth taking some time to explain message queuing and streaming first. With a basic understanding of these concepts, we will then look at some core and advanced capabilities required for a messaging system for a modern data architecture. After that, I will give a high-level overview of Apache Pulsar, a cloud-native messaging and streaming platform, and explain what makes Pulsar suitable for some common messaging scenarios.
Queuing: Decoupling
A message queue (also known as a queue) is used to store messages with a first-in-first-out (FIFO) data structure. Messages stay on a queue until they are consumed by other applications or systems. Many programming languages have built-in message queues as data pipelines between tasks for asynchronous decouplings, such as java.util.Queue in Java.