Posted on

Kafka for XML Message Integration and Processing

XML messages and XML Schema are not very common in the Apache Kafka and Event Streaming world! Why? Many people call XML legacy. It is complex, verbose, and often associated with the ugly WS-* Hell (SOAP, WSDL, etc). On the other side, every company older than five years uses XML. It is well understood, provides a good structure, and is human- and machine-readable.

This post does not want to start another flame war between XML and other technologies such as JSON (which also provides JSON Schema now), Avro, or Protobuf. Instead, I will walk you through the three main approaches to integrate between Kafka and XML messages as there is still a vast demand for implementing this integration today (often for integrating legacy applications and middleware).

Posted on

Apache Kafka and SAP ERP Integration Options

A question I get every week from customers across the globe is, "How can I integrate my SAP system with Apache Kafka?" This post explores various alternatives, including connectors, third party tools, custom glue code, and trade-offs between the different options.

After exploring what SAP is, I will discuss several integration options between Apache Kafka and SAP systems:

Posted on

Apache Kafka and Blockchain: Friends, Enemies, or Frenemies?

This blog post discusses the concepts, use cases, and architectures behind Event Streaming, Apache Kafka, Distributed Ledger (DLT), and Blockchain. A comparison of different technologies such as Confluent, AIBlockchain, Hyperledger, Ethereum, Ripple, IOTA, and Libra explores when to use Kafka, a Kafka-native blockchain, a dedicated blockchain, or Kafka in conjunction with another blockchain.

Use Cases for Secure and Tamper-Proof Data Processing With a Blockchain

Blockchain is a hype topic for many years. While many companies talk about the buzzword, it is tough to find use cases where Blockchain is the best solution. The following examples show the potential of Blockchain. Here it might make sense:

Posted on

Apache Kafka and Machine Learning in Pharma and Life Sciences Industry

This blog post covers use cases and architectures for Apache Kafka and Event Streaming in Pharma and Life Sciences. The technical example explores drug development and discovery with real time data processing, machine learning, workflow orchestration and image / video processing.

Use Cases in Pharmaceuticals and Life Sciences for Event Streaming and Apache Kafka

The following shows some of the use cases I have seen in the field in pharma and life sciences:

Posted on

IoT Architectures for Digital Twin With Apache Kafka

A digital twin is a virtual representation of something else. This can be a physical thing, process or service. This post covers the benefits and IoT architectures of a Digital Twin in various industries and its relation to Apache Kafka, IoT frameworks and Machine Learning. Kafka is often used as a central event streaming platform to build a scalable and reliable digital twin and digital thread for real-time streaming sensor data.

I already blogged about this topic recently in detail: Apache Kafka as Digital Twin for Open, Scalable, Reliable Industrial IoT (IIoT). Hence that post covers the relation to Event Streaming and why people choose Apache Kafka to build an open, scalable and reliable digital twin infrastructure.

Posted on

Event Streaming and Apache Kafka in Telco Business (OSS/BSS)

Event Streaming is a hot topic in Telecommunications Industry. In the last few months, I have seen various projects leveraging Apache Kafka and its ecosystem to implement scalable real-time infrastructure in OSS and BSS scenarios. This blog post covers the reasons for this trend. Finally, we'll show a whiteboard video recording exploring the different use cases for event streaming in telcos in detail.

The Evolution of the Telecommunications Industry

The telecommunications industries within the sector of information and communication technology is made up of all telecommunications/telephone companies and internet service providers. It plays a crucial role in the evolution of mobile communications and the information society.

Posted on

Architectures for Distributed, Hybrid, Edge, and Global Apache Kafka

Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. Learn about several scenarios that may require multi-cluster solutions and see real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka.

Key Takeaways for Multi Data Center Kafka Architectures

  • In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments.
  • Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs.
  • Learn about features and limitations of Kafka for multi-cluster deployments- Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.
  • Learn about architectures like stretched cluster, hybrid integration and fully-managed serverless Kafka in the cloud (using Confluent Cloud), and tools like MirrorMaker 2, Confluent Replicator, Multi-Region Clusters (MRP), Global Kafka, and more.

Slide Deck

 

Posted on

Smart City With an Event Streaming Platform Like Apache Kafka

A smart city is an urban area that uses different types of electronic Internet of Things (IoT) sensors to collect data and then use insights gained from that data to manage assets, resources, and services efficiently. This includes data collected from citizens, devices, and assets that are processed and analyzed to monitor and manage traffic and transportation systems, power plants, utilities, water supply networks, waste management, crime detection, information systems, schools, libraries, hospitals, and other community services.

You may also like: Smart Cities: Who Wins and Who Loses?

Posted on

Apache Kafka Is the New Black at the Edge in IoT Projects

Find out more about the Edge of IoT and Apache Kafka.

The following question comes up almost every week in conversations with customers: Can and should I deploy Apache Kafka at the edge? Or should I just deploy Kafka in a "real" data center or public cloud infrastructure? I am glad that people ask because it is a valid question in various industries, including manufacturing, automation industry, aviation, logistics, and retailing.

Posted on

Machine Learning and Real-Time Analytics in Apache Kafka Applications

The relationship between Apache Kafka and machine learning (ML) is an interesting one that I've written about quite a bit in How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning.

This post addresses a specific part of building a machine learning infrastructure: the deployment of an analytic model in a Kafka application for real-time predictions.

Posted on

IoT and Event Streaming at Scale With Kafka and MQTT

A key challenge to IoT is the ability to integrate devices and machines that can process data in real-time and at scale.

The Internet of Things (IoT) is getting more and more traction as valuable use cases come to light. A key challenge, however, is integrating devices and machines to process the data in real-time and at scale. Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets.

You may also like: IoT: Device Data and Stream Processing

Kafka-native options to note for MQTT integration beyond Kafka client APIs like Java, Python, .NET, and C/C++ are:

Posted on

Service Mesh and Cloud-Native Microservices

We all want this kind of service.


You may also like: Istio Service Mesh, the Step-by-Step Guide, Part 1: Theory

Microservices need to be decoupled, flexible, operationally transparent, data-aware and elastic. Most material from last year only discusses point-to-point architectures with tightly coupled and non-scalable technologies like REST / HTTP. This blog post takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed, and efficiency.

Posted on

Apache Kafka, KSQL, and Apache PLC4X for Industrial IoT and Automation

Learn more about IIoT automation with Apache Kafka, KSQL, and Apache PLC4X

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL), and Apache PLC4X are a great open-source choice to implement this IIoT integration end-to-end in a scalable, reliable, and flexible way.

This blog post covers a high-level overview of the challenges and good, flexible architecture to solve the problems. In the end, I share a video recording and the corresponding slide deck. These provide many more details and insights.