A Tale of Two Intersecting Quality Attributes: Security and Performance

I want to build a system that is highly secure, scalable, reliable, performant, compliant, robust, resilient, and durable.” Add more adjectives to that to really dream of a quintessential solution. Is that even possible? Where do we make the two ends meet — aspirations vs reality? What is the right intersection point? To answer that, we probably need to categorize the above, and maybe a few more quality attributes, into two major cross-cutting concerns — security and performance, and see where and how to strike a balance between the two, often referred to as architectural or design trade-offs.

Security is a requirement for each and every component involved in the overall system, which may include devices, networks, data, services, applications, storage, etc. However, it is not always mutually exclusive in the sense that the security of one component can partly or fully ensure the security of other components, depending on how they are configured. For example, we often offload SSL at the gateway level as the data is expected to travel within the internal network after that, which is deemed secure to a certain extent. Likewise, the performance of the overall system depends upon how each individual component is behaving. We may have a fast rendering UI but a slow API response that really sucks, or vice versa.

Querying Kafka Topics Using Presto

Presto is a distributed query engine that allows querying different data sources such as Kafka, MySQL, MongoDB, Oracle, Cassandra, Hive, etc. using SQL. It has the ability to analyze big data and query multiple data sources together.

In this article, we will discuss how Presto can be used to query Kafka topics. Below is the step-by-step process to set up Presto and Kafka, and connect them together. Here, I have considered MacOS, but similar setups can be done on any other system.

Kafka Event Exchange Between Local and Azure

While it may not be a daunting task to set up Kafka on a local machine or within a particular network and produce/consume messages, people do face challenges when they try to make it work across the network.

Let’s consider a hybrid scenario where your software solution is distributed across two different platforms (say AWS and Azure or on-premise and Azure), and there is a need to route messages from Kafka cluster hosted on one platform to the one hosted on another. This could be a valid business scenario wherein you are trying to consolidate your solution with one cloud platform, and in the interim, you need to have this routing in place till you complete your migration. Even in the long term, there may be a need to maintain solution across multiple platforms for various business and technical reasons.