Author: Yaniv Ben Hemo

In the realm of modern software development, managing and maintaining data integrity is paramount. Traditional approaches often involve updating the state of an application directly within a database. However, as systems grow in complexity, ensuring data consistency and traceability becomes more challenging. This is where Event Sourcing, coupled with a powerful distributed streaming platform like Memphis.dev, emerges as a robust solution and a great data structure to work with.

What Is Event Sourcing?

At its core, Event Sourcing is a design pattern that captures every change or event that occurs in a system as an immutable and sequentially ordered log of events. Instead of persisting the current state of an entity, Event Sourcing stores a sequence of state-changing events. These events serve as a single source of truth for the system’s state at any given point in time.

First things first, Event-driven architecture. EDA and serverless functions are two powerful software patterns and concepts that have become popular in recent years with the rise of cloud-native computing. While one is more of an architecture pattern and the other a deployment or implementation detail, when combined, they provide a scalable and efficient solution for modern applications.

---

What Is Event-Driven Architecture

EDA is a software architecture pattern that utilizes events to decouple various components of an application. In this context, an event is defined as a change in state. For example, for an e-commerce application, an event could be a customer clicking on a listing, adding that item to their shopping cart, or submitting their credit card information to buy. Events also encompass non-user-initiated state changes, such as scheduled jobs or notifications from a monitoring system.

Before deepening into the different supporting technologies, let’s create a baseline about schemas and message brokers or async server-server communication.

Schema = Struct.

Introduction

Supabase is an open-source Firebase alternative.

In databases, change data capture (or CDC) is a set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data or because of it.

We distilled 160 conversations with tech leaders across enterprises, startups, NASDAQ companies, and legacy organizations, from team leaders to CTOs and VP R&D. Here is their Top five list of data engineering trends that will likely come to life in 2023.

1. Data Contracts

Data Contracts are API-based agreements between Software Engineers who own services and Data Consumers that understand how the business works to generate well-modeled, high-quality, trusted data. Suppose you take a good look at it. In that case, you have at least ten different data producers and multiple consumers, written in different languages, interacting with various databases, SQL, No-SQL, and the holy grail data models. It's a mess. Data contracts are still a management or operational concept. Still, we are starting to see more and more traction and conversation around it (Chad Sanderson covers the subject in depth in his newsletter). 

We are all familiar with drifting in-app configuration and IaC. We’re starting with a specific configuration backed with IaC files. Soon after, we are facing a “drift” or a change between what is actually configured in our infrastructure and files. The same behavior happens in data. The schema starts in a specific shape. As data ingestion grows and scales to different sources, we get a schema drift, a messy, unstructured database and an analytical layer that keeps failing due to a bad schema. In this article, we will learn how to deal with the scenario and how to work with dynamic schemas.

Schemas Are a Major Struggle

A schema defines the structure of the data format. Keys/Values/Formats/Types, a combination of all, results in a defined structure or simply—schema.

What Is RabbitMQ?

RabbitMQ is a lightweight and easy-to-deploy messaging queue for on-premises and cloud environments. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

What Is Memphis.dev?

Memphis is a next-generation message broker.

What is AWS SQS?

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.

---

What is Memphis.dev?

Memphis is a next-generation messaging queue.

Among the open-source projects my college buddies (and my future co-founders of memphis.dev) and I built, you can find “Makhela,” a Hebrew word for choir.

For the sake of simplicity, we will call it "Choir" in this article.