Data Streaming for Data Ingestion Into the Data Warehouse and Data Lake

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Storing data at rest for reporting and analytics requires different capabilities and SLAs than continuously processing data in motion for real-time workloads. Many open-source frameworks, commercial products, and SaaS cloud services exist. Unfortunately, the underlying technologies are often misunderstood and overused for monolithic and inflexible architectures, and vendors pitch for the wrong use cases. Let’s explore this dilemma in a blog series. Learn how to build a modern data stack with cloud-native technologies.

Blog Series: Data Warehouse vs. Data Lake vs. Data Streaming

This blog series explores concepts, features, and trade-offs of a modern data stack using a data warehouse, data lake, and data streaming together:

Packages for Store Routines in MariaDB 11.4
No categories
MariaDB 11.4 introduced many advanced features. One that grabbed my attention is the general support of packages for stored routines. Although this was previously available by activating the Oracle compatibility mode, now the feature is available gener... […]
Maintain Chat History in Generative AI Apps With Valkey
No categories
A while back I wrote up a blog post on how to use Redis as a chat history component with LangChain. Since LangChain already had Redis chat history available as a component, it was quite convenient to write a client application. But, that's not the same... […]
Knowledge Graph Enlightenment, AI, and RAG
No categories
In the previous edition of the YotG newsletter, the wave of Generative AI hype was probably at its all-time high. Today, while Generative AI is still talked about and trialed, the hype is subsiding. Skepticism is settling in, and for good reason. Repor... […]
Building an Effective Zero Trust Security Strategy for End-To-End Cyber Risk Management
No categories
You've probably heard a lot about zero-trust security lately, and for good reason. As we move more of our applications and data to the cloud, the traditional castle-and-moat approach to security just doesn't cut it anymore. This makes me come... […]
Phased Approach to Data Warehouse Modernization
No categories
A modernized database will help you focus on building innovative solutions rather than investing your time and effort in managing these legacy systems. Based on the scale of your existing data warehouse processes or jobs, it can be an enormous task to ... […]