Posted on

Data Lake vs. Data Warehouse

Data lakes and data warehouses are critical technologies for business analysis, but the differences between the two can be confusing. How are they different? Is one more stable than the other? Which one is going to help your business the most? This article seeks to demystify these two systems for handling your data.

What Is a Data Lake?

A data lake is a centralized repository designed to store all your structured and unstructured data. Further, a data lake can store any type of data using its native format, without size limits. Data lakes were developed primarily to handle the volumes of big data, and thus they excel at handling unstructured data. You typically move all the data into a data lake without transforming it. Each data element in a lake is assigned a unique identifier, and it is extensively tagged so that you can later find the element via a query. The benefit of this is that you never lose data, it can be available for extensive periods of time, and your data is very flexible because it does not need to adhere to a particular schema before it is stored.

Posted on

What Are Data Silos?

A data silo is a collection of information in an organization that is isolated from and not accessible by other parts of the organization. Removing data silos can help you get the right information at the right time so you can make good decisions. And, you can save money by reducing storage costs for duplicate information.

How Do Data Silos Occur?

Data silos happen for three common reasons:

Posted on

Use Materialized Views to Turbo-Charge BI, Not Proprietary Middleware

Query performance has always been an issue in the world of business intelligence (BI), and many BI users would be happy to have their reports load and render quicker. Traditionally, the best way to achieve this performance (short of buying a bigger database) has been to build and maintain aggregate tables at various levels to intercept certain groups of queries to prevent repeat queries of the same raw data. Also, many BI tools pull data out of databases into their own memory, into “cubes” of some sort, and run analyses off of those extracts.

Downsides of Aggregates and Cubes

Both of these approaches have the major downside of needing to maintain the aggregate or cube as new data arrives. In the past, that has been a daily event, but most warehouses are now being stream-fed in near real-time. It’s not practical to continuously rebuild aggregate tables or in-memory cubes every time a new row arrives or a historical row is updated.