Posted on

Here’s How You Can Purge Big Data From Unstructured Data Lakes

Without a doubt, big data is becoming the biggest data with the passage of time. It’s going above and beyond. Here are some pieces of evidence as to why I said it. 

According to the big data and business analytics report from Statista,  the global cloud data IP traffic will reach approximately 19.5 zettabytes in 2021. Moreover, the big data market will strike a figure of 274.3 billion US dollars with a five-year compound annual growth rate (CAGR) of 13.2% by 2022.  Plus, Forbes predicted that over 150 trillion gigabytes or 150 zettabytes of real-time data will be required by the year 2025. Also, Forbes found that more than 95% of the companies need some assistance for the management of unstructured data, while 40% of the organizations affirmed that they need to deal with big data more habitually. 

Posted on

What Happened to Hadoop? What Should You Do Now?

Apache Hadoop emerged on the IT scene in 2006 with the promise to provide organizations with the capability to store an unprecedented volume of data using commodity hardware. This promise not only addressed the size of the data sets, but also the type of data, such as data generated by IoT devices, sensors, servers, and social media that businesses were increasingly interested in analyzing. The combination of data volume, velocity, and variety was popularly known as Big Data.

Schema-on-read played a vital role in the popularity of Hadoop. Businesses thought they no longer had to worry about the tedious process of defining which tables contained what data and how are they connected to each other — a process that took months and not a single data warehouse query could be executed before it was complete. In this brave new world, businesses could store as much data as they could get their hands on in Hadoop-based repositories known as data lakes and worry about how it is going to be analyzed later.

Posted on

Reading SUDO Logs With Apache NiFi

Log, Log, Log

Sudo logs have a lot of useful information on hosts, users, and auditable actions that may be useful for cybersecurity, capacity planning, user tracking, data lake population, user management, and general security.

Symbol Model 1

Apache NiFi