Posted on

What is Data Ingestion? The Definitive Guide

What Is Data Ingestion?

Data ingestion is an essential step of any modern data stack. At its core data ingestion is the process of moving data from various data sources to an end destination where it can be stored for analytics purposes. This data can come in multiple different formats and be generated from various external sources (e.g., website data, app data, databases, SaaS tools, etc.)

Why Is Data Ingestion Important?

The data ingestion process is important because it moves data from point A to B. Without a data ingestion pipeline, data is locked in the source it originated in and this isn’t actionable. The easiest way to understand data ingestion is to think of it as a pipeline. In the same way that oil is transported from the well to the refinery, data is transported from the source to the analytics platform. Data ingestion is important because it gives business teams the ability to extract value from data that would otherwise be inaccessible.

Posted on

Why ETL Needs Open Source to Address the Long Tail of Integrations

Over the last year, our team has interviewed more than 200 companies about their data integration use cases. What we discovered is that data integration in 2021 is still a mess.

The Unscalable Current Situation

At least 80 of the 200 interviews were with users of existing ETL technology, such as Fivetran, StitchData, and Matillion. We found that every one of them was also building and maintaining their own connectors even though they were using an ETL solution (or an ELT one — for simplicity, I will just use the term ETL). Why?

Posted on

Benefits of Data Ingestion

Introduction

In the last two decades, many businesses have had to change their models as business operations continue to complicate. The major challenge companies face today is that a large amount of data is generated from multiple data sources. So, data analytics have introduced filters to various data sources to detect this problem. They need analytics and business intelligence to access all their data sources to make better business decisions.

It is obvious that the company needs this data to make decisions based on predicted market trends, market forecasts, customer requirements, future needs, etc. But how do you get all your company data in one place to make a proper decision? Data ingestion consolidates your data and stores it in one place.

Posted on

Why You Should NOT Build Your Data Pipeline on Top of Singer

Singer.io is an open-source CLI tool that makes it easy to pipe data from one tool to another. At Airbyte, we spent time determining if we could leverage Singer to programmatically send data from any of their supported data sources (taps) to any of their supported data destinations (targets).

For the sake of this article, let’s say we are trying to build a tool that can do the following:

Posted on

Python Tutorial for Beginners: Modules Related to DateTime

Python DateTime Modules

In this article, we will look at the Python DateTime module. We will learn how to create the current time, how to calculate a time gap, and how to produce time differences. According to the Python docs:

“The python DateTime module supplies classes for manipulating dates and times in both simple and complex ways.”

So, the Python DateTime modules contain several classes. Let us discuss them one by one.

Posted on

Waking Up the World of Big Data

The term "Big Data" has lost its relevance. The fact remains, though: every dataset is becoming a big data set, whether its owners and users know (and understand) that or not. Big data isn't just something that happens to other people or giant companies like Google and Amazon. It's happening, right now, to companies like yours.

Recently, at Eureka!, our annual client conference, I presented on the evolution of Big Data technologies including the different approaches that support the complex and vast amount of data organizations are now dealing with. In this post, I'll break down some of my presentation and dig into the current state of Big Data, the trends driving its evolution, and one major shift that'll deliver up massive value for companies in the next wave of Big Data's growth.

Posted on

How to Collect Big Data Sets From Twitter

In this post, you’ll learn how to collect data from Twitter, one of biggest sources for big data sets.

You’ll also need to set up a Hadoop cluster and HDFS to store the multi-format data you’ll gather from Twitter. Though we’ll be focusing on one platform only, you can obtain more accuracy if you collect data from other channels as well.

Posted on

Pipe Elasticsearch Data to CSV in PowerShell

The CData Cmdlets Module for Elasticsearch is a standard PowerShell module offering straightforward integration with Elasticsearch. Below, you will find examples of using our Elasticsearch Cmdlets with native PowerShell cmdlets.

Creating a Connection to Your Elasticsearch Data

Set the Server and Port connection properties to connect. To authenticate, set the User and Password properties, PKI (public key infrastructure) properties, or both. To use PKI, set the SSLClientCert, SSLClientCertType, SSLClientCertSubject, and SSLClientCertPassword properties.

Posted on

Writing to a CSV File From Multiple Threads

I was writing document and metadata exporter that reads data from SharePoint and writes it to multiple files. I needed to boost up performance of my exporter and I went with multiple threads pumping out the data from SharePoint. One problem I faced — writing metadata to CSV files from multiple threads in parallel. This blog post shows how to do it using concurrent queues.

This posting uses CsvHelper library to write objects to CSV-files. Last time I covered this library in my blog post Generating CSV-files on .NET.

Posted on

Spring Boot/Batch Tutorial: Integration With HBASE REST API and Data Ingestion

Before reading this article you need some basic knowledge on REST, Spring Boot, and Spring Batch.

This article is focused on how to ingest data using Spring Boot/Batch and the HBase REST API. As Spring for Apache Hadoop project will reach End-Of-Life status on April 5th, 2019, using the REST API with Spring Batch helps us to interact with HBase directly from a Windows environment, so you don't need to deploy your jar to a Unix/Linux environment where your HBase is running.

Posted on

Loading Terabytes of Data From Postgres Into BigQuery

Despite the fact that an ETL task is pretty challenging when it comes to loading big data sets, there’s still the scenario in which you can load terabytes of data from Postgres into BigQuery relatively easily and very efficiently. This is the case when you have a lot of immutable data distributed in tables by some timestamp. For example, a transactions table with a created_at timestamp column. BigQuery and Postgres have great tools in order to do this pretty quickly and conveniently.

Preparing Postgres Tables