Using the Airflow ShortCircuitOperator to Stop Bad Data from Reaching ETL Pipelines

I'm a huge fan of Apache Airflow and how the open-source tool enables data engineers to scale data pipelines by more precisely orchestrating workloads.

But what happens when Airflow testing doesn’t catch all of your bad data? What if “unknown unknown” data quality issues fall through the cracks and affect your Airflow jobs?

Build an Advanced RAG App: Query Rewriting
No categories
In the last article, I established the basic architecture for a basic RAG app. In case you missed that, I recommend that you first read that article. That will set the base from which we can improve our RAG system. Also in that last article, I listed s... […]
Contexts in Go: A Comprehensive Guide
No categories
Contexts in Go provide a standard way to pass metadata and control signals between goroutines. They are mainly used to manage task execution time, data passing, and operation cancellation. This article covers different types of contexts in Go and examp... […]
Next-Gen Lie Detector: Stack Selection
No categories
The first lie detector which relied on eye movement appeared in 2014. The Converus team together with Dr. John C. Kircher, Dr. David C. Raskin, and Dr. Anne Cook launched EyeDetect — a brand-new solution to detect deception quickly and accurately. This... […]
Applying the Pareto Principle To Learn a New Programming Language
No categories
In this article, I will discuss how you can apply the Pareto principle to quickly learn a new programming language and start solving real-world problems while you develop a deeper understanding of the programming language. What Is the Pareto Principle?... […]
Enhance IaC Security With Mend Scans
No categories
Whether on the cloud or setting up your AIOps pipeline, automation has simplified the setup, configuration, and installation of your deployment. Infrastructure as Code(IaC) especially plays an important role in setting up the infrastructure. With IaC t... […]