With each day, enterprises increasingly rely on data to make decisions. This is true regardless of their industry: finance, media, retail, logistics, etc. Yet, the solutions that provide data to dashboards and ML models continue to grow in complexity. This is due to several reasons, including:
- The ability to process data from diverse sources at a low cost
- An explosion in the availability and variety of data tools (impacting collaboration and decision making, beyond technical work)
- Tight dependencies among data assets managed by different departments within companies
This need to run complex data pipelines with minimum rates of error in such modern environments has led to the rise of a new role: the Data Reliability Engineer. Data Reliability Engineering (DRE) addresses data quality and availability problems. Comprising practices from data engineering to system operations, DRE is emerging as its own field within the broader data domain.