From a data engineer's point of view, financial risk management is a series of data analysis activities on financial data. The financial sector imposes its unique requirements on data engineering. This post explains them with a use case of Apache Doris and provides a reference for what you should take into account when choosing an OLAP engine in a financial scenario.
Data Must Be Combined
The financial data landscape is evolving from standalone to distributed, heterogeneous systems. For example, in this use case scenario, the fintech service provider needs to connect the various transaction processing (TP) systems (MySQL, Oracle, and PostgreSQL) of its partnering banks. Before they adopted an OLAP engine, they were using Kettle to collect data. The ETL tool did not support join queries across different data sources, and it could not store data. The ever-enlarging data size at the source end was pushing the system toward latency and instability. That's when they decided to introduce an OLAP engine.