This document outlines the process of extracting table structures from SQL Server databases, converting them to JSON format, keeping them in Azure Studio, and then loading them into BigQuery using Cloud Data Fusion. The data types of the SQL Server data are converted to their corresponding BigQuery data types to ensure compatibility and accurate data analysis.
The process involves creating a JSON file with the converted data types and adding additional metadata columns. This file is then used in the Terraform code to define the infrastructure resources. Azure DevOps is integrated into the project to automate the infrastructure provisioning and data pipeline deployment.