Arrow Flight is a “new general-purpose client-server framework to simplify high-performance transport of large datasets over network interfaces.” Flight uses gRPC and IPC protocols as its foundation. The structure of Flight primarily focuses on the transmission of Arrow record batches. The protocols for transmitting methods and data use Protobuf outlines, providing compatibility with clients equipped to handle gRPC and Arrow individually, even if they do not support Flight directly. Moreover, additional enhancements fine-tune Flight to reduce performance penalties typically associated with Protobuf use, particularly those related to unnecessary memory duplication.
You can also use Apache Arrow Flight SQL to query a database. Apache Arrow Flight SQL is “a new client-server protocol developed by the Apache Arrow community for interacting with SQL databases that makes use of the Arrow in-memory columnar format and the Flight RPC framework.” Essentially, the Flight SQL clients wrap the underlying Flight client. However, it also provides methods for defining streams of Arrow record batches and how to read them. FlightSQL is useful for writing database-agnostic code for different databases that support it. This more general approach is what this article covers. For example, if you are creating a UI or other user experience that might query different kinds of databases.