What is Apache Arrow Flight, Flight SQL & ADBC?

ODBC (Open Database Connectivity) and JDBC (Java Database Connectivity) have long been the industry standard for connecting databases with analytical tools. These protocols were designed primarily for row-based data processing, which was the dominant paradigm when they were introduced.

However, in today’s analytical landscape, where OLAP workloads and columnar storage formats (such as Apache Parquet and ORC) are widely adopted, these legacy protocols introduce significant inefficiencies:

• Row-Based Data Transfers: ODBC/JDBC require data to be converted from columnar format into rows before transmission, and then back into columns by the receiving system. This serialization/deserialization (serde) overhead can consume 60–90% of transfer time, significantly impacting performance.
• Inefficient Memory Utilization: Since modern analytical databases and lakehouse formats (e.g., Apache Iceberg, Apache Hudi, Delta Lake) use columnar storage, data retrieval through row-based protocols introduces memory inefficiencies that degrade performance.
• Single-Threaded Execution: Traditional database drivers operate sequentially, limiting throughput when transferring large datasets.

Enter Apache Arrow Flight: A Columnar-Native Transfer Protocol

Apache Arrow Flight is a high-performance RPC framework designed specifically for transferring large amounts of columnar data over a network. Unlike ODBC/JDBC, it eliminates the need for intermediate serialization steps, significantly reducing transfer latency and increasing throughput.

Key Benefits of Arrow Flight

• Eliminates Row-Based Serialization: Data remains in its native Arrow columnar format, avoiding costly transformations.
• Parallel Transfers: Flight supports parallel data streaming, leveraging multiple channels to transfer data efficiently.
• Language Agnostic: Clients and servers can communicate using Arrow Flight in various programming languages, such as Python, Java, C++, and Go.
• Built-In Authentication & Security: Flight integrates with modern authentication mechanisms like OAuth and TLS for secure communication.

Arrow Flight SQL: Bringing SQL Operations to Arrow Flight

Apache Arrow Flight SQL extends Arrow Flight by providing a standardized interface for SQL-based interactions with databases. This means that developers can benefit from Arrow’s high-speed data transfers while maintaining a familiar SQL interface. Unlike traditional database protocols, Flight SQL enables direct execution of SQL queries over the high-performance Arrow Flight transport layer, eliminating unnecessary serialization overhead and reducing query latency.

How Arrow Flight SQL Works

Arrow Flight SQL introduces a set of standardized commands for executing SQL queries, managing prepared statements, and retrieving metadata such as table schemas and column details. By using Arrow’s columnar data structures, query results remain in an optimized in-memory format throughout the transfer process, improving efficiency for analytics workloads.

Flight SQL is designed to work across multiple database systems and analytics engines, making it an attractive option for cloud-native applications and distributed query engines. It supports a range of features including:

• Asynchronous and parallel query execution, enabling better scalability and reduced query times.
• Metadata retrieval capabilities, allowing clients to introspect database schemas without expensive queries.
• Interoperability with existing JDBC and ODBC clients, bridging modern and legacy systems for a smooth transition.

Advantages of Arrow Flight SQL

1. Standardization Across Clients: A single Flight SQL driver works across different database systems, reducing the need for database-specific drivers.
2. Massive Performance Gains: Parallel execution of queries and transfers significantly boosts performance compared to ODBC/JDBC.
3. Cross-Language Compatibility: SQL clients can interact with Arrow Flight SQL-enabled databases regardless of the programming language used.
4. Better Interoperability with Legacy Systems: ODBC and JDBC drivers built on Arrow Flight SQL allow for easy migration from legacy protocols.

Apache Arrow Database Connectivity (ADBC)

While Arrow Flight SQL focuses on high-performance SQL-based data transfers between clients and databases, Arrow Database Connectivity (ADBC) serves a different but complementary role. ADBC provides a standardized API for database interactions, making it easier for developers to query and work with databases using Arrow-native data (with/without Flight SQL).

Image Credit: https://arrow.apache.org/blog/2023/01/05/introducing-arrow-adbc/

Key Differences Between Flight SQL and ADBC

• Flight SQL is a transport protocol that enables SQL queries over the high-speed Arrow Flight framework, designed for databases that expose Arrow-native data transfers.
• ADBC is a programming API that abstracts database interactions, allowing applications to interact with databases using a consistent interface, even when the underlying database does not support Flight SQL.
• ADBC can use Flight SQL, but it also works with other database drivers and APIs, providing flexibility in how databases are accessed.

Why ADBC Matters

ADBC addresses key gaps in existing database APIs:

• Standardized Database Access: Provides a common API across different databases, removing the need for database-specific client drivers.
• Multi-Language Support: ADBC is available in C/C++, Go, and Java, with more language implementations on the way.
• Seamless Integration with Arrow Flight: Developers can efficiently fetch Arrow-native data from databases without expensive conversions.
• Flexible for Databases Without Flight SQL: Even if a database does not support Arrow Flight SQL, ADBC allows developers to use Arrow-native operations efficiently.

Adoption of ADBC

ADBC adoption has been increasing across modern data ecosystems as organizations look for a standardized, high-performance way to interface with databases using Arrow-native data structures. Several key projects and platforms have integrated ADBC to enhance data interoperability and performance:

• DuckDB now supports ADBC for clients that needs columnar data transmission.
• dbt supports ADBC through its new Semantic Layer beta, integrating Arrow Flight to let users buid app and integrations on top of this layer.
• Snowflake supports ADBC enabling efficient columnar data exchange without ODBC/JDBC-induced serialization bottlenecks.

Apache Arrow Flight, Flight SQL, and ADBC represent a fundamental shift in how we handle data transfers and database connectivity for analytical workloads. These technologies eliminate the inefficiencies of row-based protocols and unlock new levels of performance and interoperability.

• Arrow Flight optimizes columnar data transport (protocol) over networks.
• Arrow Flight SQL standardizes SQL-based access to Arrow-native databases.
• ADBC creates a unified API for working with databases using Arrow.

As data systems increasingly adopt columnar storage and processing, Arrow Flight and its ecosystem will play a pivotal role in shaping the future of high-performance data analytics.

I presented all about these at Dremio’s Subsurface. Check out below!