feat: Add array data type support by charlesdong1991 · Pull Request #433 · apache/fluss-rust

charlesdong1991 · 2026-03-07T16:02:37Z

Purpose

Linked issue: close #386

Brief change log

Add end-to-end ARRAY column support in fluss-rust

Tests

Yes all tests are passed locally

charlesdong1991 · 2026-03-07T16:06:56Z

crates/fluss/src/row/compacted/compacted_row_reader.rs

                        let (val, next) = reader.read_long(cursor);
-                        let decimal = Decimal::from_unscaled_long(val, precision, scale)
-                            .expect("Failed to create decimal from unscaled long");
+                        let decimal =


i think we should convert all expect to typed errors so malformed array payload handling does not leave panic backdoor

charlesdong1991 · 2026-03-07T16:10:02Z

crates/fluss/src/row/compacted/compacted_key_writer.rs

+        // silently widen key semantics.
+        if matches!(
+            field_type,
+            DataType::Array(_) | DataType::Map(_) | DataType::Row(_)


this applies to all array/map/row, so i understand this PR only adds Array, but i think better to add all of them.
if there is objection, i can remove

luoyuxia · 2026-03-08T02:31:21Z

@charlesdong1991 Thanks for the great pr. I'll find some time to review

Copilot

Pull request overview

Adds end-to-end ARRAY column support to the Fluss Rust client, including a Java-compatible binary array representation and integration across row encoders/decoders, Arrow interop, and public APIs.

Changes:

Introduces FlussArray / FlussArrayWriter (BinaryArray layout) and wires it into Datum, InternalRow, and runtime field/value writers.
Adds compacted row read/write support for arrays (length-prefixed BinaryArray bytes) and rejects ARRAY as a key column type.
Extends Arrow conversion logic and updates user/docs + tests for primitive, nullable, empty, and nested arrays.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
website/docs/user-guide/rust/data-types.md	Documents `ARRAY<T>` type mapping and how to construct/read arrays.
website/docs/user-guide/rust/api-reference.md	Adds `InternalRow::get_array` and documents `FlussArray` API.
crates/fluss/src/row/mod.rs	Exposes `binary_array` module and re-exports `FlussArray`; adds `InternalRow::get_array`.
crates/fluss/src/row/field_getter.rs	Adds `Array` field getter + tests.
crates/fluss/src/row/encode/compacted_key_encoder.rs	Adds test ensuring array types are rejected in key encoding.
crates/fluss/src/row/datum.rs	Adds `Datum::Array` and Arrow ListBuilder append/conversion logic.
crates/fluss/src/row/compacted/compacted_row_writer.rs	Adds `write_array` (delegates to length-prefixed bytes).
crates/fluss/src/row/compacted/compacted_row_reader.rs	Makes deserialization fallible and adds array decoding via `FlussArray::from_bytes`.
crates/fluss/src/row/compacted/compacted_row.rs	Propagates fallible deserialization and adds `InternalRow::get_array` + array tests.
crates/fluss/src/row/compacted/compacted_key_writer.rs	Explicitly rejects complex key types and adds `write_array` to the delegated writer surface.
crates/fluss/src/row/column.rs	Implements `ColumnarRow::get_array` by converting Arrow `ListArray` → `FlussArray`; adds tests.
crates/fluss/src/row/binary_array.rs	New Java-compatible binary array implementation + writer + tests.
crates/fluss/src/row/binary/binary_writer.rs	Adds `BinaryWriter::write_array` and `InnerValueWriter::Array`.
crates/fluss/src/record/arrow.rs	Adds Arrow `List` builder support and `from_arrow_type` mapping for list element conversion.
bindings/cpp/src/types.rs	Propagates `Datum::Array` handling in type resolution and row ownership conversion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/row/binary_array.rs

crates/fluss/src/row/datum.rs

crates/fluss/src/row/compacted/compacted_row_reader.rs

crates/fluss/src/row/compacted/compacted_key_writer.rs

crates/fluss/src/row/column.rs

crates/fluss/src/row/binary_array.rs

fresh-borzoni

@charlesdong1991 LGTM overall, left minor comments
If you fixed smth in the meantime - good, sorry for the noise, as I started reviewing before the new code :)

crates/fluss/src/row/datum.rs

crates/fluss/src/row/column.rs

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/binary_array.rs

bindings/cpp/src/types.rs

crates/fluss/src/row/datum.rs

charlesdong1991 · 2026-03-08T14:48:21Z

Hi @fresh-borzoni thanks a lot!
It would be great if you can give another round of reviews since i made some changes during your review ^^

leekeiabstraction

Thank you for the PR! Left some comments. PTAL

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

fresh-borzoni

@charlesdong1991 TY, looked through again, left comments
PTAL

crates/fluss/src/row/binary_array.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

crates/fluss/src/row/column.rs

fresh-borzoni · 2026-03-09T01:23:07Z

crates/fluss/src/row/binary_array.rs

+        (offset, size)
+    }
+
+    pub fn get_string(&self, pos: usize) -> Result<&str> {


why do we return Result in half of the getters and half of the methods go unchecked?
I doubt that Java is the same half-way, pls, check

it seems java uniformly uncheck.
and now in rust, it is because getter that involve operations that are inherently fallible (e.g. length bound checks etc), but primitive getters only do fixed-size slice conversion.

i agree we shouldn't do half-way, so either make all getters return Result with bounds checking or we also let primitive getters panic on corrupted payload so it matches java

i would prefer to have all getters return Result, wdyt? @fresh-borzoni

@charlesdong1991
Let's use Result for now, if we need to add infallible variants later - it would be non-breaking change.
So it's the safe unblocking way, to make API consistent and leave some room for improvement if needed, just add TODO or comment about this.

thanks @fresh-borzoni updated, PTAL

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/mod.rs

crates/fluss/src/record/arrow.rs

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

website/docs/user-guide/rust/data-types.md

crates/fluss/src/row/compacted/compacted_key_writer.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

fresh-borzoni

@charlesdong1991 TY, Looked through, LGTM overall, one nit comment 👍

crates/fluss/src/row/compacted/compacted_row.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

charlesdong1991 · 2026-03-11T19:57:33Z

Thanks both! PTAL @fresh-borzoni @leekeiabstraction @luoyuxia

leekeiabstraction · 2026-03-11T20:35:58Z

Test case changes looks good to me but diff seems larger, seems to be squashed with is retriable changes from another PR? Not sure if it will conflict. LGTM otherwise.

charlesdong1991 added 2 commits March 7, 2026 16:59

add array data type support

5a8ca2e

add docs

e35adce

charlesdong1991 commented Mar 7, 2026

View reviewed changes

luoyuxia requested a review from Copilot March 8, 2026 02:29

Copilot started reviewing on behalf of luoyuxia March 8, 2026 02:29 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

charlesdong1991 added 3 commits March 8, 2026 11:45

resolve conflicts

a8fdac0

address comments

8cacf63

avoid intermediate record batch in get-array

8208023

fresh-borzoni reviewed Mar 8, 2026

View reviewed changes

Address comments

c29c5ad

leekeiabstraction reviewed Mar 8, 2026

View reviewed changes

crates/fluss/src/record/arrow.rs Outdated Show resolved Hide resolved

crates/fluss/src/record/arrow.rs Outdated Show resolved Hide resolved

crates/fluss/src/row/encode/compacted_key_encoder.rs Show resolved Hide resolved

fresh-borzoni reviewed Mar 9, 2026

View reviewed changes

charlesdong1991 added 2 commits March 9, 2026 20:39

Merge remote-tracking branch 'upstream/main' into issue-386

44336c2

address review comments

750799c

charlesdong1991 requested a review from Copilot March 9, 2026 20:17

Copilot started reviewing on behalf of charlesdong1991 March 9, 2026 20:17 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

charlesdong1991 added 3 commits March 9, 2026 21:41

address comments and improve doc

83c8093

address comments

d0b7cde

add todo

66359e4

fresh-borzoni reviewed Mar 11, 2026

View reviewed changes

crates/fluss/src/row/compacted/compacted_row.rs Show resolved Hide resolved

leekeiabstraction reviewed Mar 11, 2026

View reviewed changes

crates/fluss/src/row/encode/compacted_key_encoder.rs Show resolved Hide resolved

address comments

ca96551

Conversation

charlesdong1991 commented Mar 7, 2026

Purpose

Brief change log

Tests

Uh oh!

charlesdong1991 Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 8, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 7, 2026 •

edited

Loading

fresh-borzoni left a comment •

edited

Loading