feat(arrow): export StringArray and BinaryArray directly

[ClSlaid]

Summary

This PR adds direct export to Arrow StringArray and BinaryArray.

A caller can request one of these Arrow types:

• Utf8
• LargeUtf8
• Binary
• LargeBinary

Vortex builds a VarBinArray for the requested Arrow type. Arrow uses the buffers in the VarBinArray.

The Python API accepts the requested Arrow type:

array.to_arrow_array(arrow_type=pa.string())
vxf.to_arrow(schema=pa.schema([...]))

Vortex still exports StringViewArray and BinaryViewArray when the caller does not request a type or schema.

Direct export paths

These encodings can write directly to the VarBinArray builder:

• FSST
• OnPair
• Zstd
• Dict

The Dict path also supports compressed dictionary values. Examples include Dict<FSST> and Dict<Zstd>.

Sparse and RunEnd use the normal canonical path. The Rust benchmark showed a 55% gain for the Sparse canonical path. The Vortex compressor does not create RunEnd string arrays.

Reason for the change

The old path created a StringViewArray or BinaryViewArray. Arrow then cast the result to a StringArray or BinaryArray.

The cast copied or reordered the value bytes.

The new path creates a StringArray or BinaryArray directly.

Performance

This Python benchmark used 500,000 repeated URL strings. Each result is the median of 10 runs.

Export path	Median
Direct export to StringArray	111.232 ms
Export to StringViewArray, then cast to StringArray	273.663 ms

Direct export was 59.4% faster in this benchmark.

The Rust benchmark covers these layouts and encodings:

• View
• FSST
• Sparse
• Zstd
• Dict
• Dict<FSST>
• Dict<Zstd>
• filtered compressed arrays
• Chunked<FSST>

The Filter cases do not use the direct encoding paths. Chunked<FSST> uses the direct FSST path. Chunked passes the output builder to each child.

AI assistance

I used GPT5.6sol to help develop this change. I reviewed and tested all changes.

Verification

• cargo nextest run -p vortex-arrow -p vortex-fsst -p vortex-onpair -p vortex-runend -p vortex-sparse -p vortex-zstd passed 529 tests.
• cargo nextest run -p vortex-array --no-fail-fast passed 3,003 tests and skipped 1 test.
• cargo clippy -p vortex-arrow -p vortex-sparse -p vortex-runend --all-targets --all-features -- -D warnings passed.
• cargo +nightly fmt --all passed.
• git diff --check passed.
• The targeted Python tests passed 3 tests.
• The Sphinx doctest run passed 271 tests.

Coverage

Rust coverage used LLVM branch instrumentation for vortex-array, vortex-arrow, vortex-fsst, vortex-onpair, vortex-runend, vortex-sparse, and vortex-zstd.

cargo +nightly llvm-cov nextest --branch --all-features \
  -p vortex-array -p vortex-arrow -p vortex-fsst -p vortex-onpair \
  -p vortex-runend -p vortex-sparse -p vortex-zstd \
  --no-fail-fast --json --output-path coverage.json

The original targeted run passed 3,550 tests and skipped 2 tests. After adding focused tests, this command refreshed coverage for the changed vortex-array paths:

cargo +nightly llvm-cov nextest --branch -p vortex-array --no-fail-fast --lcov

The refreshed run passed 3,003 tests and skipped 1 test.

• Changed executable Rust lines: 495 of 547 lines, or 90.49%.
• Changed Rust branch outcomes: 19 of 26 outcomes, or 73.08%.

The changed-line results compare this branch with 57962f4df9af99ebad584bf3e59310e4bf552fc1. They intersect added lines with the LLVM LCOV line and branch records. The report combines the refreshed vortex-array LCOV with the existing LCOV for the unchanged encoding and Arrow crates. This Rust scope does not include benchmarks or the PyO3 crate.

Python branch coverage used the changed array and file test modules.

uv run --all-packages --with coverage coverage run --branch -m pytest \
  vortex-python/test/test_array.py vortex-python/test/test_file.py

The run completed with 19 passed, 1 pre-existing xfail (test_scalar_at_out_of_bounds), and 0 failed.

• vortex/file.py: 32 of 47 statements, or 68.09%.
• vortex/file.py: 0 of 2 branches. Both branches are in unchanged code.
• Changed executable Python statements: 1 of 1, or 100%.
• Changed Python branches: none.

Python coverage does not instrument the Rust code in the extension. The Python tests exercise that code through the public API.

Next steps

Pass the output builder through lazy arrays such as Filter, Slice, and Take. This change can remove more intermediate canonical arrays. Track this work in #7674.

[codspeed-hq]

Merging this PR will degrade performance by 24.19%

❌ 1 regressed benchmark
✅ 1848 untouched benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	copy_nullable[65536]	1 ms	1.4 ms	-24.19%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ClSlaid:feat/direct-arrow-byte-arrays (ef75f15) with develop (b1a8345)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[ClSlaid]

The regression is strange, nowhere can the code be touched.

GPT blames codegen unit difference, it insists code in this patch disturbs LLVM's optimization and thus produced benchmark with different layouts.

I've benchmarked the patch against baseline under 1 CGU. Their results are close, and regression turns out to be solved.

 ┌────────┬───────────┬───────────┬────────┐                               
 │ Group  │ develop   │ #8902     │ Diff   │                                                
 ├────────┼───────────┼───────────┼────────┤                                                   
 │ 1      │ 60.10 µs  │ 60.24 µs  │ +0.23% │                                      
 ├────────┼───────────┼───────────┼────────┤                                     
 │ 2      │ 60.01 µs  │ 60.15 µs  │ +0.23% │                                      
 ├────────┼───────────┼───────────┼────────┤                                 
 │ 3      │ 60.01 µs  │ 60.01 µs  │ 0.00%  │                                   
 ├────────┼───────────┼───────────┼────────┤                                
 │ 4      │ 60.40 µs  │ 60.47 µs  │ +0.12% │                             
 ├────────┼───────────┼───────────┼────────┤                     
 │ 5      │ 60.23 µs  │ 59.98 µs  │ -0.42% │                           
 ├────────┼───────────┼───────────┼────────┤                         
 │ 6      │ 60.11 µs  │ 60.27 µs  │ +0.27% │                     
 ├────────┼───────────┼───────────┼────────┤                      
 │ MID    │ 60.105 µs │ 60.195 µs │ +0.15% │                      
 └────────┴───────────┴───────────┴────────┘ 

It is interesting, but I will keep this patch free of codegen changes.

[robert3005]

Some of the microbenchmarks are sensitive to code changes. I don't think it's related to your change. Thanks for your contribution. I will review it tomorrow!