Skip to content

fix(bool): trim oversized backing buffer views#8908

Open
jiaqizho wants to merge 1 commit into
vortex-data:developfrom
jiaqizho:fix-oversized-bool-array
Open

fix(bool): trim oversized backing buffer views#8908
jiaqizho wants to merge 1 commit into
vortex-data:developfrom
jiaqizho:fix-oversized-bool-array

Conversation

@jiaqizho

Copy link
Copy Markdown

Rationale for this change

Bool arrays constructed from sliced bitmaps can retain buffer views that extend beyond their logical byte range. Serialization writes the exposed buffers in full, which can unnecessarily increase file size and retain logically excluded suffix data.

This affects the release path of BoolArray::new_unchecked and callers that provide an oversized BufferHandle.

What changes are included in this PR?

Normalize unchecked host bit buffers with shrink_offset, and trim oversized host or device buffer handles to the byte range required by the Bool metadata.

Both operations create zero-copy views and preserve the underlying allocation, device residency, alignment, and ownership model.

What APIs are changed? Are there any user-facing changes?

No public API signatures are changed. Bool array constructors now expose only the backing bytes required by their logical offset and length.

Bool arrays constructed from sliced bitmaps can retain buffer views that extend beyond their logical byte range. Serialization writes the exposed buffers in full, which can unnecessarily increase file size and retain logically excluded suffix data.

This affects the release path of `BoolArray::new_unchecked` and callers that provide an oversized `BufferHandle`.

Normalize unchecked host bit buffers with `shrink_offset`, and trim oversized host or device buffer handles to the byte range required by the Bool metadata.

Both operations create zero-copy views and preserve the underlying allocation, device residency, alignment, and ownership model.

No public API signatures are changed. Bool array constructors now expose only the backing bytes required by their logical offset and length.

Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
@joseph-isaacs

Copy link
Copy Markdown
Contributor

If this problem does exist it would be great to fix this. Does it show up in a serialised file (on disk)?

@codspeed-hq

codspeed-hq Bot commented Jul 23, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 21.19%

❌ 1 regressed benchmark
✅ 1846 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decode_varbin[(1000, 8)] 27.1 µs 34.4 µs -21.19%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jiaqizho:fix-oversized-bool-array (3a8c077) with develop (30091f2)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@jiaqizho

Copy link
Copy Markdown
Author

If this problem does exist it would be great to fix this. Does it show up in a serialised file (on disk)?

Hi @joseph-isaacs

For context, I originally encountered a similar issue while using a custom layout rather than Vortex's default layout.

I reproduced the issue on master using the following minimal writer strategy. The direct FlatLayoutStrategy setup is intended only to isolate the underlying behavior from my custom layout:

  session
      .write_options()
      .with_strategy(Arc::new(FlatLayoutStrategy::default()))
      .write(&mut file, array.to_array_stream())
      .await?;

The input contains 8,192 nullable Boolean rows. The logical values and validity bitmaps are 1,024 bytes each, but each is backed by a 65,536-byte BufferHandle.

Results:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants