fix(bool): trim oversized backing buffer views#8908
Conversation
Bool arrays constructed from sliced bitmaps can retain buffer views that extend beyond their logical byte range. Serialization writes the exposed buffers in full, which can unnecessarily increase file size and retain logically excluded suffix data. This affects the release path of `BoolArray::new_unchecked` and callers that provide an oversized `BufferHandle`. Normalize unchecked host bit buffers with `shrink_offset`, and trim oversized host or device buffer handles to the byte range required by the Bool metadata. Both operations create zero-copy views and preserve the underlying allocation, device residency, alignment, and ownership model. No public API signatures are changed. Bool array constructors now expose only the backing bytes required by their logical offset and length. Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
|
If this problem does exist it would be great to fix this. Does it show up in a serialised file (on disk)? |
Merging this PR will degrade performance by 21.19%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
For context, I originally encountered a similar issue while using a custom layout rather than Vortex's default layout. I reproduced the issue on master using the following minimal writer strategy. The direct FlatLayoutStrategy setup is intended only to isolate the underlying behavior from my custom layout: The input contains 8,192 nullable Boolean rows. The logical values and validity bitmaps are 1,024 bytes each, but each is backed by a 65,536-byte BufferHandle. Results:
|
Rationale for this change
Bool arrays constructed from sliced bitmaps can retain buffer views that extend beyond their logical byte range. Serialization writes the exposed buffers in full, which can unnecessarily increase file size and retain logically excluded suffix data.
This affects the release path of
BoolArray::new_uncheckedand callers that provide an oversizedBufferHandle.What changes are included in this PR?
Normalize unchecked host bit buffers with
shrink_offset, and trim oversized host or device buffer handles to the byte range required by the Bool metadata.Both operations create zero-copy views and preserve the underlying allocation, device residency, alignment, and ownership model.
What APIs are changed? Are there any user-facing changes?
No public API signatures are changed. Bool array constructors now expose only the backing bytes required by their logical offset and length.