0.89.0 (Draft)

Draft — GitHub Release

Changes

⚠️ Breaks

• Blocked FoR: store a global reference scalar as metadata rather than child (#10222) @mhk197

• Move Shared array out of vortex-array into vortex-layout (#10495) @robert3005

✨ Features

• Optimize list_contains with prepared constant sets (#10052) @robert3005

• feat: expose Spark scan schema and pushed predicates (#9956) @xiaoh1024

• Blocked BitPacked: Add Compressor support via BitPackingScheme mode (#10245) @mhk197

• Support Polars 2 (#10398) @robert3005

• feat(layout): compile evals, row indices, and takes to pipelines (#10393) @joseph-isaacs

• feat(layout): compile segment scans, filters, concatenations, and packs to pipelines (#10392) @joseph-isaacs

• feat(layout): add push-based pipelines and the scan that runs them (#10391) @joseph-isaacs

• Support filters with limit in ScanBuilder (#8766) @AdamGS

• Add more scalar literal types to the Python and JNI bindings (#10079) @robert3005

• feat(array): support fill null for UTF-8 and binary arrays (#10044) @1fanwang

• feat(python): support row indices in VortexFile.to_arrow (#10373) @joseph-isaacs

🚀 Performance

• perf: exit compute_referenced_values_mask early once every value is referenced (#10423) @joseph-isaacs

• Decode nullable RunEnd validity by clearing null streaks (#10487) @joseph-isaacs

• Drop DuckDB file scan state outside the multi-file lock (#10486) @robert3005

• perf(layout): skip chunks without selected rows (#10491) @robert3005

• Bound DashMap shard counts for per-reader caches (#10485) @robert3005

• Blocked BitPacked: fuse FoR decode over blocked children (#10251) @mhk197

• perf(mask): read intersect_by_rank rank bits at an absolute offset (#10480) @mightsleep

• avx2 branch for bool take kernel (#10385) @myrrc

• perf: enable POPCNT in the BMI2, AVX2 and AVX-512 kernels (#10456) @mightsleep

• feat(compressor): measure inputs and blocks by the bytes their rows reference (#10445) @joseph-isaacs

• perf(mask): SVE2 BDEP and a branch-free portable deposit for intersect_by_rank (#10168) @mightsleep

• feat(array): add precision option to uncompressed_size_in_bytes (#10439) @joseph-isaacs

• perf(layout): lower zoned pruning predicates once (#10428) @robert3005

• Optimize SearchSorted to reuse probes and avoid repeated context creation (#9904) @joseph-isaacs

• Speed up scans with a roaring row selection (#10359) @joseph-isaacs

• perf: batch fragmented CASE branch execution (#9820) @gatesn

🐛 Bug Fixes

27 changes

• fix(python): translate Polars struct field access (#10194) @danking

• Struct reader partitioning respects struct validity (#10449) @myrrc

• Update rustls to 0.23.46 (#10492) @palaska

• fix: omit zone-map stats that a nullable struct’s nulls would corrupt (#10413) @joseph-isaacs

• Fix DataFusion expression conversion and pushdown semantics (#9785) @AdamGS

• fix(array): scale decimal-to-float scalar casts across the whole scale domain (#9883) @jackylee-ch

• Only gather sparse takes from OnPair and FSST (#10421) @joseph-isaacs

• fix(layout): stop dict writer looping on an oversized value (#10400) @vortex-claude[bot]

• fix(file): validate metadata segments against the file size too (#9881) @jackylee-ch

• fix(python): support Polars datetime literals and timezone expressions (#10183) @danking

• fix: report a canonical NaN for a float sum, not the platform’s default (#10122) @gerchowl

• fix(python): translate Polars fill_null with expression fills (#10192) @danking

• feat(expr): add timestamp timezone replacement (#10182) @danking

• fix(python): cast scan indices to u64 in read_url and to_array (#10277) @jackylee-ch

• fix(cloud): parse HF ALLOW_HTTP the way object_store does (#10278) @jackylee-ch

• Fix nullability of bool take (#10411) @robert3005

• Expression optimisation rules don’t introduce irreducible children (#9625) @robert3005

• fix(python): translate Polars boolean NOT expressions (#10190) @danking

• fix(python): translate Polars is_null expressions (#10189) @danking

• fix(python): translate Polars binary scalar literals (#10188) @danking

• fix(python): translate Polars date literals (#10185) @danking

• fix(python): support Polars is_not_null predicates (#10184) @danking

• fix(python): support unsigned Polars scalar literals (#10170) @danking

• fix(python): translate Polars is_between expressions (#10191) @danking

• fix(python): translate Polars decimal literals (#10187) @danking

• fix(python): translate Polars conditional expressions (#10193) @danking

• Canonical dict execute handles edge cases in Decimal/Bool logic (#10098) @robert3005

🧰 Maintenance

18 changes

• Use mimalloc as the global allocator in every benchmark binary (#10483) @joseph-isaacs

• Strengthen test assertions and remove fuzz reference checks (#10465) @robert3005

• Blocked BitPacked: add fastlanes.bitpacked.v2 serialization (#10244) @mhk197

• [bench] sparse bitpacked filter and take (#9900) @robert3005

• Update lance benchmark dependencies (major) (#10436) @renovate[bot]

• Update gradle/actions digest to 3f5f9ad (#10434) @renovate[bot]

• Update CodSpeedHQ/action digest to f842b9a (#10432) @renovate[bot]

• Update dtolnay/rust-toolchain digest to 686976e (#10433) @renovate[bot]

• Lock file maintenance (#10438) @renovate[bot]

• Update taiki-e/install-action digest to 1168f68 (#10435) @renovate[bot]

• Update Rust crate zip to v9 (#10437) @renovate[bot]

• Update actions/upload-artifact digest to cf430e0 (#10431) @renovate[bot]

• Update actions/setup-node digest to 949feb2 - autoclosed (#10430) @renovate[bot]

• Update actions/download-artifact digest to 9000827 (#10429) @renovate[bot]

• Return owned results from the optimizer and log every parent rewrite (#10407) @robert3005

• Expose slots, dtype, len and is_empty on TypedArrayRef (#10406) @robert3005

• Don’t require boxing ChunkedArray chunk iterators (#9918) @robert3005

• Cache aggregate results on the requested array, not its canonical form (#10380) @robert3005