| ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 |
| chore: Vendor parquet-format-safe dependency (#4544) ## Changes Made Vendor [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe/tree/main) so that we can patch some bugs. This is a stop gap measure. As we migrate away from parquet2 to parquet-rs we can remove this dependency. ## Related Issues Step 1 of 2 to fix https://github.com/Eventual-Inc/Daft/issues/4515 | 1 年前 |
| fix: Resolve mismatch with Thrift compact protocol (#4545) ## Changes Made The [Thrift compact protocol](https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md) is used for Parquet file metadata. [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe) and other Rust implementations of the protocol eagerly read string/binary fields as UTF-8. However, based on the protocol which states that > Strings are first encoded to UTF-8, and then send as binary it cannot be known upfront, without using the schema to disambiguate the field type, whether a field is a string or a binary. This means that when the field is actually a binary field and contains invalid UTF-8, Rust libraries error out when reading the field with File out of specification: Invalid thrift: bad data. To fix this, we patch the protocol implementation to correctly interpret string/binary fields as binary. ## Related Issues Closes #4515 | 1 年前 |
| chore: Vendor parquet-format-safe dependency (#4544) ## Changes Made Vendor [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe/tree/main) so that we can patch some bugs. This is a stop gap measure. As we migrate away from parquet2 to parquet-rs we can remove this dependency. ## Related Issues Step 1 of 2 to fix https://github.com/Eventual-Inc/Daft/issues/4515 | 1 年前 |
| chore: Vendor parquet-format-safe dependency (#4544) ## Changes Made Vendor [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe/tree/main) so that we can patch some bugs. This is a stop gap measure. As we migrate away from parquet2 to parquet-rs we can remove this dependency. ## Related Issues Step 1 of 2 to fix https://github.com/Eventual-Inc/Daft/issues/4515 | 1 年前 |
| chore: Vendor parquet-format-safe dependency (#4544) ## Changes Made Vendor [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe/tree/main) so that we can patch some bugs. This is a stop gap measure. As we migrate away from parquet2 to parquet-rs we can remove this dependency. ## Related Issues Step 1 of 2 to fix https://github.com/Eventual-Inc/Daft/issues/4515 | 1 年前 |
| chore: Vendor parquet-format-safe dependency (#4544) ## Changes Made Vendor [parquet-format-safe](https://github.com/jorgecarleitao/parquet-format-safe/tree/main) so that we can patch some bugs. This is a stop gap measure. As we migrate away from parquet2 to parquet-rs we can remove this dependency. ## Related Issues Step 1 of 2 to fix https://github.com/Eventual-Inc/Daft/issues/4515 | 1 年前 |