chore: create in-memory scans using rust arrow arrays (#4005) ## Summary This PR adds the ability to create daft recordbatches and micropartitions from arrow arrays, to ultimately use arrow arrays as an in-memory scan sources. This is much like how the python DataFrame.from_ methods work, but using (almost) all rust things. The runner still requires python so we have a PyMicroPartitionSet which implements the PartitionSet ABC so it can be used with the runner's put into cache method. Once we fully lower to rust, we can remove the bridge. ## Related Issues #3543 ## Changes Made The overall goal is to have a pure-rust Logical in-memory scan from arrow arrays. * added from_arrow to rust RecordBatch * added from_arrow to rust MicroPartition * added from_arrow in daft-sql (will be used for SHOW statement) * added from_micropartitions in daft-sql Ultimately will move the last two methods to a more appropriate place, but want to more comprehensively evaluate the dep tree. ## Checklist - [x] All tests have passed (make test locally) - [n/a] Documented in API Docs - [n/a] Documented in User Guide - [n/a] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [n/a] Documentation builds and is formatted properly (tag [@ccmao1130](https://github.com/ccmao1130) for docs review)
1 年前
README
daft-scan
This crate is used to create scan operators in logical plans.
Contributing
Suppose your format is jsonl - you should add two things.
Export top-level scan_jsonl which takes all arguments.