Daft/daft/runners · xuanwu/Daft

GGitHubfeat: Support configuring Conda Env for Class UDFs in Flotilla (#5117 )

文件	最后提交记录	最后更新时间
__init__.py	chore: Make the runner a separate global singleton (#5185)	1 年前
flotilla.py	chore: bump `mypy` and `ruff` in pre-commit (#5836) ## Changes Made Bump `mypy` to 1.19.1 and `ruff` to 0.14.10 and resolve pre-commit errors. Rationale: my local `ruff` version is much higher than that in the project `pre-commit`, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" -->	9 个月前
native_runner.py	feat: Use JSON Serialization for Plans in Subscribers (#5709) ## Changes Made Use our JSON plan serialization instead of the NodeInfo struct or our ASCII repr for sending plans to subscribers, including the dashboard. This will make it easier to support subscribers in Flotilla, since we don't need to use Swordfish-specific properties like whats in NodeInfo. Plus, the dashboard can now use / visualize additional fields in specific operators, like Project expressions. Tagging @samstokes and @ohbh for visibility on consuming the optimized plan. Feel free to review too	9 个月前
partitioning.py	chore: bump `mypy` and `ruff` in pre-commit (#5836) ## Changes Made Bump `mypy` to 1.19.1 and `ruff` to 0.14.10 and resolve pre-commit errors. Rationale: my local `ruff` version is much higher than that in the project `pre-commit`, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" -->	9 个月前
profiler.py	feat: Add pow expression (#5237) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues #4704 <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com>	10 个月前
progress_bar.py	fix: use single consolidated progress bar in Jupyter notebooks (#5774) ## Problem Multiple tqdm progress bars with `position` parameter don't render correctly in Jupyter notebooks - they spam output instead of updating in place. Each progress update would append a new line, causing excessive output. ## Solution Instead of creating multiple tqdm progress bars (one per pipeline node), consolidate all pipeline statuses into a single tqdm progress bar separated by `\|`. ### Changes: - Python (`daft/runners/progress_bar.py`): Modified `SwordfishProgressBar` to use a single combined progress bar - Store messages per pipeline and combine them into one line - Use `leave=True` to avoid output corruption when clearing (Jupyter doesn't handle tqdm clearing reliably) - Print empty line after close for cleaner separation - Show elapsed time with `[{elapsed}]` format - Rust (`progress_bar.rs`): Call Python's `close()` method from `finish()` (was previously a no-op) ## Before `🗡️ 🐟 Parquet Scan: 00:00 0 rows out, 128.00 KiB bytes read 🗡️ 🐟 Limit 100: 00:00 🗡️ 🐟 Limit 100: 00:00 100 rows in, 100 rows out d 🗡️ 🐟 Project: 00:00 🗡️ 🐟 Project: 00:00 100 rows in, 100 rows out ... (many more spam lines)` ## After `[00:11] 🗡️ 🐟 Parquet Scan: 100 rows out, 892.48 KiB bytes read \| 🗡️ 🐟 Limit 100: 100 rows in, 100 rows out \| 🗡️ 🐟 Project: 100 rows in, 100 rows out \| ... [00:00] 🗡️ 🐟 InMemorySource: 100 rows out \| 🗡️ 🐟 Filter: 100 rows in, 5 rows out \| ... Out of 100 products analyzed: - 5 are made of wood - 95 are not made of wood` ## Notes - This only affects Jupyter notebooks - terminal output still uses the Rust `indicatif` progress bars which work correctly	9 个月前
ray_compat.py	feat: Support configuring Conda Env for Class UDFs in Flotilla (#5117) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> #5104 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) Signed-off-by: plotor <zhenchao.wang@hotmail.com>	8 个月前
ray_runner.py	chore: bump `mypy` and `ruff` in pre-commit (#5836) ## Changes Made Bump `mypy` to 1.19.1 and `ruff` to 0.14.10 and resolve pre-commit errors. Rationale: my local `ruff` version is much higher than that in the project `pre-commit`, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" -->	9 个月前
runner.py	chore: Remove the PyRunner (#4458)	1 年前
runner_io.py	feat: support glob multiple path (#4811) ## Changes Made Fix the doc of `daft.from_glob_path`, and support passing multiple paths, even through we can passed a regex pattern, but sometimes it's hard to use a single regex pattern to cover all case, so it's better to support more glob paths, but need to ensure the result doesn't contains duplicate file path if the glob paths have intersection. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist The documents need to be refresh, the `from_glob_path` didn't return `type` column. - [ x] Documented in API Docs (if applicable) - [ x] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: wuzhiping <wuzhiping.007@bytedance.com>	1 年前