| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[PERF] Lazily import heavy modules to speed up import times (#2826) Introduce lazy imports for heavy modules that are not needed as top-level imports. For example, ray does not need to be a top level import (it should only be imported when using the ray runner or when specific ray data extension types needed. Another example would be UnityCatalogTable, which is a relatively heavy import despite only being needed when using delta lake. Modules to import lazily were determined by the proportion of import time as shown by importtime-output-wrapper -c 'import daft' --format waterfall --depth 25. The list of newly lazily imported modules are: - daft.unity_catalog - fsspec - numpy - pandas - PIL.Image - pyarrow - pyarrow.csv - pyarrow.dataset - pyarrow.fs - pyarrow.json - pyarrow.parquet - ray - ray.data.extensions - xml.etree.ElementTree Uses https://github.com/Eventual-Inc/Daft/pull/2836 in order to defer the import of pyarrow. Additionally, we move all type-checking-only module imports into type checking blocks. With these changes, import times go from roughly 0.6-0.7s to ~0.045s (~13-15x faster). --------- Co-authored-by: Sammy Sidhu <samster25@users.noreply.github.com> | 2 年前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 2 年前 | ||
| 9 个月前 |