Parquet Benchmarks

Goals:

1. Find Parquet features that Daft underperforms on
2. Compare Daft against other frameworks

Setup

Create a new virtual environment and install the dependencies

python -m venv .venv
source .venv/bin/activate
pip install -r benchmark-requirements.txt

Now, install the version of Daft you wish to use for benchmarking (either a released wheel, or if you want, a local build)

pip install daft

Running the benchmarks:

Goal 1: Find Parquet features that Daft underperforms on

pytest benchmarking/parquet/ -m benchmark --benchmark-group-by=group -k daft

Goal 2: Compare Daft against other frameworks

pytest benchmarking/parquet/ -m benchmark --benchmark-group-by=param:path

Check peak memory usage

Ensure that you have pytest-memray installed.

pytest benchmarking/parquet/ -m benchmark --memray