| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: Add WARC reader (#3871) Adds a reader for .warc and .warc.gz files. Currently optimized for reading from S3. Some numbers: - Downloading a single common crawl file from S3, e.g. s3://commoncrawl/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz, takes ~16.5s. - Gunzipping this file and processing it with fastwarc takes ~33s. - With swordfish, collecting that same file as a daft dataframe takes ~20s on an m7g.4xlarge instance. - With swordfish, collecting 10 common crawl files take ~3min 12s. - With swordfish, processing (read then sum on content length) 1 common crawl file takes ~20s. - With swordfish, processing 2 common crawl files still takes ~20s. - With swordfish, processing 10 common crawl files takes ~40s. This is because we've set the max number of parallel reads to 8. So 10 scan tasks take 2x20s to read. If we increase the max number of parallel reads to 10, the runtime drops to ~30s. - With ray, collecting 1 file takes ~25s. - **Unfortunately, with ray, collecting 10 files caused the instance to become unresponsive.** Followup work: - Extracting fields from the warc_headers json is not very fast. We can do better here by allowing users to specify the metadata headers that they want to extract. --------- Co-authored-by: Sammy Sidhu <sammy.sidhu@gmail.com> Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain> | 1 年前 | |
[CHORE] auto-fix prefer Self over explicit type (#2908) | 2 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 年前 | ||
| 2 年前 |