| feat: make track deletes work across execution (#5517) The track deletes features currently assume the entire dataset is being fetched/saved within a single execution. Which means it isn't compatible with checkpoints and fetching/saving across multiple executions. To solve this problem, this PR is introducing a way to explicitly define the start and the end of the track deletes interval, storing the starting point in a special checkpoint that survive across executions, instead of assuming the start is always the beginning of the current execution cc @bastienbeurier to agree on the naming `trackDeletesStart/trackDeletesEnd` ex: ``` exec: async (nango) => { await nango.trackDeletesStart('MyModel'); ... await nango.batchSave([...], 'MyModel'); ... const deleted = await nango.trackDeletesEnd('MyModel'); } ``` <!-- Summary by @propel-code-bot --> --- The runner stores a per-model delete-window checkpoint keyed off the sync checkpoint and, when the window closes, uses it to invoke deletion of outdated records before clearing the checkpoint. <details> <summary><strong>Key Changes</strong></summary> • Added `trackDeletesStart`/`trackDeletesEnd` to `NangoSyncBase` and `NangoSyncRunner`, with per-model checkpoint keys and `deleteOutdatedRecords` using stored `syncJobId` • Refactored `Checkpointing` in `packages/runner/lib/sdk/checkpointing.ts` to manage per-key state and accept `key` arguments for checkpoint operations • Expanded CLI parser/compiler validations to track `trackDeletes` calls per model and enforce ordering around `batchSave` • Updated docs and examples to use `trackDeletesStart`/`trackDeletesEnd` and mark `deleteRecordsFromPreviousExecutions` as deprecated • Reset behavior in `packages/shared/lib/clients/orchestrator.ts` now hard-deletes all checkpoints with a key prefix </details> <details> <summary><strong>Possible Issues</strong></summary> • A run that calls `trackDeletesStart` but exits before `trackDeletesEnd` will leave the delete-window checkpoint in place for future runs • The per-key `stateByKey` map in `Checkpointing` is not pruned, which could grow in long-lived processes with many dynamic keys • Full reset now returns an error if `hardDeleteCheckpoints` fails, which could block user-initiated resets </details> --- *This summary was automatically generated by @propel-code-bot* | 6 个月前 |