SQLite Storage Components
Depot is split into the conveyer hot path plus workflow compaction. Workflow compaction is the only compaction publish/delete authority.
Conveyer
The conveyer is the request path used by the SQLite VFS.
Responsibilities:
- Resolve bucket/database branch ancestry for reads.
- Commit dirty pages as LTX DELTA chunks under
BR/{database_id}/DELTA/{txid}/{chunk}. - Write PIDX owner rows for dirty pages.
- Write
COMMITS/{txid}andVTX/{versionstamp}in the same commit transaction. - Maintain
META/head, quota counters, and access-touch manifest fields. - Update
SQLITE_CMP_DIRTY/{database_branch_id}and send throttledDeltasAvailableworkflow wakeups when hot lag crosses compaction thresholds. - Create buckets, create databases, fork buckets, fork databases, and write branch records/catalog markers.
- Create and resolve restore points. Pinned restore points write FDB pins directly and start as
PinStatus::Ready.
Lease ownership: none. Correctness relies on Pegboard single-writer exclusivity for a live database plus FDB transaction fences. The conveyer must not take compactor leases.
Workflow Compaction
The workflow compaction path uses one persistent DB manager plus hot and reclaim companion workflows per database branch.
Responsibilities:
- Coalesce commit wakeups through
SQLITE_CMP_DIRTY/{database_branch_id}andDeltasAvailablesignals. - Plan hot jobs from current FDB state instead of trusting signal payloads.
- Cap each hot job's drain to
MAX_HOT_DRAIN_SPAN_TXIDSpast the hot watermark (raised to the first fold when it sits past the window). A large backlog, such as one accumulated while compaction was disabled, catches up incrementally across manager refresh cycles instead of one unbounded drain. - Carry the branch lifecycle generation through planned jobs and reject stale stage, publish, or reclaim work after branch deletion or recreation.
- Have the hot companion write staged shard blobs under
CMP/stage/{job_id}/hot_shard. - Install matching hot job output by copying staged blobs to reader-visible
SHARD, advancingCMP/root, and compare-and-clearing expected PIDX rows. The install activity returns a resume cursor onceCMP_BULK_ACTIVITY_EARLY_TIMEOUTelapses and the manager re-dispatches from it, so slow FDB transactions shrink each activity call instead of racing the hard activity timeout. - Have the reclaimer delete only manager-authorized FDB rows and staged output.
- Reclaim a folded
DELTA/{txid}/*(no live PIDX entry owns its pages) only when the shard-materialization gate passes: for the smallest coverage fold at or above the txid, every shard the delta touched has a materialized version in[txid, fold](read from the C1 fold index). The freed DELTA bytes are credited back toMETA/quota. - Reclaim a non-fold commit's
COMMITS/{txid}+VTX/{vs}only at or below the delete bound, which ismin(pin txids, unexpired PITR rep txids) - 1(falling back to the hot watermark when there is no pin). COMMITS/VTX are not billable keys, so their deletion takes no quota credit. - Budget every reclaim slice's delete candidates (expired PITR interval rows, shard-cache demotes, dead shard versions, commits/deltas) by key count and value bytes against one shared
CompactionBatchBudget, because each lane'sCOMPARE_AND_CLEARcarries the compared value into the single slice transaction. The plan and delete paths consume the budget in the same deterministic order so both derive identical sets, and capped tails drain across later slices. - Keep automatic PITR interval coverage and retained restore point pins live until reclaim can prove they are no longer needed.
- Stop the manager and companion workflows through
DestroyDatabaseBranchwhen a database branch is no longer live.
Lease ownership: none. Gasoline workflow uniqueness uses only the database branch id tag.
Ownership Summary
| Component | Main writes | Lease |
|---|---|---|
| Conveyer | META/head, COMMITS, VTX, PIDX, DELTA, branch records, restore points |
None |
| Workflow DB manager | CMP/root, live SHARD, PITR_INTERVAL, matching PIDX clears |
None |
| Workflow companions | Staged hot output, manager-authorized deletes and cleanup | None |
The components share branch metadata and pin counters, but each mutable manifest field has one owner.