| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Re-work sync engine (#138) Re-implement the workspace sync to operate in chunked batches rather than a single stream. The push/pull methods now become iterators and exec supports a `sync: "defer"` option. This allows implementors to implement a durable sync using a mix of alarms and queues to ensure the sync completes even if the durable object crashes or restarts. | 19 天前 | |
dofs: Add exclusions to recursive grep (#150) Find could prune directories from a traversal but grep could not, so there was no way to search a workspace while skipping node_modules -- the thing callers most often want to skip. A caller could pass include to name what they wanted, but not exclude to name what they did not. grep already delegates its traversal to find's walker via iterateFoundEntries, which has accepted exclusion globs since find gained them. So this passes them through rather than reimplementing matching: an excluded directory is pruned before its children are queried, instead of being walked and read with its matches discarded afterwards. Semantics follow find exactly, because sharing the walker means sharing its rules: globs match the directory-relative path, exclusion is applied before inclusion so it always wins, and a subtree needs both forms (node_modules and node_modules/**) since /** matches what is below a directory rather than the directory itself. A single-file search ignores exclude -- the caller named the file and there is no traversal to prune. The pruning test asserts the query count rather than just the results, because filtering after the walk would produce identical matches while still paying for the excluded tree. Writing it first with only 'vendor/**' showed the walker descending and excluding each child, which is what pinned the both-forms requirement. Also threads exclude through the computer-level grep tool, matching the option find's tool already exposes. * docs: Add PR.md describing the grep exclude change * dofs: Thread grep exclude through the tool contract and docs Review caught two places the new option was missing from contracts a consumer actually reads. tools/fs/store.ts declares WorkspaceLike by inlining the grep options rather than importing GrepOptions, so it kept typechecking while silently not admitting exclude. A caller holding that structural type could not pass the option the tool now accepts. docs/09_tool_interface.md documents exclude under find but not grep, so the two tools read as if only one supports pruning. Adds the field and states the semantics it shares with find, including the both-forms requirement. Also drops PR.md, which was a description for review rather than repository content. Note: the same inlined WorkspaceLike omits exclude from find too, which predates this change and is left alone here. * Update changeset | 9 天前 | |
perf(dofs): Batch sync path resolution and index tombstone scans (#146) A push tick resolved every touched inode to its path(s) with one dirent lookup per ancestor, issuing O(N x depth) statements. On a node_modules-sized change set that is tens of thousands of round-trips, and the cost is the statement count rather than the index: every one of those lookups was already a covering-index hit. pathsOfMany resolves a batch of inodes in a fixed number of recursive-CTE statements, seeding one walk per (target, dirent) so hardlink names all reach the wire. Measured on a node_modules-shaped tree: 6,300 statements to 2 at 900 files, 31,500 to 9 at 4,500 files, with output identical to the per-inode implementation for every inode. The tombstone scan (rev > ? AND op = 'delete' GROUP BY path) had no index it could drive both halves from, so the planner scanned vfs_changes in path order, ignored the rev predicate, and read the whole table to return the few rows in the watermark window. Schema v8 adds vfs_changes_by_op_rev; the upgrade is index-only and rewrites no rows. pathsOf and pathOf are retained: fs/rename, fs/rm, fs/writeFile and sync/apply still resolve a single inode. | 16 天前 | |
perf(dofs): Batch sync path resolution and index tombstone scans (#146) A push tick resolved every touched inode to its path(s) with one dirent lookup per ancestor, issuing O(N x depth) statements. On a node_modules-sized change set that is tens of thousands of round-trips, and the cost is the statement count rather than the index: every one of those lookups was already a covering-index hit. pathsOfMany resolves a batch of inodes in a fixed number of recursive-CTE statements, seeding one walk per (target, dirent) so hardlink names all reach the wire. Measured on a node_modules-shaped tree: 6,300 statements to 2 at 900 files, 31,500 to 9 at 4,500 files, with output identical to the per-inode implementation for every inode. The tombstone scan (rev > ? AND op = 'delete' GROUP BY path) had no index it could drive both halves from, so the planner scanned vfs_changes in path order, ignored the rev predicate, and read the whole table to return the few rows in the watermark window. Schema v8 adds vfs_changes_by_op_rev; the upgrade is index-only and rewrites no rows. pathsOf and pathOf are retained: fs/rename, fs/rm, fs/writeFile and sync/apply still resolve a single inode. | 16 天前 | |
workspace-fs: throw EUNKNOWN_HASH on missing blob pushObjects() threw a plain Error with the message "pushObjects: missing blob for requested hash" when asked for a hash that wasn't in vfs_blob_bytes. The wire contract on SyncRPC.fetchObjects (which this helper backs on the container side) documents EUNKNOWN_HASH as the code callers branch on, and WireErrorCode already reserves it, but no call site raised it. Add EUNKNOWN_HASH to WorkspaceErrorCode and throw via createWorkspaceError so err.code carries it across the wire. WorkspaceFsError's own-enumerable props ride through capnweb, and the round-trip test in rpc/tests/wire.test.ts asserts err.code on the receiving side. | 3 个月前 | |
Fixes for latest issues (#124) * dofs: follow symbolic links in mkdir mkdir walked the target path with a helper that read each dirent directly and treated every node other than a directory as ENOTDIR, so a symbolic link to a directory blocked directory creation rather than resolving through it. Every other write path, writeFile included, follows links and shares one forty-hop budget. The parent walk now expands intermediate links the way writeFile's does, tracking the resolved path so the new directory lands under the directory the link points at and the read-only mount guard sees the location actually written. Recursive creation places its missing ancestors under that resolved parent. A resolved parent that is a file still reports ENOTDIR, a dangling parent link reports ENOENT in both modes rather than being materialised, and a chain beyond forty hops reports ELOOP. Closes #119. * dofs, computer: add exclusion globs to find The walker tested the inclusion glob before yielding an entry but descended into every directory regardless, so a search in a workspace holding node_modules, .git, or generated build output paid for those trees even when the caller wanted nothing from them. FindOptions gains exclude, a list of globs of the same shape as the inclusion pattern and matched against the same directory-relative path. An exclusion is decided before inclusion, so it always wins, and before any child query, so an excluded directory takes its whole subtree with it rather than being filtered out afterwards. Traversal stays deterministic and limit and offset apply to what survives. The option reaches the public find tool, whose schema now advertises it. Closes #121. * dofs, computer: expose rename on the public filesystem The store has implemented transactional file, directory, and symbolic link moves for some time, covering destination replacement, non-empty directories, read-only mounts, tombstones, revision stamping, and subtree tracking. None of that reached Workspace.fs, so a caller had to copy the source and then delete it, and the Worker shell used that fallback for mv. A failure between the two steps left the entry at both paths or a directory half copied. WorkspaceFilesystem now forwards rename, and WorkspaceFilesystemStub mirrors it with the usual filesystem observation span, so the Workers RPC surface matches the in-process one. The shell adapter calls it and keeps copy-then-delete only for a destination rename refuses to replace, which is what the shell expects when it merges a tree. No new method crosses the Cap'n Web boundary: the existing synchronisation protocol already carries the resulting live entries and tombstones. Closes #120. * docs: describe the shipped symbolic-link surface The filesystem specification said symbolic links were an internal primitive, that Workspace.fs exposed neither symlink nor readlink, that there was no lstat, and that an existing file's mode could not be changed. The shipped API contradicts all four: WorkspaceFilesystem exposes symlink, readlink, lstat, and chmod, the stub mirrors them across the Workers RPC boundary, and the Dynamic Worker filesystem adapters rely on them for the node:fs behaviour a shell expects. Removing the methods would be a breaking change and would leave those adapters without a way to serve ln -s, readlink, or test -L, so the document follows the code. Each of the four methods gains a section with its return value and its errors, the comparison with node:fs/promises maps them rather than striking them out, and the note on symbolic links now states the two rules that cover the surface: intermediate segments are always followed, and a trailing link is followed by everything except lstat and readlink. The rename and find entries added alongside are documented in the same pass. Closes #118. --------- Co-authored-by: agent <agent@users.noreply.github.com> | 19 天前 | |
computerd: Add an optional on-disk SQLite store (#123) computerd keeps its workspace in memory, so a restart loses it and the durable object has to send every path again. On a large workspace that replay is most of the time it takes to get back to work. Set COMPUTERD_DB to an absolute path and the store goes on the container's disk instead. The sync positions live in the same database, so a restarted daemon still knows what it was sent and the durable object only sends the difference: about 25ms instead of a full resend, whatever the size of the tree. dofs gains a ./node export with NodeSQLiteStorage, which runs node:sqlite against a file or memory. It stays out of the main entry point because that has to load under workerd, which has no node:sqlite. SQLiteTestStorage is now the in-memory pinning of the same class. Also adds POST /__computerd/checkpoint for folding the write-ahead log back before a disk snapshot, reports the store on /__computerd/info, and adds store size and free pages to /__computerd/stats. Measured with script/store-compare.mjs and script/restore-time.mjs; numbers in packages/computerd/bench-results.md. Reads are not slower on disk. Writes cost 10 to 20 percent more through a real FUSE mount. The in-memory store stays the default. Co-authored-by: Pi Agent <pi-agent@cloudflare.local> | 19 天前 | |
computerd: Add an optional on-disk SQLite store (#123) computerd keeps its workspace in memory, so a restart loses it and the durable object has to send every path again. On a large workspace that replay is most of the time it takes to get back to work. Set COMPUTERD_DB to an absolute path and the store goes on the container's disk instead. The sync positions live in the same database, so a restarted daemon still knows what it was sent and the durable object only sends the difference: about 25ms instead of a full resend, whatever the size of the tree. dofs gains a ./node export with NodeSQLiteStorage, which runs node:sqlite against a file or memory. It stays out of the main entry point because that has to load under workerd, which has no node:sqlite. SQLiteTestStorage is now the in-memory pinning of the same class. Also adds POST /__computerd/checkpoint for folding the write-ahead log back before a disk snapshot, reports the store on /__computerd/info, and adds store size and free pages to /__computerd/stats. Measured with script/store-compare.mjs and script/restore-time.mjs; numbers in packages/computerd/bench-results.md. Reads are not slower on disk. Writes cost 10 to 20 percent more through a real FUSE mount. The in-memory store stays the default. Co-authored-by: Pi Agent <pi-agent@cloudflare.local> | 19 天前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
dofs: expose direct write provider methods Expose createFileSync, writeRangeSync, and truncateFileSync on the SQLite workspace provider so local callers can use the direct incremental write path without going through file descriptors. The provider tests cover direct create, range overwrite, readback, and truncate through the new surface. | 3 个月前 | |
dofs: Resolve pending parent aliases Index pending creates by parent inode and leaf name, falling back to that index when another path alias misses the fast path. | 1 个月前 | |
dofs: Resolve pending parent aliases Index pending creates by parent inode and leaf name, falling back to that index when another path alias misses the fast path. | 1 个月前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
dofs: fold rowid and rev reads into INSERT/UPDATE via RETURNING Every FS mutation pays the rev counter; tiny-file create also reads back last_insert_rowid. SQLite's RETURNING lets us fold both reads into the same statement, cutting one round-trip off each. incrementRev now does a single UPDATE ... RETURNING v instead of an UPDATE followed by a SELECT. The new file/dir/symlink inode inserts do INSERT ... RETURNING inode, so the bare last_insert_rowid lookup goes away. createFileSync also reorders so rev is computed up front and the node row lands with its final stamp in one INSERT, removing the post-insert UPDATE entirely. Cuts createFileSync from 7 SQL statements down to 5 per file (the existing-file overwrite branch already had the optimal shape). | 3 个月前 | |
dofs: resolve paths in one statement and cache path lookups Resolve the read path with a single recursive-CTE statement and add a per-Database path-to-inode cache, so repeat stat/exists/read of a path cost one statement, and one node re-read once warm, independent of depth. Both fast paths serve only follow-symlinks resolutions outside a transaction. A cache hit re-reads just the node row, so mode/size/mtime stay current — only the path-to-inode mapping is cached. A miss runs one recursive walk of vfs_dirents joined to vfs_nodes; it descends only through directories, so a missing segment or non-directory intermediate yields null exactly as the loop, and any symlink on the path falls back to the per-component loop (which follows links and enforces ELOOP). Symlink resolutions are not cached. lstat/readlink and every resolve inside a mutation keep the loop: those are shallow and hot, and the cache must not be populated mid-transaction so a rollback leaves no stale entry. The cache is invalidated at every dirent move — create, unlink, delete, rename, and the sync-apply structural cleanup — dropping one entry for a leaf and the whole subtree for a directory or symlink change. | 2 个月前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 | |
computerd: Add an optional on-disk SQLite store (#123) computerd keeps its workspace in memory, so a restart loses it and the durable object has to send every path again. On a large workspace that replay is most of the time it takes to get back to work. Set COMPUTERD_DB to an absolute path and the store goes on the container's disk instead. The sync positions live in the same database, so a restarted daemon still knows what it was sent and the durable object only sends the difference: about 25ms instead of a full resend, whatever the size of the tree. dofs gains a ./node export with NodeSQLiteStorage, which runs node:sqlite against a file or memory. It stays out of the main entry point because that has to load under workerd, which has no node:sqlite. SQLiteTestStorage is now the in-memory pinning of the same class. Also adds POST /__computerd/checkpoint for folding the write-ahead log back before a disk snapshot, reports the store on /__computerd/info, and adds store size and free pages to /__computerd/stats. Measured with script/store-compare.mjs and script/restore-time.mjs; numbers in packages/computerd/bench-results.md. Reads are not slower on disk. Writes cost 10 to 20 percent more through a real FUSE mount. The in-memory store stays the default. Co-authored-by: Pi Agent <pi-agent@cloudflare.local> | 19 天前 | |
dofs: rename workspace-fs to dofs The package's job is narrow and well-defined: a Durable Object- backed virtual filesystem. The name "workspace-fs" tied it to one consumer (the Workspace package on top); "dofs" names what the package actually is, so it can stand on its own outside the Workspace stack. The rename is mechanical. The directory moves from packages/workspace-fs to packages/dofs, the npm name from @cloudflare/workspace-fs to @cloudflare/dofs, and the tests-only wrangler worker from workspace-fs-tests to dofs-tests. Every import site in workspace, workspace-rpc, wsd, the root README, AGENTS.md, and script/exec-tests follows. Comments that named "workspace-fs" in prose are rewritten to "dofs" so the source no longer mentions the old name. package-lock.json is updated in place; the symlink under node_modules/@cloudflare/dofs resolves to packages/dofs. | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 19 天前 | ||
| 9 天前 | ||
| 16 天前 | ||
| 16 天前 | ||
| 3 个月前 | ||
| 19 天前 | ||
| 19 天前 | ||
| 19 天前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 19 天前 | ||
| 3 个月前 |