| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: support optional pin names (#10261) | 2 年前 | |
fix(fuse): keep what you write after a rename (#11430) * fix(fuse): keep a rename's writes go-fuse hands the kernel's existing node to the new name once Dir.Rename returns, and that node still held the MFS handle the rename had unlinked. MFS treats such a handle as gone: a write through it was accepted and then dropped, so `mv a b` followed by a write to `b` read back the old contents a second later, once the entry cache expired. A directory was worse. Creating a file in one that had just been renamed flushed through the dead handle, which carried the name the rename had moved away from, so the new file was lost and the old directory came back for good. Each node now reaches its MFS handle through an atomic, and a rename points the moved node at the entry that exists afterwards. Entries the kernel had already looked up underneath a renamed directory hang off the handle it was reached through, so the walk follows them down; it covers what the kernel is holding, not the whole tree. Invalidating the entry instead was tried and does not work: the kernel processes FUSE_NOTIFY_INVAL_ENTRY while holding the parent inode lock, so notifying from inside Rename deadlocks, and notifying asynchronously still loses most of the writes it races. Left unfixed: a write through a file descriptor held open across the rename still goes to the descriptor opened from the old handle. * fix(fuse): refuse to replace a non-empty directory A rename may only overwrite a directory that is empty. MFS removes a directory and everything under it without complaint, so `mv -T src dst` took dst's contents with it and reported success. Rmdir already had the check; Rename now makes it too, before it unlinks anything. The check for an absent destination also goes through errors.Is now. It compares against a sentinel that boxo returns bare today, and the cost of that changing is the source file, which by then has been unlinked. * fix(fuse): read /ipfs blocks we cannot decode A UnixFS directory can link to a block of any codec. stat reports one the mount cannot decode as a file the size of the block, but every read of it failed, because the read path went looking for a UnixFS DAG that is not there. A size stat promises has to be a size reads deliver, so serve the block itself. * fix(fuse): list a directory with a missing block One child whose block is not held locally failed the whole listing, and with an errno the caller could make nothing of: ipld.ErrNotFound has no mapping, so it arrived as ENOSYS. The readable entries are worth having, so report the one that is missing with no type and let a stat of it say what is wrong. * fix(fuse): report the CID the path used The ipfs.cid xattr answered with a CID the caller had never seen. A lookup rebuilds the node by decoding the block, which drops the version and codec of the path it came from, so a v1 dag-pb path reported its v0 form. Keep the CID the entry resolved to and report that. The changelog entry also covers the rename check from the commit before it, which landed without one. * test(fuse): make the rename tests catch their bugs TestRenameOntoNamespaceRoot read the file back through the mount, which answers from the entry the kernel still has cached and so succeeds whether or not the rename took the file away. It passed against the bug it was written for. Ask MFS instead. The dirent helper also loops on a record length it never checks, which would spin rather than fail if the kernel ever sent zero. * test(ipns): settle the repo path before mounting TestStatfs assigned Root.RepoPath once the server was already serving, and Statfs reads it from a FUSE handler goroutine, so `go test -race ./fuse/...` reported a data race on every run. The mfs and readonly tests already settle it before their mount; do the same here. | 1 个月前 | |
fix(fuse): give mounts stable inode numbers (#11429) * fix(fuse): report a link count of 1 st_nlink was left at 0, which POSIX gives an inode with no remaining names, so tools can read a live file as one on its way out. Neither IPFS nor MFS has hard links. Directories report 1 as well, which keeps GNU find from trusting a subdirectory count and skipping entries. * fix(ipns): fill attrs in key directory lookups The /ipns root answers lookups for its key directories and alias symlinks itself, and the reply carried zeroed attributes. Every later lookup refreshed the kernel's cache with the same zeroes, so the Getattr that would have corrected them never ran and a key directory showed up as d--------- with no link count. * fix(fuse): give mounts stable inode numbers go-fuse numbers any node left with a zero StableAttr.Ino itself, and picks a new number every time. The kernel drops a mount's dentries once EntryTimeout expires, so a file nobody touched came back from the next lookup under a different st_ino, and programs that compare file identity over time read that as the file being replaced. vim abandons a save with "E949: File changed while writing", which is what made the FUSE CI job flaky once it moved to slower runners and the save started crossing the one second cache boundary. - /ipfs takes the number from the multihash digest inside the CID, so the same content is one object whichever path reaches it; inline CIDs hash the whole CID instead, their digest being the content itself - /ipns and /mfs allocate per mount from a counter keyed by parent and name, retired on unlink, rmdir and rename so a name that is created again is never handed a removed entry's number - writable nodes carry a generation of their own, so go-fuse builds a fresh node per lookup instead of reusing one bound to an *mfs.File that boxo has since replaced - mount points report inode 1 instead of 0, and readdir reports the same numbers as stat * fix(fuse): tell two CIDs apart on /ipfs go-fuse matches a lookup against the nodes it already holds by the whole of StableAttr, so two entries that agree on it are served as one object. The inode number alone is 63 bits, and two CIDs can end up sharing one, by chance or by choice. Reading the second one then returned the first one's bytes. A dag-pb CID and the raw CID of the same block hit this every time, because the number ignored the codec. The number and the generation now both come from a hash of the codec and the multihash, so it takes a match on 128 bits to confuse two entries. The mount also sets FirstAutomaticIno instead of relying on go-fuse's default, so the range it keeps for itself stays where the code says it is. * fix(fuse): stat /ipfs entries we cannot read A stat of a child whose block is missing, or of one in a codec this mount does not decode, read UnixFS metadata that was never loaded and panicked. go-fuse does not recover a panic in its serve loop, so this took the whole daemon down. Neither case is exotic: a dag-cbor object linked from a UnixFS directory needs no missing blocks at all. A lookup that cannot read the block now fails instead of building an entry from it, and a block that is not UnixFS is reported as a file of its own size. * fix(fuse): keep a file a rename cannot move `mv /ipns/<key>/f /ipns/f` unlinked the source before finding out that the /ipns root holds no files of its own, then failed with EINVAL and left the file nowhere. The destination is now checked before anything is written. * fix(fuse): store a moved file where it landed A rename across directories only wrote the source directory back, so the file's new name lived in memory until something else flushed it, and a daemon that stopped first lost the file. The destination is written back first, so an interrupted rename leaves the file under both names rather than under neither. * fix(fuse): keep the inode number over a rename Both names gave up their inode numbers on rename, so a moved file came back about a second later as a different file, which is the problem this branch fixes everywhere else. A moved directory was worse: its entries are keyed by its number, so the whole subtree was renumbered and the old keys were left behind until unmount. The number now moves with the entry. Nothing else has to change to make that safe: go-fuse cannot hand back the old node anyway, because every node gets a generation of its own. The comments that credited the renumbering for it were wrong. * test(fuse): check listings and stat agree Each mount fills in the inode number of a directory entry separately from the one it reports to stat, and nothing compared the two. Tools read whichever is cheaper for them. * docs(config): warn about writing to a mounted mfs `ipfs files` writes to the same tree as the /mfs and /ipns mounts without the mount knowing, so the two can lose each other's writes. | 1 个月前 | |
fix(fuse): give mounts stable inode numbers (#11429) * fix(fuse): report a link count of 1 st_nlink was left at 0, which POSIX gives an inode with no remaining names, so tools can read a live file as one on its way out. Neither IPFS nor MFS has hard links. Directories report 1 as well, which keeps GNU find from trusting a subdirectory count and skipping entries. * fix(ipns): fill attrs in key directory lookups The /ipns root answers lookups for its key directories and alias symlinks itself, and the reply carried zeroed attributes. Every later lookup refreshed the kernel's cache with the same zeroes, so the Getattr that would have corrected them never ran and a key directory showed up as d--------- with no link count. * fix(fuse): give mounts stable inode numbers go-fuse numbers any node left with a zero StableAttr.Ino itself, and picks a new number every time. The kernel drops a mount's dentries once EntryTimeout expires, so a file nobody touched came back from the next lookup under a different st_ino, and programs that compare file identity over time read that as the file being replaced. vim abandons a save with "E949: File changed while writing", which is what made the FUSE CI job flaky once it moved to slower runners and the save started crossing the one second cache boundary. - /ipfs takes the number from the multihash digest inside the CID, so the same content is one object whichever path reaches it; inline CIDs hash the whole CID instead, their digest being the content itself - /ipns and /mfs allocate per mount from a counter keyed by parent and name, retired on unlink, rmdir and rename so a name that is created again is never handed a removed entry's number - writable nodes carry a generation of their own, so go-fuse builds a fresh node per lookup instead of reusing one bound to an *mfs.File that boxo has since replaced - mount points report inode 1 instead of 0, and readdir reports the same numbers as stat * fix(fuse): tell two CIDs apart on /ipfs go-fuse matches a lookup against the nodes it already holds by the whole of StableAttr, so two entries that agree on it are served as one object. The inode number alone is 63 bits, and two CIDs can end up sharing one, by chance or by choice. Reading the second one then returned the first one's bytes. A dag-pb CID and the raw CID of the same block hit this every time, because the number ignored the codec. The number and the generation now both come from a hash of the codec and the multihash, so it takes a match on 128 bits to confuse two entries. The mount also sets FirstAutomaticIno instead of relying on go-fuse's default, so the range it keeps for itself stays where the code says it is. * fix(fuse): stat /ipfs entries we cannot read A stat of a child whose block is missing, or of one in a codec this mount does not decode, read UnixFS metadata that was never loaded and panicked. go-fuse does not recover a panic in its serve loop, so this took the whole daemon down. Neither case is exotic: a dag-cbor object linked from a UnixFS directory needs no missing blocks at all. A lookup that cannot read the block now fails instead of building an entry from it, and a block that is not UnixFS is reported as a file of its own size. * fix(fuse): keep a file a rename cannot move `mv /ipns/<key>/f /ipns/f` unlinked the source before finding out that the /ipns root holds no files of its own, then failed with EINVAL and left the file nowhere. The destination is now checked before anything is written. * fix(fuse): store a moved file where it landed A rename across directories only wrote the source directory back, so the file's new name lived in memory until something else flushed it, and a daemon that stopped first lost the file. The destination is written back first, so an interrupted rename leaves the file under both names rather than under neither. * fix(fuse): keep the inode number over a rename Both names gave up their inode numbers on rename, so a moved file came back about a second later as a different file, which is the problem this branch fixes everywhere else. A moved directory was worse: its entries are keyed by its number, so the whole subtree was renumbered and the old keys were left behind until unmount. The number now moves with the entry. Nothing else has to change to make that safe: go-fuse cannot hand back the old node anyway, because every node gets a generation of its own. The comments that credited the renumbering for it were wrong. * test(fuse): check listings and stat agree Each mount fills in the inode number of a directory entry separately from the one it reports to stat, and nothing compared the two. Tools read whichever is cheaper for them. * docs(config): warn about writing to a mounted mfs `ipfs files` writes to the same tree as the /mfs and /ipns mounts without the mount knowing, so the two can lose each other's writes. | 1 个月前 | |
fix(mfs): stop repo gc from freezing files ops (#11386) * fix(mfs): stop repo gc from freezing files ops Running `ipfs repo gc` alongside `ipfs files` writes could leave MFS permanently hung: GC deleted directory-node blocks a write had written to the blockstore but not yet linked into the persisted MFS root, and the next path lookup blocked forever fetching the missing block while holding the MFS directory lock, so every later files command piled up behind it. MFS mutations now take the pin lock, the same lock `ipfs add` uses, and GC computes the MFS root only after it holds the GC lock. A write's blocks are therefore either fully linked into the root before GC runs, or the write waits for GC to finish, so GC never collects data a live write still needs. - core/commands/files.go: hold the pin lock across write, cp, mkdir, mv, rm, flush, chcid, chmod, and touch - gc/gc.go: take the best-effort MFS root snapshot after acquiring the GC lock, closing the snapshot-before-lock window - core/corerepo/gc.go: pass the root as a callback evaluated under the lock - core/node/core.go: document why MFS keeps its online DAG service so lazy `ipfs files cp /ipfs/<cid>` pointers still resolve - gc/gc_test.go: assert the root snapshot is taken under the GC lock Closes #10842 * fix(mfs): extend gc pin lock to add and fuse The pin lock that stops garbage collection from collecting live MFS blocks now covers every path that mutates MFS, not just `ipfs files`, and every live MFS root is part of the GC live set. - core/commands/add.go: hold the pin lock while `ipfs add --to-files` links the added content into the MFS root - fuse/writable, fuse/mfs: hold the pin lock across FUSE `/mfs` structural writes and file flush, fsync, and release - fuse/ipns: same for the per-key `/ipns` mounts, and register their roots so GC keeps their blocks live - core/core.go: track mounted MFS roots for the GC live set - core/corerepo/gc.go: build the live set from the files root plus every registered root, skipping a root that errors instead of aborting the whole GC Closes #6113 Closes #7008 Closes #9553 * fix(mfs): fail fast on a missing block A directory-node block that is missing locally and unreachable would block an MFS operation forever while it held the directory lock, wedging the whole MFS and a clean shutdown until the process was killed. This is what remained after a repo was damaged by an older Kubo, a manual `ipfs block rm`, or a crash (the lockup in #7844). MFS now bounds those network reads: an unreachable block fails the operation with a timeout and releases the lock, while lazily-referenced content (`ipfs files cp /ipfs/<cid>`) still loads as before. - config: add DefaultMFSFetchTimeout and pass it to the MFS root via mfs.WithFetchTimeout in Import.MFSRootOptions - go.mod: bump boxo to pull in mfs.WithFetchTimeout * feat(mfs): honor --timeout in files read and write Adopt boxo's mfs.File.Open(ctx) so MFS file operations carry a context and a read or write stuck on a missing block can be cancelled instead of blocking forever. - go.mod: bump boxo to pick up mfs.File.Open(ctx) - core/commands/files.go: pass the request context to files read and write, so a client --timeout ends a stuck read or write - fuse/writable: bind long-lived FUSE descriptors (Create, Open) to the mount context via the new Config.MountCtx; the transient Setattr truncate uses the per-operation context - fuse/mfs, fuse/ipns: set MountCtx to the node/mount context - test/cli: cover files read and write honoring --timeout on unreachable content * chore(deps): bump boxo to merged mfs fix Switch from the pre-merge branch pseudo-version to the merged commit of ipfs/boxo#1185, the boxo side of the MFS under-lock work this branch depends on. * refactor(fuse): pass request ctx to pin lock Flush, Release, and Fsync passed context.Background() to the pin lock while their own ctx argument was in scope. Thread the passed ctx through instead, matching the other handlers. boxo's default GCLocker discards the context, so this is a consistency change, not a behavior change. https://github.com/ipfs/kubo/pull/11386#discussion_r3534131824 https://github.com/ipfs/kubo/pull/11386#discussion_r3534138912 https://github.com/ipfs/kubo/pull/11386#discussion_r3534140861 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(gc): const for buffer, simplify roots Name the GC result channel buffer size as a const, drop the temporary in the best-effort root snapshot, and use t.Context() in the snapshot test. The buffer size predates this branch; it only shifted in the diff. https://github.com/ipfs/kubo/pull/11386#discussion_r3534154606 https://github.com/ipfs/kubo/pull/11386#discussion_r3534077507 https://github.com/ipfs/kubo/pull/11386#discussion_r3534085418 https://github.com/ipfs/kubo/pull/11386#discussion_r3534106766 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(add): collapse pin lock defer Fold the pin lock's unlock into the defer at both --to-files call sites, matching the one-line style used elsewhere. https://github.com/ipfs/kubo/pull/11386#discussion_r3534006489 https://github.com/ipfs/kubo/pull/11386#discussion_r3534013057 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 2 年前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 3 个月前 |