| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
refactor: migrate away from cheggaaa/pb v1 (#11322) * refactor: migrate away from cheggaaa/pb v1 * updated changelogs * fix: add space after comment slashes for consistency * refactor: share terminal detection in cmdenv Replace three duplicate TTY checks (get.go, dag/export.go, dag/stat.go) with `cmdenv.IsTerminal(*os.File)` backed by `mattn/go-isatty`. The helper uses `IsTerminal || IsCygwinTerminal`, which also detects MSYS2 and Git Bash on Windows. Those terminals expose stdio as a named pipe rather than a character device, so the previous `ModeCharDevice` check suppressed the progress bar on real terminals. - core/commands/cmdenv/tty.go: new helper - core/commands/{add,cat,get}.go: drop local isStderrTTY - core/commands/dag/{export,stat}.go: drop inline stat() block - go.mod: promote mattn/go-isatty to direct (was indirect via pb/v3) * refactor: cmdenv.ShouldShowProgress helper Collapse the explicit-flag-or-TTY-default logic at four call sites (`cat`, `get`, `dag export`, `dag stat`) into a single helper. * refactor: dedupe `ipfs add` progress template The full bar template (counters, bar, speed, percent, ETA) was inlined at two call sites in add.go. Move it to a file-level const. * fix: progress bar shows MiB/s, not MiB p/s pb v3's speed element defaults to suffix "%s p/s", so even with pb.Bytes set, `ipfs add`, `ipfs cat`, `ipfs get`, and `ipfs dag export` rendered the rate as "713.04 MiB p/s" instead of "713.04 MiB/s". Pass explicit format args to the speed and rtime template elements: rate now renders as "MiB/s", and the unknown-state fallback reads "?/s" / "ETA ?" instead of bare "?". The four templates move to package-level consts. * docs: rewrite v0.42 progress bar entry Describe only the user-visible changes; skip library-migration detail and intermediate-state claims that never shipped. * chore: drop unused pb v1 dependabot ignore The `github.com/cheggaaa/pb` (v1) module path is no longer in `go.mod` after the migration to `pb/v3`, so the ignore rule never fires. * fix(dag): unify --progress help text Match the wording used by `add`, `cat`, and `get`: "Stream progress data. Defaults to true when stderr is a terminal." * fix(add): finalize progress bar after upload Call `bar.Finish()` and a final `bar.Write()` after the progress loop. Without it, fast adds (under ~500ms, where pb/v3's EWMA never accumulates a speed sample) render `?/s ... ETA ?` in the last frame. Finishing the bar switches the speed element to its absolute-rate branch (total/elapsed), so the final frame now reads e.g. `792.04 MiB/s 100.00% 100ms`. * test(cmdenv): cover ShouldShowProgress Exercise the explicit-true, explicit-false, unset, and non-bool paths. Unset and non-bool fall back to IsTerminal(os.Stderr), which the test compares against directly so it works in both TTY and CI environments. * refactor: share full progress bar template Move the "total known" pb/v3 template to cmdenv.ProgressBarFullTemplate so add.go and get.go reference the same string instead of keeping byte-identical local copies. The add init template and dag/export streaming template stay local because each is single-use and shaped differently. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 3 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
fix: harden CAR streaming and truncation (#11409) * fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch. | 1 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
chore: bump go-libp2p v0.22.0 & go1.18&go1.19 Fixes: #9225 | 3 年前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
fix(cli/rpc): --cid-base works in all commands (#11239) * fix: --cid-base works in all commands and auto-upgrades CIDv0 Passing --cid-base=base32 now returns CIDv1 in base32 everywhere, including block, dag stat, and object patch which previously ignored it. - cidbase: auto-upgrade CIDv0 when base is not base58btc, deprecate --upgrade-cidv0-in-output, remove GetLowLevelCidEncoder - block stat/put/rm: use GetCidEncoder - dag stat: store CID as pre-encoded string, drop MarshalJSON/UnmarshalJSON - object patch rm-link/add-link: use GetCidEncoder - bitswap: switch to GetCidEncoder * test: add harness tests for --cid-base flag Remove unused DagStat.String() which truncated CIDs. Add CLI tests for --cid-base across block, dag stat, and object patch commands, including the --format=v0 interaction. * fix: respect --cid-base in refs local, object diff, pin remote, files chroot Use GetCidEncoder in commands that were still outputting CIDs via raw .String() calls. - refs local: encode blockstore keys with the requested base - object diff: encode Before/After CIDs in text encoder - pin remote add/ls: pass encoder through toRemotePinOutput - files chroot: encode old/new root CIDs in status message - tests: use base16 to avoid false positives if base32 becomes default * docs: update changelog entry for --cid-base fixes * test: cover --cid-base for add, pin ls, dag import Add harness tests for add, add -Q, pin ls, and dag import. Fix object patch tests broken by upstream UnixFS validation. Use base16 in all tests to avoid false positives. * docs: add metrics and CARv2 highlights to v0.41 changelog | 4 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
refactor: use slices.Sort where appropriate (#10858) | 1 年前 | |
fix(mfs): stop repo gc from freezing files ops (#11386) * fix(mfs): stop repo gc from freezing files ops Running `ipfs repo gc` alongside `ipfs files` writes could leave MFS permanently hung: GC deleted directory-node blocks a write had written to the blockstore but not yet linked into the persisted MFS root, and the next path lookup blocked forever fetching the missing block while holding the MFS directory lock, so every later files command piled up behind it. MFS mutations now take the pin lock, the same lock `ipfs add` uses, and GC computes the MFS root only after it holds the GC lock. A write's blocks are therefore either fully linked into the root before GC runs, or the write waits for GC to finish, so GC never collects data a live write still needs. - core/commands/files.go: hold the pin lock across write, cp, mkdir, mv, rm, flush, chcid, chmod, and touch - gc/gc.go: take the best-effort MFS root snapshot after acquiring the GC lock, closing the snapshot-before-lock window - core/corerepo/gc.go: pass the root as a callback evaluated under the lock - core/node/core.go: document why MFS keeps its online DAG service so lazy `ipfs files cp /ipfs/<cid>` pointers still resolve - gc/gc_test.go: assert the root snapshot is taken under the GC lock Closes #10842 * fix(mfs): extend gc pin lock to add and fuse The pin lock that stops garbage collection from collecting live MFS blocks now covers every path that mutates MFS, not just `ipfs files`, and every live MFS root is part of the GC live set. - core/commands/add.go: hold the pin lock while `ipfs add --to-files` links the added content into the MFS root - fuse/writable, fuse/mfs: hold the pin lock across FUSE `/mfs` structural writes and file flush, fsync, and release - fuse/ipns: same for the per-key `/ipns` mounts, and register their roots so GC keeps their blocks live - core/core.go: track mounted MFS roots for the GC live set - core/corerepo/gc.go: build the live set from the files root plus every registered root, skipping a root that errors instead of aborting the whole GC Closes #6113 Closes #7008 Closes #9553 * fix(mfs): fail fast on a missing block A directory-node block that is missing locally and unreachable would block an MFS operation forever while it held the directory lock, wedging the whole MFS and a clean shutdown until the process was killed. This is what remained after a repo was damaged by an older Kubo, a manual `ipfs block rm`, or a crash (the lockup in #7844). MFS now bounds those network reads: an unreachable block fails the operation with a timeout and releases the lock, while lazily-referenced content (`ipfs files cp /ipfs/<cid>`) still loads as before. - config: add DefaultMFSFetchTimeout and pass it to the MFS root via mfs.WithFetchTimeout in Import.MFSRootOptions - go.mod: bump boxo to pull in mfs.WithFetchTimeout * feat(mfs): honor --timeout in files read and write Adopt boxo's mfs.File.Open(ctx) so MFS file operations carry a context and a read or write stuck on a missing block can be cancelled instead of blocking forever. - go.mod: bump boxo to pick up mfs.File.Open(ctx) - core/commands/files.go: pass the request context to files read and write, so a client --timeout ends a stuck read or write - fuse/writable: bind long-lived FUSE descriptors (Create, Open) to the mount context via the new Config.MountCtx; the transient Setattr truncate uses the per-operation context - fuse/mfs, fuse/ipns: set MountCtx to the node/mount context - test/cli: cover files read and write honoring --timeout on unreachable content * chore(deps): bump boxo to merged mfs fix Switch from the pre-merge branch pseudo-version to the merged commit of ipfs/boxo#1185, the boxo side of the MFS under-lock work this branch depends on. * refactor(fuse): pass request ctx to pin lock Flush, Release, and Fsync passed context.Background() to the pin lock while their own ctx argument was in scope. Thread the passed ctx through instead, matching the other handlers. boxo's default GCLocker discards the context, so this is a consistency change, not a behavior change. https://github.com/ipfs/kubo/pull/11386#discussion_r3534131824 https://github.com/ipfs/kubo/pull/11386#discussion_r3534138912 https://github.com/ipfs/kubo/pull/11386#discussion_r3534140861 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(gc): const for buffer, simplify roots Name the GC result channel buffer size as a const, drop the temporary in the best-effort root snapshot, and use t.Context() in the snapshot test. The buffer size predates this branch; it only shifted in the diff. https://github.com/ipfs/kubo/pull/11386#discussion_r3534154606 https://github.com/ipfs/kubo/pull/11386#discussion_r3534077507 https://github.com/ipfs/kubo/pull/11386#discussion_r3534085418 https://github.com/ipfs/kubo/pull/11386#discussion_r3534106766 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(add): collapse pin lock defer Fold the pin lock's unlock into the defer at both --to-files call sites, matching the one-line style used elsewhere. https://github.com/ipfs/kubo/pull/11386#discussion_r3534006489 https://github.com/ipfs/kubo/pull/11386#discussion_r3534013057 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
feat(cli): add --human and --sort-size to ipfs ls (#11408) * feat(cli): add --human and --sort-size to ipfs ls - --human (-H): SI human-readable sizes in text output (humanize.Bytes) - --sort-size (-S): sort directory entries by size, largest first - Validation: --sort-size + --stream and --sort-size + --size=false errors - Unit tests for formatSize and sort helpers - CLI integration tests for both flags - Changelog highlight in v0.44 * test(ls): make sort tests fail when sorting breaks The tests covering --sort-size could not detect a broken feature. The unit tests copied the comparator into the test body and sorted with their own copy, so they passed regardless of what ls.go did. The CLI tests named each fixture after its size, which left alphabetical order and size order in agreement, so every ordering subtest passed even with --sort-size disabled outright. - extract lsLinkByName and lsLinkBySize so the tests exercise shipped code - point the unit tests at those two functions - name fixtures so their alphabetical order disagrees with their sizes - pin the real directory behaviour: UnixFS directories carry no Filesize, so they sort as 0, tie with empty files and break by name rather than landing strictly last * fix(ls): return 400 not 500 for bad flag combos Passing --sort-size together with --stream or --size=false made /api/v0/ls answer 500 Internal Server Error, telling API clients the server had broken when the caller had simply combined flags that cannot work together. Clients that retry on 5xx would retry a request that can never succeed. cmds.ErrClient maps to 400, matching how the rest of core/commands reports caller mistakes. CLI output is unchanged. * docs: fix --human size examples and JSON claims The help for --human advertised "1K 234M 2G", which no kubo command has ever printed. All three use humanize.Bytes, so the real output is SI with a space: 1.2 kB, 234 MB, 2.0 GB. The stale example was copied into 'ipfs ls' from the two commands that already carried it, so correct all three together. - ls, repo stat, bitswap stat: examples now match real output - drop the claim that --enc=json reports bytes. On the CLI, 'ipfs ls' has a PostRun that prints the text table whatever --enc says, so no JSON is produced there at all. Only the /api/v0/ls response is JSON, and that part is true - directories have no UnixFS Filesize, so they sort as 0 and tie with empty files. They do not land strictly last, so stop saying they do - changelog: a #### highlight with a TOC entry, kept short and pointing at 'ipfs ls --help' for the details - format sizes with strconv.FormatUint rather than fmt.Sprintf("%d") --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 个月前 | |
fix(cli/rpc): --cid-base works in all commands (#11239) * fix: --cid-base works in all commands and auto-upgrades CIDv0 Passing --cid-base=base32 now returns CIDv1 in base32 everywhere, including block, dag stat, and object patch which previously ignored it. - cidbase: auto-upgrade CIDv0 when base is not base58btc, deprecate --upgrade-cidv0-in-output, remove GetLowLevelCidEncoder - block stat/put/rm: use GetCidEncoder - dag stat: store CID as pre-encoded string, drop MarshalJSON/UnmarshalJSON - object patch rm-link/add-link: use GetCidEncoder - bitswap: switch to GetCidEncoder * test: add harness tests for --cid-base flag Remove unused DagStat.String() which truncated CIDs. Add CLI tests for --cid-base across block, dag stat, and object patch commands, including the --format=v0 interaction. * fix: respect --cid-base in refs local, object diff, pin remote, files chroot Use GetCidEncoder in commands that were still outputting CIDs via raw .String() calls. - refs local: encode blockstore keys with the requested base - object diff: encode Before/After CIDs in text encoder - pin remote add/ls: pass encoder through toRemotePinOutput - files chroot: encode old/new root CIDs in status message - tests: use base16 to avoid false positives if base32 becomes default * docs: update changelog entry for --cid-base fixes * test: cover --cid-base for add, pin ls, dag import Add harness tests for add, add -Q, pin ls, and dag import. Fix object patch tests broken by upstream UnixFS validation. Use base16 in all tests to avoid false positives. * docs: add metrics and CARv2 highlights to v0.41 changelog | 4 个月前 | |
feat(config): AutoConf with "auto" placeholders (#10883) https://github.com/ipfs/kubo/pull/10883 https://github.com/ipshipyard/config.ipfs-mainnet.org/issues/3 --------- Co-authored-by: gammazero <gammazero@users.noreply.github.com> | 1 年前 | |
refactor: migrate away from cheggaaa/pb v1 (#11322) * refactor: migrate away from cheggaaa/pb v1 * updated changelogs * fix: add space after comment slashes for consistency * refactor: share terminal detection in cmdenv Replace three duplicate TTY checks (get.go, dag/export.go, dag/stat.go) with `cmdenv.IsTerminal(*os.File)` backed by `mattn/go-isatty`. The helper uses `IsTerminal || IsCygwinTerminal`, which also detects MSYS2 and Git Bash on Windows. Those terminals expose stdio as a named pipe rather than a character device, so the previous `ModeCharDevice` check suppressed the progress bar on real terminals. - core/commands/cmdenv/tty.go: new helper - core/commands/{add,cat,get}.go: drop local isStderrTTY - core/commands/dag/{export,stat}.go: drop inline stat() block - go.mod: promote mattn/go-isatty to direct (was indirect via pb/v3) * refactor: cmdenv.ShouldShowProgress helper Collapse the explicit-flag-or-TTY-default logic at four call sites (`cat`, `get`, `dag export`, `dag stat`) into a single helper. * refactor: dedupe `ipfs add` progress template The full bar template (counters, bar, speed, percent, ETA) was inlined at two call sites in add.go. Move it to a file-level const. * fix: progress bar shows MiB/s, not MiB p/s pb v3's speed element defaults to suffix "%s p/s", so even with pb.Bytes set, `ipfs add`, `ipfs cat`, `ipfs get`, and `ipfs dag export` rendered the rate as "713.04 MiB p/s" instead of "713.04 MiB/s". Pass explicit format args to the speed and rtime template elements: rate now renders as "MiB/s", and the unknown-state fallback reads "?/s" / "ETA ?" instead of bare "?". The four templates move to package-level consts. * docs: rewrite v0.42 progress bar entry Describe only the user-visible changes; skip library-migration detail and intermediate-state claims that never shipped. * chore: drop unused pb v1 dependabot ignore The `github.com/cheggaaa/pb` (v1) module path is no longer in `go.mod` after the migration to `pb/v3`, so the ignore rule never fires. * fix(dag): unify --progress help text Match the wording used by `add`, `cat`, and `get`: "Stream progress data. Defaults to true when stderr is a terminal." * fix(add): finalize progress bar after upload Call `bar.Finish()` and a final `bar.Write()` after the progress loop. Without it, fast adds (under ~500ms, where pb/v3's EWMA never accumulates a speed sample) render `?/s ... ETA ?` in the last frame. Finishing the bar switches the speed element to its absolute-rate branch (total/elapsed), so the final frame now reads e.g. `792.04 MiB/s 100.00% 100ms`. * test(cmdenv): cover ShouldShowProgress Exercise the explicit-true, explicit-false, unset, and non-bool paths. Unset and non-bool fall back to IsTerminal(os.Stderr), which the test compares against directly so it works in both TTY and CI environments. * refactor: share full progress bar template Move the "total known" pb/v3 template to cmdenv.ProgressBarFullTemplate so add.go and get.go reference the same string instead of keeping byte-identical local copies. The add init template and dag/export streaming template stay local because each is single-use and shaped differently. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 3 个月前 | |
feat(cmd): add 'ipfs cid inspect' command (#11241) * feat(cmd): add 'ipfs cid inspect' command Adds a new subcommand to inspect and display detailed CID information including version, multibase encoding, multicodec, and multihash components. Also shows equivalent CIDv0/CIDv1 representations. Example output: CID: bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi Version: 1 Multibase: base32 (b) Multicodec: dag-pb (0x70) Multihash: sha2-256 (0x12) Length: 32 bytes Digest: c3c4733ec8affd06cf9e9ff50ffc6bcd2ec85a6170004bb709669c31de94391a CIDv0: QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR CIDv1: bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi Supports --enc=json for machine-readable output. Fixes #11205 * refactor: tidy up CidInspectRes struct and add '/cid/inspect' route to tests This commit refines the CidInspectRes struct for better readability and consistency. Additionally, it includes the '/cid/inspect' route in the command tests to ensure comprehensive coverage of the new CID inspection functionality. * docs: update changelog for v0.42 to include new `ipfs cid inspect` command Added a section highlighting the new `ipfs cid inspect <cid>` command, detailing its functionality to display comprehensive CID information, including version, encoding, and hash details. The command supports machine-readable output and operates offline. * feat(cmd): improve ipfs cid inspect - multibase: always shown (implicit for CIDv0), prefix as string - multicodec/multihash: annotated (implicit) for CIDv0 - digest: uppercase hex with 0x prefix - cidV0: empty in JSON when not possible, text encoder explains why - cidV1: base36 for libp2p-key codec, base32 otherwise - errors: ErrorMsg kept for HTTP RPC API, text encoder returns non-zero exit - PeerID fallback: helpful hint with equivalent CID on invalid input - unknown codec/hash: graceful "unknown" label - stdin support via .EnableStdin() - inspect listed first in subcommands, cid format points to inspect - cli tests for all cases including JSON, PeerID, unknown codec * chore: move cid inspect changelog to v0.41 - digest: bare lowercase hex (no 0x prefix), matching sha256sum --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 5 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
feat(provide): add `ipfs provide once` and support Interval=0 mode (#11321) * feat(provide): add ipfs provide once for ad-hoc announcements Adds an experimental subcommand that submits provider records for the given CIDs through the provider system right away, without waiting for the next reprovide cycle. Use -r to walk the DAG and announce every reachable block. Designed against the sweep provider (the default since v0.39): StartProviding queues to the burst-provide workers, which publish records to the DHT efficiently. Works with the legacy provider too, though it queues into the slower serial worker pool. CIDs must already exist in the local blockstore. Re-announcement on the regular schedule is governed by Provide.Strategy and Provide.DHT.Interval; this command does not change either. * refactor(routing): deprecate ipfs routing provide Marks `ipfs routing provide` as deprecated and points users at the new `ipfs provide once`. The command keeps its existing Run, Encoders, and flags so existing scripts continue to work; only the status flag and helptext change. * docs(routing): clarify when ipfs routing reprovide applies Tightens the helptext and the sweep-mode error message so the constraint is obvious: this command only triggers a cycle on the legacy provider, and points users at 'ipfs provide stat --all' for monitoring the default sweep schedule. * docs: tighten provide helptext and update routing-provide references Updates docs/config.md and docs/experimental-features.md to reference 'ipfs provide once' instead of 'ipfs routing provide'. Tightens the helptext for 'ipfs provide clear' and the 'ipfs provide stat' overview: drops headings around short paragraphs, prefers active voice, and notes that the sweep provider is the default. * docs: changelog entry for ipfs provide once * docs: use 'provide system' wording consistently * test(provide): cover --recursive and multi-CID paths for provide once Adds two subtests under runProviderSuite (run for both Legacy and Sweep): - --recursive walks the DAG and announces every chunk of a 2 MiB file added with --pin=false under Provide.Strategy=roots, so the auto- provide path stays out of the way. - multiple CIDs in a single invocation succeed and the text encoder reports 'queued 3 CID(s) for immediate provide'. * feat(provide): stream cids and per-cid output for ipfs provide once Each CID flows through the command independently, so stdin can be piped without buffering and consumers see results as they happen. - Run reads CIDs from argv and then from BodyArgs (stdin scanner) one at a time, calling StartProviding per CID. - With -r, the dag.Walk visit callback emits per visited block; the walk cancels its context on the first announce error to stop fetching. - A typed ProvideOnceEvent (one per queued CID) replaces the prior batch result. JSON output streams {"Queued":"<cid>"} per line. - Text output via PostRun: when stderr is a tty, the running count is redrawn on a single line; otherwise a final count is printed. The text encoder still works for HTTP/RPC consumers (one CID per line). - Adds tests for stdin streaming and --enc=json one-event-per-line. * feat(provide): dedupe across all roots and recursive walks Previously the cid set was scoped per root, so a CID shared by two arguments or by two recursive DAG walks was announced twice. Move the set out to the Run scope so each unique CID is announced exactly once per invocation, regardless of how many times it shows up in argv, stdin, or the DAG walks. For -r, hitting an already-seen CID also stops descent into that subtree, avoiding redundant block fetches when DAGs overlap. * style(provide): rename useTTY to isTTY in PostRun * refactor(provide): align ipfs provide once with kubo cmds-lib idioms - Use the existing argumentIterator helper from cid.go to read argv followed by stdin, replacing the inlined two-loop variant. - Document why PostRun forks on encoder type (TTY redraw needs to bypass the encoder; json/xml must keep streaming through it). - Log an ERROR for unexpected response types instead of dropping them silently, mirroring the defensive pattern in cat.go's PostRun. * docs(routing): document streaming limitations of routing provide Spell out what 'ipfs routing provide' does worse than 'ipfs provide once' so users on the deprecation path know why to switch: input buffering, no per-cid output, no dedup across recursive roots, and the sync dht lookup that defeats sweep batching. * docs(changelog): rewrite ipfs provide once entry around user impact Recasts the highlight to lead with what the user can now do, not what the code does internally. Adds a one-line example showing the streaming stdin path that the previous version did not surface, and replaces "namespace" plumbing language with the actual capabilities (running count, json-per-line, single announcement per shared block under -r). * feat(provide): use boxo BloomTracker for cross-input dedup Swaps the cid.Set used by 'ipfs provide once' for the autoscaling boxo BloomTracker, the same dedup mechanism that powers Provide.Strategy=+unique. Run executes on the daemon, not the cli, so this caps daemon memory under hostile or accidental input: a user piping 100M cids previously would have grown the daemon's set to ~7 gb of resident memory; with the bloom chain it plateaus around 700 mb at the default fp rate, and under 100 mb up to 10m unique cids. The trade-off is a small false-positive rate (~1 in 4.75m, the kubo default) that can cause an occasional cid to be silently skipped. For ad-hoc providing this is acceptable; the regular reprovide cycle will pick up anything matched by Provide.Strategy on the next pass. * docs(changelog): use ipfs refs as the provide once example * style(provide): goimports import order * docs(provide): soften dedup wording, comment re.Emit gate, cover Provide.Enabled=false - Change "exactly once per invocation" to acknowledge the bloom false-positive rate now that the dedup is probabilistic. - Add a comment to the text branch of PostRun warning future readers not to call re.Emit there, since the encoder would race with the TTY counter. - Add a runProviderSuite subtest that exercises Provide.Enabled=false through the new code path (the existing routing-provide test only covers the deprecated alias's Run). * docs(changelog): clarify provide once use case and add second example - Note that provide once is also for fine-tuned control over which CIDs get announced when, alongside the regular reprovide schedule. - Add a second example using ipfs pin ls so users see the pattern for replaying their pinset alongside the dag-walk pattern. * feat(provide): error on ipfs provide once with Provide.DHT.Interval=0 When Provide.DHT.Interval=0, kubo wires NoopProvider via OnlineProviders -> OfflineProviders, so StartProviding silently no-ops and the cid never gets announced. provide once was returning success without any DHT publish: a footgun. Add an explicit precondition check that mirrors the routing reprovide error path. Decoupling the wiring so ad-hoc provide works under Interval=0 is tracked separately. * chore(deps): pin go-libp2p-kad-dht to PR #1246 head Pulls in the WithReprovideInterval(0) burst-only mode from https://github.com/libp2p/go-libp2p-kad-dht/pull/1246 so the kubo side of the Provide.DHT.Interval=0 decoupling can be developed against it. * chore(deps): re-pin go-libp2p-kad-dht to PR #1246 head Updates to the latest commit on the upstream branch (817031b) which also relaxes the dual SweepingProvider's reprovide-interval validator to accept 0, on top of the single-provider relaxation in the previous pseudo-version. * feat(provide): decouple Provide.DHT.Interval=0 from the master kill-switch Provide.Enabled is now the only switch that fully turns off the provide system. Provide.DHT.Interval=0 disables only the periodic reprovide schedule; new CIDs still announce via fast-provide-root and 'ipfs provide once'. - groups.go: drop the Interval=0 factor from isProviderEnabled. The real provider (sweep or legacy) is now wired even when Interval=0. - provider.go: skip the keystore sync goroutine in no-schedule mode. The ticker would panic on a zero interval, and with no schedule the keystore has no reader. - cmdenv/env.go: drop the fast-provide-root short-circuit on Interval=0. Provide.Enabled=false is now the only short-circuit. - commands/provide.go: drop the temporary 'cannot provide: Provide.DHT.Interval is 0' error from 'ipfs provide once'. - test/cli: replace the 'Reprovide.Interval=0 disables announcement of new CID too' test (premise is now false) with one asserting that Interval=0 + Enabled=true keeps announcing. Convert the provide-once + Interval=0 test from error path to success path. Tighten the legacy 'Manual Reprovide trigger' test to focus on the error contract. Requires upstream go-libp2p-kad-dht support for WithReprovideInterval(0) (kept under PR #1246). * feat(config): require explicit Provide.Enabled when Provide.DHT.Interval=0 Provide.DHT.Interval=0 used to disable the entire provide system as a side effect. After the decoupling it disables only the periodic reprovide schedule, while new CIDs still announce via fast-provide-root and 'ipfs provide once'. To prevent silent semantic drift on upgrade, the daemon now refuses to start when Interval is explicitly set to 0 unless Provide.Enabled is also set explicitly: - Provide.Enabled=false fully disables providing (the old behaviour). - Provide.Enabled=true keeps ad-hoc providing while skipping the periodic reprovide schedule. The error message names both options so operators can pick the one that matches their intent without reading the changelog. * docs: explain new Provide.DHT.Interval=0 semantic Updates docs/config.md and the v0.42 changelog: Interval=0 now disables only the periodic reprovide schedule, and the daemon refuses to start without an explicit Provide.Enabled in that configuration. Calls out both upgrade paths (Provide.Enabled=false to fully disable, or =true to keep ad-hoc providing). * chore(deps): re-pin go-libp2p-kad-dht to amended PR #1246 head Picks up the timeOffset/timeBetween zero-guards so SweepingProvider.Stats() no longer panics with reprovideInterval=0. Required for 'ipfs provide stat' to work in no-schedule mode. * test(provide): align test expectations with new no-schedule semantic - core/commands/commands_test.go: register /provide/once in the expected command list. - test/cli/provide_stats_test.go: 'ipfs provide stat' with Provide.DHT.Interval=0 now returns valid stats (with the schedule timing fields zeroed) instead of erroring out. Update the assertion to match. * chore(deps): re-pin go-libp2p-kad-dht to amended PR #1246 head Picks up the scheduleEnabled() consistency cleanup so timeOffset and timeBetween match the rest of the upstream gates. * chore(deps): re-pin go-libp2p-kad-dht to PR #1246 merge on master picks up the three follow-up commits guillaumemichel pushed before merging libp2p/go-libp2p-kad-dht#1246: - refactor: simplify StartProvide() - refactor: minimize change diff - fix: don't remove from keystore on StopProviding() * fix(provide): use ProvideOnce in `ipfs provide once` `ipfs provide once` was calling StartProviding, which in sweep mode persists keys to the keystore and adds them to the periodic reprovide schedule. that contradicts the command's name and help text. switch to ProvideOnce so the command publishes once and leaves the schedule untouched. for the legacy provider StartProviding already wraps ProvideOnce, so legacy behaviour is unchanged. also tighten the help text to state plainly that the schedule is not modified. * fix(provider): keep keystore inert when Provide.DHT.Interval=0 In no-schedule mode the keystore has no reader (no reprovide loop) and no writer (kad-dht's burst path skips Put/Delete). Until now we still opened on-disk leveldb/pebble files for it: wasted disk and noise on upgrade/downgrade. Switch the keystore to an in-memory map in no-schedule mode and make destroyDs a no-op. Also purge any pre-existing keystore directory once at startup so users who toggle from schedule to no-schedule reclaim disk. Replace the literal `reprovideInterval == 0` check at the second call site with the named noScheduleMode flag for consistency. | 3 个月前 | |
refactor: use slices.Sort where appropriate (#10858) | 1 年前 | |
fix(config): protect and derive PeerID during JSON replace (#11344) * commands: derive peer ID on config replace Signed-off-by: morning-verlu <258725120+morning-verlu@users.noreply.github.com> * feat: validate Identity.PeerID against node key Setting Identity.PeerID to a value that disagrees with the node's private key produces a config the node refuses to start with. The config command now validates the field against the stored key, the same way config replace re-derives it. - reject a mismatched Identity.PeerID and point at `ipfs key rotate`; accept the node's own PeerID in any base58 or CIDv1 form and store the canonical base58 string - share the PeerID derivation between the set path and config replace - document the identity fields and `ipfs key rotate` in docs/config.md - cover the set-path guard and base36 normalization in test/cli - switch the t0070 sharness probe off Identity.PeerID, now validated --------- Signed-off-by: morning-verlu <258725120+morning-verlu@users.noreply.github.com> Co-authored-by: morning-verlu <258725120+morning-verlu@users.noreply.github.com> Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 2 个月前 | |
test: TestEditorParsing (cherry picked from commit e44b53a7c9b73dd2b7df514f9633b1c992504d4c) | 1 年前 | |
feat(cli/rpc/add): fast provide of root CID (#11046) * feat: fast provide * Check error from provideRoot * do not provide if nil router * fix(commands): prevent panic from typed nil DHTClient interface Fixes panic when ipfsNode.DHTClient is a non-nil interface containing a nil pointer value (typed nil). This happened when Routing.Type=delegated or when using HTTP-only routing without DHT. The panic occurred because: - Go interfaces can be non-nil while containing nil pointer values - Simple `if DHTClient == nil` checks pass, but calling methods panics - Example: `(*ddht.DHT)(nil)` stored in interface passes nil check Solution: - Add HasActiveDHTClient() method to check both interface and concrete value - Update all 7 call sites to use proper check before DHT operations - Rename provideRoot → provideCIDSync for clarity - Add structured logging with "fast-provide" prefix for easier filtering - Add tests covering nil cases and valid DHT configurations Fixes: https://github.com/ipfs/kubo/pull/11046#issuecomment-3525313349 * feat(add): split fast-provide into two flags for async/sync control Renames --fast-provide to --fast-provide-root and adds --fast-provide-wait to give users control over synchronous vs asynchronous providing behavior. Changes: - --fast-provide-root (default: true): enables immediate root CID providing - --fast-provide-wait (default: false): controls whether to block until complete - Default behavior: async provide (fast, non-blocking) - Opt-in: --fast-provide-wait for guaranteed discoverability (slower, blocking) - Can disable with --fast-provide-root=false to rely on background reproviding Implementation: - Async mode: launches goroutine with detached context for fire-and-forget - Added 10 second timeout to prevent hanging on network issues - Timeout aligns with other kubo operations (ping, DNS resolve, p2p) - Sufficient for DHT with sweep provider or accelerated client - Sync mode: blocks on provideCIDSync until completion (uses req.Context) - Improved structured logging with "fast-provide-root:" prefix - Removed redundant "root CID" from messages (already in prefix) - Clear async/sync distinction in log messages - Added FAST PROVIDE OPTIMIZATION section to ipfs add --help explaining: - The problem: background queue takes time, content not immediately discoverable - The solution: extra immediate announcement of just the root CID - The benefit: peers can find content right away while queue handles rest - Usage: async by default, --fast-provide-wait for guaranteed completion Changelog: - Added highlight section for fast root CID providing feature - Updated TOC and overview - Included usage examples with clear comments explaining each mode - Emphasized this is extra announcement independent of background queue The feature works best with sweep provider and accelerated DHT client where provide operations are significantly faster. * fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * Update core/commands/add.go --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 9 个月前 | |
refactor: namesys cleanup, gateway /ipns/ ttl (#10115) | 2 年前 | |
feat: bound graceful shutdown, add diag healthy (#11329) * feat: bound graceful shutdown, add diag healthy Replace unbounded app.Stop(context.Background()) with a deadline-bounded context driven by a new Internal.ShutdownTimeout config (default 12h, 0 disables). Add an os.Exit(1) watchdog at the same deadline so an FX OnStop hook that never returns can no longer hang the daemon. Add ipfs diag healthy: fails when shutdown has been initiated or when the DAG pipeline cannot resolve the well-known empty-directory CID. Dockerfile HEALTHCHECK now uses it so orchestrators recycle half- shutdown daemons. - core/shutdown: new pkg; atomic startedAt + CloseWithCtx helper - core/builder.go: app.Stop bounded by ShutdownTimeout - cmd/ipfs/kubo/daemon.go: watchdog + MarkStarted on signal - core/commands/diag.go: new healthy subcommand - core/node/{bitswap,libp2p/host,libp2p/routing}.go: OnStop hooks wrapped - config/internal.go: ShutdownTimeout + DefaultShutdownTimeout=12h - Dockerfile: HEALTHCHECK uses "ipfs diag healthy" - docs/{config,changelogs/v0.42}.md: documented - test/cli: enabled + disabled path tests * feat: bound provider stats and ADD_PROVIDER sends bumps go-libp2p-kad-dht past v0.39.2 to b73e1e8 to pick up two related provider bug fixes. - ipfs provide stat now honors client cancellation and deadlines instead of blocking indefinitely behind a slow keystore lookup - adds Provide.DHT.SendProviderRecordTimeout capping each ADD_PROVIDER RPC so unresponsive peers cannot pin a provide worker and stall reprovide cycles - internal reprovide-alert poller bounds its Stats call so a hung keystore.Size cannot delay shutdown * test(shutdown): use synctest for timeout test, document sleep CloseWithCtx_timesOut now runs in a synctest bubble so the deadline assertion is exact (no wall-clock slack), and the simulated close uses a release channel to drain the bubble cleanly after the leak point. The two happy-path tests stay unchanged because their close funcs return immediately and gain nothing from a fake clock. Comment the 2ms sleep in TestMarkStartedPreservesFirstTimestamp so its role (forcing time.Now() to advance between the two MarkStarted calls so a CAS to Store regression is detectable) is not lost. Addresses ipfs/kubo#11329 (review). * fix(pinner): bound pinner Close with shutdown deadline The boxo Pinner.Close contract notes that an in-flight op ignoring its ctx (a downstream bug) can block Close, so the host must bound it at the call site. Wrapping the OnStop hook with CloseWithCtx honors Internal.ShutdownTimeout and surfaces an actionable "subsystem 'pinner' failed to close" log on hang instead of leaving only the watchdog os.Exit(1) trace. * fix(shutdown): bound remaining I/O-touching OnStop hooks Wrap the OnStop hooks whose Close can plausibly block on disk or network: repo (datastore flush + lock release), mfs-root (datastore writes via DAGService), peering (waits on libp2p peer goroutines), legacy-provider (in-flight reprovide RPCs), and the dht-provider plus keystore pair under SweepingProvider. In-memory closes (blockservice, peerstore, resource-manager) are left as-is since they cannot realistically hang. For the dht-provider/keystore pair, provider closes first so nothing can access the keystore afterwards. If the shutdown ctx fires mid-provider-drain, the keystore close sees an expired ctx and returns immediately; the watchdog os.Exit(1) is the ultimate backstop, and keystore writes are fsync'd on put so missing the explicit close is recoverable on next boot. * fix(shutdown): bound remaining in-memory OnStop hooks Wrap blockservice, peerstore, and resource-manager Close hooks with CloseWithCtx for uniformity. These are pure in-memory operations unlikely to hang in practice, but wrapping costs nothing and makes the shutdown audit trail uniform: every OnStop hook now honors the deadline and surfaces a named subsystem on timeout. * fix(shutdown): bound autoRelayFeeder OnStop on ctx OnStop waited on the feeder goroutine via <-done without honoring the shutdown ctx. The goroutine itself selects on ctx in every loop case, so cancel() normally suffices, but a stuck downstream dht.WAN.GetClosestPeers that ignored its ctx could block fx.Stop indefinitely. Adding the ctx.Done() select case mirrors the reprovideAlert pattern in provider.go and lets the shutdown deadline reclaim control even with a misbehaving DHT. * docs(changelog): merge shutdown entries into one user-facing section Combine the pinner-on-shutdown paragraph with the bounded-shutdown section under a single "Reliable shutdown and container health checks" heading. Lead with the visible symptoms (half-shutdown daemons, healthy-but-dead container reports, manual docker restart) instead of fx OnStop jargon. Frame Internal.ShutdownTimeout as a belt-and-suspenders ceiling, with the 12-hour default sized against the 22-hour DHT provider record expiration. | 3 个月前 | |
style: gofumpt and godot [skip changelog] (#10081) | 3 年前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
fix(mfs): stop repo gc from freezing files ops (#11386) * fix(mfs): stop repo gc from freezing files ops Running `ipfs repo gc` alongside `ipfs files` writes could leave MFS permanently hung: GC deleted directory-node blocks a write had written to the blockstore but not yet linked into the persisted MFS root, and the next path lookup blocked forever fetching the missing block while holding the MFS directory lock, so every later files command piled up behind it. MFS mutations now take the pin lock, the same lock `ipfs add` uses, and GC computes the MFS root only after it holds the GC lock. A write's blocks are therefore either fully linked into the root before GC runs, or the write waits for GC to finish, so GC never collects data a live write still needs. - core/commands/files.go: hold the pin lock across write, cp, mkdir, mv, rm, flush, chcid, chmod, and touch - gc/gc.go: take the best-effort MFS root snapshot after acquiring the GC lock, closing the snapshot-before-lock window - core/corerepo/gc.go: pass the root as a callback evaluated under the lock - core/node/core.go: document why MFS keeps its online DAG service so lazy `ipfs files cp /ipfs/<cid>` pointers still resolve - gc/gc_test.go: assert the root snapshot is taken under the GC lock Closes #10842 * fix(mfs): extend gc pin lock to add and fuse The pin lock that stops garbage collection from collecting live MFS blocks now covers every path that mutates MFS, not just `ipfs files`, and every live MFS root is part of the GC live set. - core/commands/add.go: hold the pin lock while `ipfs add --to-files` links the added content into the MFS root - fuse/writable, fuse/mfs: hold the pin lock across FUSE `/mfs` structural writes and file flush, fsync, and release - fuse/ipns: same for the per-key `/ipns` mounts, and register their roots so GC keeps their blocks live - core/core.go: track mounted MFS roots for the GC live set - core/corerepo/gc.go: build the live set from the files root plus every registered root, skipping a root that errors instead of aborting the whole GC Closes #6113 Closes #7008 Closes #9553 * fix(mfs): fail fast on a missing block A directory-node block that is missing locally and unreachable would block an MFS operation forever while it held the directory lock, wedging the whole MFS and a clean shutdown until the process was killed. This is what remained after a repo was damaged by an older Kubo, a manual `ipfs block rm`, or a crash (the lockup in #7844). MFS now bounds those network reads: an unreachable block fails the operation with a timeout and releases the lock, while lazily-referenced content (`ipfs files cp /ipfs/<cid>`) still loads as before. - config: add DefaultMFSFetchTimeout and pass it to the MFS root via mfs.WithFetchTimeout in Import.MFSRootOptions - go.mod: bump boxo to pull in mfs.WithFetchTimeout * feat(mfs): honor --timeout in files read and write Adopt boxo's mfs.File.Open(ctx) so MFS file operations carry a context and a read or write stuck on a missing block can be cancelled instead of blocking forever. - go.mod: bump boxo to pick up mfs.File.Open(ctx) - core/commands/files.go: pass the request context to files read and write, so a client --timeout ends a stuck read or write - fuse/writable: bind long-lived FUSE descriptors (Create, Open) to the mount context via the new Config.MountCtx; the transient Setattr truncate uses the per-operation context - fuse/mfs, fuse/ipns: set MountCtx to the node/mount context - test/cli: cover files read and write honoring --timeout on unreachable content * chore(deps): bump boxo to merged mfs fix Switch from the pre-merge branch pseudo-version to the merged commit of ipfs/boxo#1185, the boxo side of the MFS under-lock work this branch depends on. * refactor(fuse): pass request ctx to pin lock Flush, Release, and Fsync passed context.Background() to the pin lock while their own ctx argument was in scope. Thread the passed ctx through instead, matching the other handlers. boxo's default GCLocker discards the context, so this is a consistency change, not a behavior change. https://github.com/ipfs/kubo/pull/11386#discussion_r3534131824 https://github.com/ipfs/kubo/pull/11386#discussion_r3534138912 https://github.com/ipfs/kubo/pull/11386#discussion_r3534140861 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(gc): const for buffer, simplify roots Name the GC result channel buffer size as a const, drop the temporary in the best-effort root snapshot, and use t.Context() in the snapshot test. The buffer size predates this branch; it only shifted in the diff. https://github.com/ipfs/kubo/pull/11386#discussion_r3534154606 https://github.com/ipfs/kubo/pull/11386#discussion_r3534077507 https://github.com/ipfs/kubo/pull/11386#discussion_r3534085418 https://github.com/ipfs/kubo/pull/11386#discussion_r3534106766 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * refactor(add): collapse pin lock defer Fold the pin lock's unlock into the defer at both --to-files call sites, matching the one-line style used elsewhere. https://github.com/ipfs/kubo/pull/11386#discussion_r3534006489 https://github.com/ipfs/kubo/pull/11386#discussion_r3534013057 Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
refactor: migrate away from cheggaaa/pb v1 (#11322) * refactor: migrate away from cheggaaa/pb v1 * updated changelogs * fix: add space after comment slashes for consistency * refactor: share terminal detection in cmdenv Replace three duplicate TTY checks (get.go, dag/export.go, dag/stat.go) with `cmdenv.IsTerminal(*os.File)` backed by `mattn/go-isatty`. The helper uses `IsTerminal || IsCygwinTerminal`, which also detects MSYS2 and Git Bash on Windows. Those terminals expose stdio as a named pipe rather than a character device, so the previous `ModeCharDevice` check suppressed the progress bar on real terminals. - core/commands/cmdenv/tty.go: new helper - core/commands/{add,cat,get}.go: drop local isStderrTTY - core/commands/dag/{export,stat}.go: drop inline stat() block - go.mod: promote mattn/go-isatty to direct (was indirect via pb/v3) * refactor: cmdenv.ShouldShowProgress helper Collapse the explicit-flag-or-TTY-default logic at four call sites (`cat`, `get`, `dag export`, `dag stat`) into a single helper. * refactor: dedupe `ipfs add` progress template The full bar template (counters, bar, speed, percent, ETA) was inlined at two call sites in add.go. Move it to a file-level const. * fix: progress bar shows MiB/s, not MiB p/s pb v3's speed element defaults to suffix "%s p/s", so even with pb.Bytes set, `ipfs add`, `ipfs cat`, `ipfs get`, and `ipfs dag export` rendered the rate as "713.04 MiB p/s" instead of "713.04 MiB/s". Pass explicit format args to the speed and rtime template elements: rate now renders as "MiB/s", and the unknown-state fallback reads "?/s" / "ETA ?" instead of bare "?". The four templates move to package-level consts. * docs: rewrite v0.42 progress bar entry Describe only the user-visible changes; skip library-migration detail and intermediate-state claims that never shipped. * chore: drop unused pb v1 dependabot ignore The `github.com/cheggaaa/pb` (v1) module path is no longer in `go.mod` after the migration to `pb/v3`, so the ignore rule never fires. * fix(dag): unify --progress help text Match the wording used by `add`, `cat`, and `get`: "Stream progress data. Defaults to true when stderr is a terminal." * fix(add): finalize progress bar after upload Call `bar.Finish()` and a final `bar.Write()` after the progress loop. Without it, fast adds (under ~500ms, where pb/v3's EWMA never accumulates a speed sample) render `?/s ... ETA ?` in the last frame. Finishing the bar switches the speed element to its absolute-rate branch (total/elapsed), so the final frame now reads e.g. `792.04 MiB/s 100.00% 100ms`. * test(cmdenv): cover ShouldShowProgress Exercise the explicit-true, explicit-false, unset, and non-bool paths. Unset and non-bool fall back to IsTerminal(os.Stderr), which the test compares against directly so it works in both TTY and CI environments. * refactor: share full progress bar template Move the "total known" pb/v3 template to cmdenv.ProgressBarFullTemplate so add.go and get.go reference the same string instead of keeping byte-identical local copies. The add init template and dag/export streaming template stay local because each is single-use and shaped differently. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 3 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
gx: unrewrite License: MIT Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch> | 7 年前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
fix(key): restrict overwritten key exports to owner-only permissions (#11428) * fix(key): restrict overwritten key exports to owner-only permissions Signed-off-by: questfever <questfever@outlook.com> * fix(atomicfile): temp file leak and name limit Both problems surface through `ipfs key export`, which now writes through this helper, but they affect every caller: config writes, repo migrations and `ipfs update`. - remove the temporary file when the rename fails, so one holding private key material is not left next to the target - keep the ".tmp-" prefix and the random suffix within the 255 byte file name limit, so a target with a long name can still be written * fix(key): route key export by target type Choosing the write path with os.Lstat treated /dev/stdout, /dev/stderr and /dev/fd/N as plain paths, because they are symlinks into /proc/self/fd, so the atomic write failed on targets that worked before. Decide by what the path resolves to, and state the whole contract in the command help. - regular file or nothing yet: written to a temporary file and renamed over the target, following symlinks, including one whose target does not exist yet - character device or pipe: streamed in place, confirmed on the open descriptor and without O_TRUNC, so a path swapped for a regular file can neither receive the key nor be emptied - anything else: refused, naming the path - errors name the file the user asked for, and the temporary file is flushed before the rename * fix(key): stop export landing on the wrong file An export could replace a file the target symlink does not point at. resolveSymlink joined a relative link target onto the path as typed, so ".." collapsed lexically. Where a parent component was itself a symlink, the join named a file outside the directory the link resolves to: the key was renamed over that file, and the intended target was never written. The link's parent is now resolved with filepath.EvalSymlinks before the join. * test(key): drop umask dependency in export test os.WriteFile applies the umask, so under umask 077 the fixture was created 0600 and the check that a failed export leaves the file at 0644 failed for reasons unrelated to the code under test. The CLI test already chmods for the same reason. --------- Signed-off-by: questfever <questfever@outlook.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 26 天前 | |
fix(key): restrict overwritten key exports to owner-only permissions (#11428) * fix(key): restrict overwritten key exports to owner-only permissions Signed-off-by: questfever <questfever@outlook.com> * fix(atomicfile): temp file leak and name limit Both problems surface through `ipfs key export`, which now writes through this helper, but they affect every caller: config writes, repo migrations and `ipfs update`. - remove the temporary file when the rename fails, so one holding private key material is not left next to the target - keep the ".tmp-" prefix and the random suffix within the 255 byte file name limit, so a target with a long name can still be written * fix(key): route key export by target type Choosing the write path with os.Lstat treated /dev/stdout, /dev/stderr and /dev/fd/N as plain paths, because they are symlinks into /proc/self/fd, so the atomic write failed on targets that worked before. Decide by what the path resolves to, and state the whole contract in the command help. - regular file or nothing yet: written to a temporary file and renamed over the target, following symlinks, including one whose target does not exist yet - character device or pipe: streamed in place, confirmed on the open descriptor and without O_TRUNC, so a path swapped for a regular file can neither receive the key nor be emptied - anything else: refused, naming the path - errors name the file the user asked for, and the temporary file is flushed before the rename * fix(key): stop export landing on the wrong file An export could replace a file the target symlink does not point at. resolveSymlink joined a relative link target onto the path as typed, so ".." collapsed lexically. Where a parent component was itself a symlink, the join named a file outside the directory the link resolves to: the key was renamed over that file, and the intended target was never written. The link's parent is now resolved with filepath.EvalSymlinks before the join. * test(key): drop umask dependency in export test os.WriteFile applies the umask, so under umask 077 the fixture was created 0600 and the check that a failed export leaves the file at 0644 failed for reasons unrelated to the code under test. The CLI test already chmods for the same reason. --------- Signed-off-by: questfever <questfever@outlook.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 26 天前 | |
feat: add query functionality to log level command (#10885) * feat: update log level command to show log levels * test: add log level tests * update TestCommands test * docs: relation to GOLOG_LOG_LEVEL * chore: update to latest go-log * fix: do not output single subsystem name in CLI * test: explicit subsystem request dont output subsystem * LevelFromString renamed to Parse * Modify `ipfs log level` * Denote default level with sdubsystem name '(defult)'. * make "*" an dalias for "all". Test to make sure both work the same. | 1 年前 | |
feat(cli): add --human and --sort-size to ipfs ls (#11408) * feat(cli): add --human and --sort-size to ipfs ls - --human (-H): SI human-readable sizes in text output (humanize.Bytes) - --sort-size (-S): sort directory entries by size, largest first - Validation: --sort-size + --stream and --sort-size + --size=false errors - Unit tests for formatSize and sort helpers - CLI integration tests for both flags - Changelog highlight in v0.44 * test(ls): make sort tests fail when sorting breaks The tests covering --sort-size could not detect a broken feature. The unit tests copied the comparator into the test body and sorted with their own copy, so they passed regardless of what ls.go did. The CLI tests named each fixture after its size, which left alphabetical order and size order in agreement, so every ordering subtest passed even with --sort-size disabled outright. - extract lsLinkByName and lsLinkBySize so the tests exercise shipped code - point the unit tests at those two functions - name fixtures so their alphabetical order disagrees with their sizes - pin the real directory behaviour: UnixFS directories carry no Filesize, so they sort as 0, tie with empty files and break by name rather than landing strictly last * fix(ls): return 400 not 500 for bad flag combos Passing --sort-size together with --stream or --size=false made /api/v0/ls answer 500 Internal Server Error, telling API clients the server had broken when the caller had simply combined flags that cannot work together. Clients that retry on 5xx would retry a request that can never succeed. cmds.ErrClient maps to 400, matching how the rest of core/commands reports caller mistakes. CLI output is unchanged. * docs: fix --human size examples and JSON claims The help for --human advertised "1K 234M 2G", which no kubo command has ever printed. All three use humanize.Bytes, so the real output is SI with a space: 1.2 kB, 234 MB, 2.0 GB. The stale example was copied into 'ipfs ls' from the two commands that already carried it, so correct all three together. - ls, repo stat, bitswap stat: examples now match real output - drop the claim that --enc=json reports bytes. On the CLI, 'ipfs ls' has a PostRun that prints the text table whatever --enc says, so no JSON is produced there at all. Only the /api/v0/ls response is JSON, and that part is true - directories have no UnixFS Filesize, so they sort as 0 and tie with empty files. They do not land strictly last, so stop saying they do - changelog: a #### highlight with a TOC entry, kept short and pointing at 'ipfs ls --help' for the details - format sizes with strconv.FormatUint rather than fmt.Sprintf("%d") --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 个月前 | |
feat(cli): add --human and --sort-size to ipfs ls (#11408) * feat(cli): add --human and --sort-size to ipfs ls - --human (-H): SI human-readable sizes in text output (humanize.Bytes) - --sort-size (-S): sort directory entries by size, largest first - Validation: --sort-size + --stream and --sort-size + --size=false errors - Unit tests for formatSize and sort helpers - CLI integration tests for both flags - Changelog highlight in v0.44 * test(ls): make sort tests fail when sorting breaks The tests covering --sort-size could not detect a broken feature. The unit tests copied the comparator into the test body and sorted with their own copy, so they passed regardless of what ls.go did. The CLI tests named each fixture after its size, which left alphabetical order and size order in agreement, so every ordering subtest passed even with --sort-size disabled outright. - extract lsLinkByName and lsLinkBySize so the tests exercise shipped code - point the unit tests at those two functions - name fixtures so their alphabetical order disagrees with their sizes - pin the real directory behaviour: UnixFS directories carry no Filesize, so they sort as 0, tie with empty files and break by name rather than landing strictly last * fix(ls): return 400 not 500 for bad flag combos Passing --sort-size together with --stream or --size=false made /api/v0/ls answer 500 Internal Server Error, telling API clients the server had broken when the caller had simply combined flags that cannot work together. Clients that retry on 5xx would retry a request that can never succeed. cmds.ErrClient maps to 400, matching how the rest of core/commands reports caller mistakes. CLI output is unchanged. * docs: fix --human size examples and JSON claims The help for --human advertised "1K 234M 2G", which no kubo command has ever printed. All three use humanize.Bytes, so the real output is SI with a space: 1.2 kB, 234 MB, 2.0 GB. The stale example was copied into 'ipfs ls' from the two commands that already carried it, so correct all three together. - ls, repo stat, bitswap stat: examples now match real output - drop the claim that --enc=json reports bytes. On the CLI, 'ipfs ls' has a PostRun that prints the text table whatever --enc says, so no JSON is produced there at all. Only the /api/v0/ls response is JSON, and that part is true - directories have no UnixFS Filesize, so they sort as 0 and tie with empty files. They do not land strictly last, so stop saying they do - changelog: a #### highlight with a TOC entry, kept short and pointing at 'ipfs ls --help' for the details - format sizes with strconv.FormatUint rather than fmt.Sprintf("%d") --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 个月前 | |
fix(fuse): switch to hanwen/go-fuse (#11272) * test(fuse): consolidate FUSE tests into test/cli/fuse Move FUSE integration tests from sharness shell scripts (t0030, t0031, t0032) and test/cli/fuse_test.go into a dedicated test/cli/fuse/ Go sub-package, ensuring all FUSE test cases run in CI. - git mv test/cli/fuse_test.go to test/cli/fuse/ (package fuse) - convert all sharness FUSE tests to Go subtests under TestFUSE: mount failure, IPNS symlink, IPNS NS map resolution, MFS file/dir creation, xattr (Linux), files write, add --to-files, file removal, nested dirs, publish-while-mounted block, sharded directory reads - add xattr helpers with build tags (linux/other) using unix.Getxattr - split make test_fuse into test_fuse_unit (./fuse/...) and test_fuse_cli (./test/cli/fuse/...) sub-targets - set TEST_FUSE=0 in test_cli so FUSE tests skip in cli-tests CI job - increase fuse-tests CI timeout from 5m to 10m for CLI tests - delete sharness t0030, t0031, t0032 (were always skipped in CI) * docs: document FUSE test split between unit and e2e Add cross-reference comments between the unit tests in fuse/readonly/, fuse/ipns/, fuse/mfs/ and the end-to-end CLI tests in test/cli/fuse/. Also fix AGENTS.md to use a temp dir for fusermount symlink instead of sudo. * ci: prevent stale FUSE mounts from failing fuse-tests On shared self-hosted runners, leftover mount points from previous runs can exhaust the kernel FUSE mount limit. - add job-level concurrency group so only one fuse-tests runs at a time - lazy-unmount stale /tmp/fusetest* mounts before running tests * ci: only symlink fusermount3 when fusermount is missing * fix(fuse): remove goroutine leak in IPNS Flush handler The Flush handler wrapped fi.fi.Flush() in a goroutine so it could return early when the FUSE context was canceled. But the goroutine kept running in the background, and when Release arrived it called Close on the same file descriptor concurrently. The two paths both entered DagModifier.Sync, racing on its internal write buffer and causing a nil pointer panic. The fix is to call Flush directly without a goroutine. The MFS flush cannot be safely canceled mid-operation anyway, so the goroutine only added the illusion of cancellation while leaking work and masking the real error. Also bumps boxo to pick up the matching defense-in-depth fix that serializes FileDescriptor.Flush and Close with a mutex. * fix(fuse): add mutex to IPNS file handle operations bazil/fuse dispatches each FUSE request in its own goroutine. The IPNS File handle had no synchronization, so concurrent Read/Write/Flush/Release calls could overlap on the underlying DagModifier which is not safe for concurrent use. Add sync.Mutex to File, matching the pattern already used by the MFS FileHandler. * refactor(fuse): remove dead File.Forget method bazil/fuse only dispatches Forget to nodes via the NodeForgetter interface. File is a handle, not a node, so this method was never called. The /mfs mount has no equivalent. * fix(fuse): flush IPNS directory after Remove and Rename The /mfs mount flushes the directory after Unlink and Rename so changes propagate to the MFS root immediately. The /ipns mount did not, leaving mutations pending until an unrelated flush. Also add an empty-directory check before removing directories, matching the /mfs mount's safety check. * fix(fuse): inherit CID builder and flush on IPNS Create New files created via the /ipns FUSE mount now inherit the CID builder from their parent directory, preventing CIDv0 nodes from appearing inside a CIDv1 tree. The directory is also flushed after AddChild so the new entry propagates to the MFS root immediately, matching the /mfs mount. * test(fuse): add IPNS Remove and non-empty rmdir tests Cover the file removal path and the empty-directory safety check added in the previous commit. TestRemoveFile verifies a created file can be removed and is gone afterwards. TestRemoveNonEmptyDirectory verifies that rmdir on a directory with children fails, and succeeds once the children are removed first. * feat(fuse): read UnixFS mode/mtime, add StoreMtime/StoreMode config All three FUSE mounts now read mode and mtime from UnixFS metadata when present, falling back to POSIX defaults when absent. Most IPFS data does not include this optional metadata. Writing mode and mtime is opt-in via two new config flags: - Mounts.StoreMtime: persist mtime on file create and open-for-write - Mounts.StoreMode: persist mode on chmod Other changes in this commit: - align default file/dir modes across /ipns and /mfs to 0644/0755 - share mode constants via fuse/mount/mode.go - convert Mounts.FuseAllowOther from bool to Flag for consistency - add Setattr to /ipns FileNode and /mfs File for chmod and touch - move dead File.Setattr from IPNS handle to FileNode (node) - bump boxo for Directory.Mode() and Directory.ModTime() getters * feat(fuse): add ipfs.cid xattr to all mounts All three FUSE mounts now expose the node's CID via the ipfs.cid extended attribute on both files and directories. The /mfs mount also accepts the old ipfs_cid name for backward compatibility. The /ipfs mount previously had a stub that returned nil for all xattrs; it now returns the correct CID. The xattr name follows the convention used by CephFS (ceph.*), Btrfs (btrfs.*), and GlusterFS (glusterfs.*). * feat(fuse): switch from bazil.org/fuse to hanwen/go-fuse v2 Replace the unmaintained bazil.org/fuse (last commit 2020) with hanwen/go-fuse v2.9.0, fixing two architectural issues that could not be solved with the old library. ftruncate now works: hanwen/go-fuse passes the open file handle to NodeSetattrer, so Setattr can truncate through the existing write descriptor instead of trying to open a second one (which deadlocks on MFS's single-writer lock). fsync now works: FileFsyncer runs on the handle directly, flushing the write buffer through the open descriptor. Previously a no-op because bazil dispatched Fsync to the inode only. mount package: - NewMount takes (InodeEmbedder, mountpoint, *fs.Options) instead of (fs.FS, mountpoint, allowOther) - mount/unmount collapses to a single fs.Mount call - fusermount3 tried before fusermount in ForceUnmount all three mounts: - structs embed fs.Inode (hanwen's InodeEmbedder pattern) - Remove split into Unlink + Rmdir (separate FUSE interfaces) - ReadDirAll replaced with Readdir returning DirStream - fillAttr helper shared between Getattr and Lookup responses - kernel cache invalidation via NotifyContent after Flush - 1s entry/attr timeout for writable mounts (matches go-fuse default, gocryptfs, rclone) - O_APPEND tracked on file handle, writes seek to end - build tags standardized to (linux || darwin || freebsd) && !nofuse tests: - replaced bazil fstestutil.MountedT with shared fusetest.TestMount - fixed TestConcurrentRW: channel drain mismatch and missing sync between write Close and read start - added TestFsync, TestFtruncate, TestReadlink, TestSeekRead, TestLargeFile, TestRmdir, TestCrossDirRename, TestUnknownXattr - added StoreMtime disabled/enabled subtests * fix(fuse): close fd on error in Open to prevent leak MFS enforces a single-writer lock, so a leaked write descriptor blocks all subsequent opens of that file until GC. * fix(fuse): detect external unmount via server.Wait Without this, IsActive stays true after `fusermount -u` and Unmount returns nil instead of ErrNotMounted. * fix(fuse): return actual error from Unlink/Rmdir, not ENOENT After confirming the child exists, an Unlink failure could be an IO error. Returning ENOENT would hide the real cause. * fix(fuse): reuse DagReader per open, pass ctx to all reads Readonly Open now returns a file handle holding a DagReader instead of recreating one per Read call. Sequential reads no longer re-traverse the DAG from the root on each kernel request. All three mounts now use CtxReadFull with the kernel's per-request context so killing a process mid-read cancels in-flight block fetches instead of letting them complete uselessly. * chore(fuse): cleanup dead code, add var comments - remove dead `_ = mntDir` in TestXattrCID - comment why immutableAttrCacheTime and mutableCacheTime are var - add TODO for using IPNS record TTL as cache timeout * chore(fuse): replace OSXFUSE 2.x check with macFUSE detection The old check tried to verify OSXFUSE >= 2.7.2 to avoid a kernel panic from 2015. It used sysctl, tried to `go install` a third-party tool at runtime, and referenced paths that no longer exist. Replace with a simple check for the macFUSE mount helper, matching the same paths go-fuse looks for. If neither macFUSE nor OSXFUSE is found, point the user to the install page. Also standardize build tags to (linux || darwin || freebsd) && !nofuse and use strings.ReplaceAll. * fix(fuse): include mountpoint path in mount errors go-fuse's fusermount errors don't include the path, so tools that check error messages for the mountpoint name couldn't tell which mount failed. * chore(ci): remove bazil fusermount workaround go-fuse finds fusermount3 natively, no symlink needed. The stale mount cleanup was for bazil's fstestutil which we no longer use. * docs: update v0.41 changelog for FUSE rewrite * chore(deps): bump boxo for full FileDescriptor serialization boxo@64be0815 extends the mutex from Flush/Close to all FileDescriptor operations (Read, Write, Seek, Truncate, Size), preventing data races on the underlying DagModifier. * chore(deps): bump boxo to merged ipfs/boxo#1133 Picks up full FileDescriptor serialization: the mutex now covers all operations (Read, Write, Seek, Truncate, Size), not just Flush and Close. * feat(fuse): CAP_ATOMIC_O_TRUNC, new integration tests Advertise CAP_ATOMIC_O_TRUNC so the kernel sends O_TRUNC inside Open instead of doing a separate SETATTR(size=0) first. Without this, the kernel's SETATTR needs to open a write descriptor inside Setattr, which deadlocks on MFS's single-writer lock. Move kernel cache invalidation from Flush to Release because mfsFD.Close (in Release) is where the final DAG node is committed. Upgrade go-fuse to latest for ExtraCapabilities support. New tests for both MFS and IPNS: - TestOpenTrunc, TestSeekAndWrite, TestOverwriteExisting - TestTempFileRename, TestVimSavePattern, TestRsyncPattern (skipped pending rename-over-existing and cache fixes) * fix(fuse): rename-over-existing, bump boxo for flushUp race fix IPNS Rename now unlinks the target before AddChild, matching MFS. Without this, renaming onto an existing name returned "directory already has entry". Bump boxo to pick up the flushUp unlinked-entry fix (ipfs/boxo@8ae46d5): when a file descriptor outlives its directory entry (FUSE RELEASE racing with RENAME), flushUp no longer re-adds the stale name. Unskip TestTempFileRename and TestRsyncPattern on both mounts. * fix(fuse): unskip VimSavePattern, bump boxo for setNodeData fix boxo@552d8e7 fixes File.setNodeData dropping content links when updating metadata (mode, mtime). chmod or touch after write no longer makes the file appear empty. Unskip TestVimSavePattern on both mounts. Remove debug logging and temporary test functions added during investigation. * fix(fuse): build tags for cross-compilation go-fuse does not compile on windows/openbsd/netbsd/plan9. Move WritableMountCapabilities (which imports go-fuse) from mode.go (no build tag) to caps.go (platform-gated). Align build tags on fusetest and core/commands/mount stubs so unsupported platforms don't pull in go-fuse transitively. * fix(test): use fusermount3 in CLI FUSE tests The doUnmount helper hardcoded fusermount, but systems with only fuse3 installed have fusermount3. Try fusermount3 first, matching what go-fuse and our ForceUnmount already do. * feat(fuse): symlink support on writable mounts Add NodeSymlinker to MFS and IPNS directories. Symlinks are stored as UnixFS TSymlink nodes in the DAG, the same format used by `ipfs add` for directories containing symlinks. The readonly /ipfs mount already rendered existing symlinks; now /mfs and /ipns can create them too. The target string is cached at Lookup time to avoid re-parsing the DAG node on every Readlink call. Symlink permissions are always 0777 per POSIX convention (access control uses the target's mode). * fix(fuse): checked type assertion in MFS Rename The direct type assertion on newParent could panic if the kernel passed a non-directory inode. Use a checked assertion with EINVAL fallback, matching the type-switch pattern in the IPNS mount. * fix(test): add missing continue in stress test Missing continue after error sends let execution fall through to nil type assertions (read.(files.File)) that would panic on error. Also cancel the context before continuing to avoid leaking it. * fix(fuse): return error from Readdir when DAG.Get fails Abort the directory listing instead of silently omitting the unretrievable entry. Callers get EIO, which is more honest than a partial listing that hides missing blocks. * docs: remove duplicate fsync bullet in changelog * ci: clean up stale FUSE mounts in fuse-tests job On shared self-hosted runners, leftover mounts from crashed runs can exhaust the kernel mount_max limit. Lazy-unmount kubo-test and harness temp mounts before and after tests. * chore(deps): bump boxo to merged ipfs/boxo#1134 Picks up flushUp unlinked-entry guard and setNodeData content link preservation. * docs: add build tag comments, normalize tag style Add a one-line comment above every //go:build directive explaining why the constraint exists. Normalize tag style: positive platform constraints first, then feature flags/negations. Simplify redundant expressions. * fix(fuse): add Setattr to directories for chmod and mtime Tools like tar and rsync call utimensat on directories after extraction. Without Setattr on Dir, this returned ENOTSUP. Add Setattr to Dir (MFS) and Directory (IPNS) that handles mode and mtime the same way as the file-level Setattr. When StoreMtime or StoreMode is disabled the call succeeds silently, matching the file-level behavior. * docs: clarify directory support and spec link for StoreMtime/StoreMode - mention that touch and chmod work on both files and directories - note tar and rsync as practical use cases - link to UnixFS spec for optional metadata storage * fix(fuse): use proper mode conversion, document 9-bit limit Use files.UnixPermsToModePerms and files.ModePermsToUnixPerms for converting between FUSE kernel mode (unix 12-bit layout) and Go's os.FileMode (different bit positions for setuid/setgid/sticky). The UnixFS spec supports all 12 permission bits, but boxo's MFS layer (File.Mode, Directory.Mode) exposes only the lower 9. FUSE mounts are always nosuid so the upper 3 bits would have no effect. Add TestSetuidBitsStripped to both mounts confirming the behavior. * feat(fuse): symlink Setattr with mtime persistence Wire the backing mfs.File into the FUSE Symlink struct so Setattr can call SetModTime when StoreMtime is enabled. boxo's File methods (SetModTime, ModTime) already work on TSymlink nodes since they operate on the FSNode protobuf without checking the type. Without Setattr, rsync -a fails with "failed to set times" on symlinks. Every major FUSE filesystem (gocryptfs, rclone, sshfs, s3fs) implements Setattr on symlinks for this reason. Mode is always 0777 per POSIX convention, so chmod requests are silently accepted but not stored. * fix(fuse): return EIO instead of panicking on unknown node type Replace panic with log.Errorf + syscall.EIO in IPNS Directory.Lookup for unexpected MFS node types. Also remove duplicate comment block on File.Flush. * docs: update FUSE docs for go-fuse migration - fuse.md: replace stale OSXFUSE section with macFUSE, remove obsolete go-fuse-version tool, fix broken FreeBSD sudo echo, update xattr example to ipfs.cid with CIDv1, add mode/mtime section, add unixfs-v1-2025 tip, add debug logging section, add TOC, link to hanwen/go-fuse - changelog: refine bullet wording, link to fuse.md - config.md: fix double space, update fuse.md link text - experimental-features.md: fix double space, soften wording - README.md: add FUSE to features list and docs table * refactor(fuse): extract shared writable types and test suite Extract duplicated code from fuse/mfs and fuse/ipns into a shared fuse/writable package, and consolidate duplicated tests into a reusable suite in fuse/fusetest. - fuse/writable: Dir, FileInode, FileHandle, Symlink types with all FUSE interface methods, shared by both mounts - fuse/fusetest: RunWritableSuite with helpers, exercised by both mfs and ipns via mount-specific factories - fix cache invalidation race: NotifyContent in Flush (synchronous) in addition to Release (async), so stat after close sees new size - drop deprecated ipfs_cid xattr, log error guiding users to ipfs.cid - mfs_unix.go: 632 -> 19 lines (thin wrapper over writable.Dir) - ipns_unix.go: 795 -> 170 lines (Root + key resolution only) - mfs_test.go: 1183 -> 95 lines (factory + persistence test) - ipns_test.go: 1309 -> 162 lines (factory + IPNS-specific tests) - tests that were only in one mount now run on both * feat(fuse): add macOS-specific mount options Set volname, noapplexattr, and noappledouble on macOS via PlatformMountOpts, applied in NewMount so all three mounts benefit automatically. - volname: shows mount name in Finder instead of "macfuse Volume 0" - noapplexattr: suppresses Finder's com.apple.* xattr probes - noappledouble: prevents ._ resource fork sidecar files * fix(fuse): detect symlinks in readdir, fix stale refs Readdir on writable mounts now checks the underlying DAG node type for TFile entries, reporting S_IFLNK for symlinks instead of regular file. This makes ls -l and find -type l work correctly. - writable: Readdir checks SymlinkTarget for TFile entries - writablesuite: add SymlinkReaddir regression test - readonly: add TestReaddirSymlink regression test - test/cli/fuse: fix stale bazil.org/fuse reference in doc comment * fix(fuse): normalize deprecated ipfs_cid xattr to ipfs.cid Getxattr for the old "ipfs_cid" name now returns the CID instead of ENOATTR, keeping existing tooling working during the deprecation period. A log error is emitted on each access to nudge migration. * fix(fuse): serialize concurrent reads on readonly file handles The go-fuse server dispatches each FUSE request in its own goroutine. On files larger than 128 KB the kernel issues concurrent readahead Read requests on the same file handle, racing on the shared DagReader's Seek+CtxReadFull sequence and corrupting its internal state. Add sync.Mutex to roFileHandle (matching the existing pattern in writable.FileHandle) and lock in Read and Release. - fuse/readonly/readonly_unix.go: add mu sync.Mutex to roFileHandle - fuse/readonly/ipfs_test.go: add TestConcurrentLargeFileRead - fuse/fusetest/writablesuite.go: add LargeFileConcurrentRead to shared writable suite (exercised by both /mfs and /ipns tests) * fix(fuse): bypass MFS locking for read-only opens MFS uses an RWMutex (desclock) that holds RLock for the lifetime of a read descriptor and requires exclusive Lock for writes. Tools like rsync --inplace open the same file for reading and writing from separate processes, deadlocking on this mutex. For O_RDONLY opens, create a DagReader directly from the current DAG node instead of going through MFS. The reader gets a point-in-time snapshot and never touches desclock, so writers proceed independently. - fuse/writable/writable.go: add roFileHandle with DagReader for read-only opens, add DAG field to Config - fuse/mfs/mfs_unix.go: pass ipfs.DAG to writable Config - fuse/ipns/ipns_unix.go: pass ipfs.Dag() to writable Config - fuse/fusetest/writablesuite.go: add ConcurrentReadWrite test exercising simultaneous read and write on the same file * fix(fuse): support truncate(path, size) without open fd Open a temporary write descriptor in Setattr when the kernel sends a size change without a file handle (the truncate(2) syscall, as opposed to ftruncate(fd) which passes the handle). Previously this returned ENOTSUP. - fuse/writable: open, truncate, flush, close in Setattr else branch - fuse/fusetest: add TruncatePath to the shared writable suite - test/cli/fuse: add end-to-end truncation test covering ftruncate(fd), syscall.Truncate(path), and open(O_TRUNC) through a real daemon * ci(fuse): get stack traces on test hangs The fuse-tests job was being silently cancelled by GitHub at 10min because Go's per-test timeout (5m) was the same order as the job timeout, and GOTRACEBACK=single hid the hung goroutines anyway. - shrink TEST_FUSE_TIMEOUT to 4m so Go's panic fires first - shrink job timeout-minutes to 6 (normal run is ~3min) - set GOTRACEBACK=all so the panic dumps every goroutine, not just the timer * fix(fuse): fill attrs in FileInode.Setattr response Without this, the kernel could cache zero attrs after a chmod, touch, or ftruncate until AttrTimeout (1s) expired. Dir.Setattr and Symlink.Setattr already fill out.Attr; FileInode.Setattr now matches. * docs(config): clarify Mounts.IPNS writability scope Only directories backed by keys the node holds are writable. All other names resolve via IPNS to read-only symlinks into the /ipfs mount. * fuse: review cleanup for go-fuse migration Final pass on #11272 addressing review feedback. - writable: panic in NewDir if Config.DAG is nil. Both call sites already supply it, but a nil value silently fell back to the MFS path in FileInode.Open, re-introducing the rsync --inplace deadlock the read-only fast path was added to fix. - writable: document Dir.Rename non-atomicity. Source unlink happens before destination add, so any failure between the two loses the source. An atomic fix requires changes in boxo/mfs. - writable: add unit test locking in that Symlink.Setattr accepts a mode-only request without erroring and does not store the requested mode (POSIX symlinks have no meaningful permission bits). - docs/config: correct StoreMode default modes; the previous text listed 0666 for files, which the code never uses. * docs(config): list StoreMtime and StoreMode in Mounts TOC * fix(fuse): fill EntryOut attrs in Dir.Create and Dir.Mkdir Without this, fstat on the file handle returned by Create reports mode 0 and size 0 for up to AttrTimeout (1s), because the kernel caches the empty attrs from the Create response. Path-based stat goes through Lookup which already fills attrs, so the bug only shows up via fstat. Mirrors the same fix already applied to FileInode.Setattr. Dir.Mkdir gets the same fillAttr treatment for consistency, plus a TODO noting that boxo's mfs.Directory.Mkdir accepts no mode arg so the caller's mode is dropped on creation. Adds CreateAttrsImmediate and MkdirAttrsImmediate to the shared writable suite to guard both paths against future regressions. * fix(fuse): map context cancellation to EINTR in read paths When a userspace process is killed mid-read (Ctrl-C, SIGKILL on a stuck cat) the kernel sends FUSE_INTERRUPT and go-fuse cancels the per-request context. fs.ToErrno does not recognise context.Canceled and falls through to "function not implemented", which the kernel cannot act on. Map context.Canceled and DeadlineExceeded to EINTR so the syscall is correctly aborted. - mount/errno.go: new ReadErrno helper used by all context-aware read paths in both readonly and writable mounts - readonly: applied to Node.Open, Node.Readdir, roFileHandle.Read - writable: applied to FileInode.Open, FileHandle.Read, roFileHandle.Read - readonly/ipfs_test.go: TestReadCancellationUnblocks guards the contract via a blocking DagReader fake; without ReadErrno the test reports "function not implemented" instead of EINTR * test(fuse): add OExcl, DirRename, SparseWrite, FsyncCrossHandle Coverage gaps in the shared writable suite: - OExcl: lock files and atomic-create patterns rely on the second open with O_CREATE|O_EXCL failing with EEXIST - DirRename: previously only file rename and cross-dir file rename were tested; this exercises Rename on a directory inode - SparseWrite: WriteAt past the end of an empty file must report the correct size and return zeros for the gap - FsyncCrossHandle: a reader on a fresh fd must see data flushed by fsync on the writer fd, not just after close * test(fuse): cover external unmount on /ipns and /mfs Previously TestExternalUnmount only exercised /ipfs, leaving the goroutine that watches fuse.Server.Wait() untested for the other two mounts. Refactor into a table-driven test that runs the same fusermount/umount-then-IsActive flow against all three mounts. Switch to coremock.NewMockNode so the node is online: doMount only attaches the /ipns mount when node.IsOnline is true, and the table needs all three populated. * fix(commands): align 'ipfs mount' output columns MountCmd's LongDescription has "MFS mounted at:" with two spaces so the column lines up with the 4-char "IPFS" and "IPNS" rows above, but the runtime encoder and the daemon's startup print used a single space and produced misaligned output. Bring both runtime sites in line with the help text, and update the two existing test fixtures (test/cli/fuse and the sharness test-lib helper that t0040-add-and-cat.sh still uses) to expect the aligned form. * fix(fuse): invalidate kernel cache on Fsync FileHandle.Fsync only flushed the MFS file descriptor and left the kernel's cached attrs and content for the inode untouched. A fresh reader on the same path then saw the size cached from the original Create response (zero), reading zero bytes regardless of how much the writer had synced. Mirror the cache invalidation already done in Flush via inode.NotifyContent(0, 0) so a writer that fsyncs while another process opens the file (vim then a follow-up cat, IDE then a language server) sees consistent state. Sharpen the FsyncCrossHandle assertion to report the size delta on failure; the bug surfaced as got=0/want=500 only after switching from bytes.Equal to require.Equal. * chore(gitignore): ignore test_fuse_unit and test_fuse_cli json output The new test_fuse_unit and test_fuse_cli make targets emit test/fuse/fuse-unit-tests.json and test/fuse/fuse-cli-tests.json respectively, the same gotestsum --jsonfile pattern that test_unit and test_cli already use. Add them to the same .gitignore section so a local test run does not leave the working tree dirty. * test(fuse): end-to-end coverage with real POSIX tools Adds TestFUSERealWorld in test/cli/fuse/realworld_test.go: a single shared-daemon test with 18 subtests that exercise the writable /mfs mount through the actual binaries users invoke (sh, cat, seq, wc, ls, stat, cp, mv, rm, ln, readlink, find, dd, sha256sum, tar, rsync, vim). Each subtest verifies the result both via the FUSE filesystem and via 'ipfs files read|stat|ls' so both views agree. Synthetic payloads default to 1 MiB + 1 byte so multi-chunk read/write paths are exercised, not just single-chunk fast paths. External tools are required, not optional: a missing binary fails the test loudly so a CI image change cannot silently turn the suite green. The whole-suite TEST_FUSE gate is the only place a developer is allowed to skip. runCmd forces LC_ALL=C so locale-sensitive tool output (date formats in 'ls -l', decimal separators in 'wc', localized error messages, find/ls collation) is deterministic regardless of the runner's locale settings. One shared daemon across all 18 subtests keeps total runtime under two seconds; isolation comes from per-subtest subdirectories under the mount. | 4 个月前 | |
fix(fuse): switch to hanwen/go-fuse (#11272) * test(fuse): consolidate FUSE tests into test/cli/fuse Move FUSE integration tests from sharness shell scripts (t0030, t0031, t0032) and test/cli/fuse_test.go into a dedicated test/cli/fuse/ Go sub-package, ensuring all FUSE test cases run in CI. - git mv test/cli/fuse_test.go to test/cli/fuse/ (package fuse) - convert all sharness FUSE tests to Go subtests under TestFUSE: mount failure, IPNS symlink, IPNS NS map resolution, MFS file/dir creation, xattr (Linux), files write, add --to-files, file removal, nested dirs, publish-while-mounted block, sharded directory reads - add xattr helpers with build tags (linux/other) using unix.Getxattr - split make test_fuse into test_fuse_unit (./fuse/...) and test_fuse_cli (./test/cli/fuse/...) sub-targets - set TEST_FUSE=0 in test_cli so FUSE tests skip in cli-tests CI job - increase fuse-tests CI timeout from 5m to 10m for CLI tests - delete sharness t0030, t0031, t0032 (were always skipped in CI) * docs: document FUSE test split between unit and e2e Add cross-reference comments between the unit tests in fuse/readonly/, fuse/ipns/, fuse/mfs/ and the end-to-end CLI tests in test/cli/fuse/. Also fix AGENTS.md to use a temp dir for fusermount symlink instead of sudo. * ci: prevent stale FUSE mounts from failing fuse-tests On shared self-hosted runners, leftover mount points from previous runs can exhaust the kernel FUSE mount limit. - add job-level concurrency group so only one fuse-tests runs at a time - lazy-unmount stale /tmp/fusetest* mounts before running tests * ci: only symlink fusermount3 when fusermount is missing * fix(fuse): remove goroutine leak in IPNS Flush handler The Flush handler wrapped fi.fi.Flush() in a goroutine so it could return early when the FUSE context was canceled. But the goroutine kept running in the background, and when Release arrived it called Close on the same file descriptor concurrently. The two paths both entered DagModifier.Sync, racing on its internal write buffer and causing a nil pointer panic. The fix is to call Flush directly without a goroutine. The MFS flush cannot be safely canceled mid-operation anyway, so the goroutine only added the illusion of cancellation while leaking work and masking the real error. Also bumps boxo to pick up the matching defense-in-depth fix that serializes FileDescriptor.Flush and Close with a mutex. * fix(fuse): add mutex to IPNS file handle operations bazil/fuse dispatches each FUSE request in its own goroutine. The IPNS File handle had no synchronization, so concurrent Read/Write/Flush/Release calls could overlap on the underlying DagModifier which is not safe for concurrent use. Add sync.Mutex to File, matching the pattern already used by the MFS FileHandler. * refactor(fuse): remove dead File.Forget method bazil/fuse only dispatches Forget to nodes via the NodeForgetter interface. File is a handle, not a node, so this method was never called. The /mfs mount has no equivalent. * fix(fuse): flush IPNS directory after Remove and Rename The /mfs mount flushes the directory after Unlink and Rename so changes propagate to the MFS root immediately. The /ipns mount did not, leaving mutations pending until an unrelated flush. Also add an empty-directory check before removing directories, matching the /mfs mount's safety check. * fix(fuse): inherit CID builder and flush on IPNS Create New files created via the /ipns FUSE mount now inherit the CID builder from their parent directory, preventing CIDv0 nodes from appearing inside a CIDv1 tree. The directory is also flushed after AddChild so the new entry propagates to the MFS root immediately, matching the /mfs mount. * test(fuse): add IPNS Remove and non-empty rmdir tests Cover the file removal path and the empty-directory safety check added in the previous commit. TestRemoveFile verifies a created file can be removed and is gone afterwards. TestRemoveNonEmptyDirectory verifies that rmdir on a directory with children fails, and succeeds once the children are removed first. * feat(fuse): read UnixFS mode/mtime, add StoreMtime/StoreMode config All three FUSE mounts now read mode and mtime from UnixFS metadata when present, falling back to POSIX defaults when absent. Most IPFS data does not include this optional metadata. Writing mode and mtime is opt-in via two new config flags: - Mounts.StoreMtime: persist mtime on file create and open-for-write - Mounts.StoreMode: persist mode on chmod Other changes in this commit: - align default file/dir modes across /ipns and /mfs to 0644/0755 - share mode constants via fuse/mount/mode.go - convert Mounts.FuseAllowOther from bool to Flag for consistency - add Setattr to /ipns FileNode and /mfs File for chmod and touch - move dead File.Setattr from IPNS handle to FileNode (node) - bump boxo for Directory.Mode() and Directory.ModTime() getters * feat(fuse): add ipfs.cid xattr to all mounts All three FUSE mounts now expose the node's CID via the ipfs.cid extended attribute on both files and directories. The /mfs mount also accepts the old ipfs_cid name for backward compatibility. The /ipfs mount previously had a stub that returned nil for all xattrs; it now returns the correct CID. The xattr name follows the convention used by CephFS (ceph.*), Btrfs (btrfs.*), and GlusterFS (glusterfs.*). * feat(fuse): switch from bazil.org/fuse to hanwen/go-fuse v2 Replace the unmaintained bazil.org/fuse (last commit 2020) with hanwen/go-fuse v2.9.0, fixing two architectural issues that could not be solved with the old library. ftruncate now works: hanwen/go-fuse passes the open file handle to NodeSetattrer, so Setattr can truncate through the existing write descriptor instead of trying to open a second one (which deadlocks on MFS's single-writer lock). fsync now works: FileFsyncer runs on the handle directly, flushing the write buffer through the open descriptor. Previously a no-op because bazil dispatched Fsync to the inode only. mount package: - NewMount takes (InodeEmbedder, mountpoint, *fs.Options) instead of (fs.FS, mountpoint, allowOther) - mount/unmount collapses to a single fs.Mount call - fusermount3 tried before fusermount in ForceUnmount all three mounts: - structs embed fs.Inode (hanwen's InodeEmbedder pattern) - Remove split into Unlink + Rmdir (separate FUSE interfaces) - ReadDirAll replaced with Readdir returning DirStream - fillAttr helper shared between Getattr and Lookup responses - kernel cache invalidation via NotifyContent after Flush - 1s entry/attr timeout for writable mounts (matches go-fuse default, gocryptfs, rclone) - O_APPEND tracked on file handle, writes seek to end - build tags standardized to (linux || darwin || freebsd) && !nofuse tests: - replaced bazil fstestutil.MountedT with shared fusetest.TestMount - fixed TestConcurrentRW: channel drain mismatch and missing sync between write Close and read start - added TestFsync, TestFtruncate, TestReadlink, TestSeekRead, TestLargeFile, TestRmdir, TestCrossDirRename, TestUnknownXattr - added StoreMtime disabled/enabled subtests * fix(fuse): close fd on error in Open to prevent leak MFS enforces a single-writer lock, so a leaked write descriptor blocks all subsequent opens of that file until GC. * fix(fuse): detect external unmount via server.Wait Without this, IsActive stays true after `fusermount -u` and Unmount returns nil instead of ErrNotMounted. * fix(fuse): return actual error from Unlink/Rmdir, not ENOENT After confirming the child exists, an Unlink failure could be an IO error. Returning ENOENT would hide the real cause. * fix(fuse): reuse DagReader per open, pass ctx to all reads Readonly Open now returns a file handle holding a DagReader instead of recreating one per Read call. Sequential reads no longer re-traverse the DAG from the root on each kernel request. All three mounts now use CtxReadFull with the kernel's per-request context so killing a process mid-read cancels in-flight block fetches instead of letting them complete uselessly. * chore(fuse): cleanup dead code, add var comments - remove dead `_ = mntDir` in TestXattrCID - comment why immutableAttrCacheTime and mutableCacheTime are var - add TODO for using IPNS record TTL as cache timeout * chore(fuse): replace OSXFUSE 2.x check with macFUSE detection The old check tried to verify OSXFUSE >= 2.7.2 to avoid a kernel panic from 2015. It used sysctl, tried to `go install` a third-party tool at runtime, and referenced paths that no longer exist. Replace with a simple check for the macFUSE mount helper, matching the same paths go-fuse looks for. If neither macFUSE nor OSXFUSE is found, point the user to the install page. Also standardize build tags to (linux || darwin || freebsd) && !nofuse and use strings.ReplaceAll. * fix(fuse): include mountpoint path in mount errors go-fuse's fusermount errors don't include the path, so tools that check error messages for the mountpoint name couldn't tell which mount failed. * chore(ci): remove bazil fusermount workaround go-fuse finds fusermount3 natively, no symlink needed. The stale mount cleanup was for bazil's fstestutil which we no longer use. * docs: update v0.41 changelog for FUSE rewrite * chore(deps): bump boxo for full FileDescriptor serialization boxo@64be0815 extends the mutex from Flush/Close to all FileDescriptor operations (Read, Write, Seek, Truncate, Size), preventing data races on the underlying DagModifier. * chore(deps): bump boxo to merged ipfs/boxo#1133 Picks up full FileDescriptor serialization: the mutex now covers all operations (Read, Write, Seek, Truncate, Size), not just Flush and Close. * feat(fuse): CAP_ATOMIC_O_TRUNC, new integration tests Advertise CAP_ATOMIC_O_TRUNC so the kernel sends O_TRUNC inside Open instead of doing a separate SETATTR(size=0) first. Without this, the kernel's SETATTR needs to open a write descriptor inside Setattr, which deadlocks on MFS's single-writer lock. Move kernel cache invalidation from Flush to Release because mfsFD.Close (in Release) is where the final DAG node is committed. Upgrade go-fuse to latest for ExtraCapabilities support. New tests for both MFS and IPNS: - TestOpenTrunc, TestSeekAndWrite, TestOverwriteExisting - TestTempFileRename, TestVimSavePattern, TestRsyncPattern (skipped pending rename-over-existing and cache fixes) * fix(fuse): rename-over-existing, bump boxo for flushUp race fix IPNS Rename now unlinks the target before AddChild, matching MFS. Without this, renaming onto an existing name returned "directory already has entry". Bump boxo to pick up the flushUp unlinked-entry fix (ipfs/boxo@8ae46d5): when a file descriptor outlives its directory entry (FUSE RELEASE racing with RENAME), flushUp no longer re-adds the stale name. Unskip TestTempFileRename and TestRsyncPattern on both mounts. * fix(fuse): unskip VimSavePattern, bump boxo for setNodeData fix boxo@552d8e7 fixes File.setNodeData dropping content links when updating metadata (mode, mtime). chmod or touch after write no longer makes the file appear empty. Unskip TestVimSavePattern on both mounts. Remove debug logging and temporary test functions added during investigation. * fix(fuse): build tags for cross-compilation go-fuse does not compile on windows/openbsd/netbsd/plan9. Move WritableMountCapabilities (which imports go-fuse) from mode.go (no build tag) to caps.go (platform-gated). Align build tags on fusetest and core/commands/mount stubs so unsupported platforms don't pull in go-fuse transitively. * fix(test): use fusermount3 in CLI FUSE tests The doUnmount helper hardcoded fusermount, but systems with only fuse3 installed have fusermount3. Try fusermount3 first, matching what go-fuse and our ForceUnmount already do. * feat(fuse): symlink support on writable mounts Add NodeSymlinker to MFS and IPNS directories. Symlinks are stored as UnixFS TSymlink nodes in the DAG, the same format used by `ipfs add` for directories containing symlinks. The readonly /ipfs mount already rendered existing symlinks; now /mfs and /ipns can create them too. The target string is cached at Lookup time to avoid re-parsing the DAG node on every Readlink call. Symlink permissions are always 0777 per POSIX convention (access control uses the target's mode). * fix(fuse): checked type assertion in MFS Rename The direct type assertion on newParent could panic if the kernel passed a non-directory inode. Use a checked assertion with EINVAL fallback, matching the type-switch pattern in the IPNS mount. * fix(test): add missing continue in stress test Missing continue after error sends let execution fall through to nil type assertions (read.(files.File)) that would panic on error. Also cancel the context before continuing to avoid leaking it. * fix(fuse): return error from Readdir when DAG.Get fails Abort the directory listing instead of silently omitting the unretrievable entry. Callers get EIO, which is more honest than a partial listing that hides missing blocks. * docs: remove duplicate fsync bullet in changelog * ci: clean up stale FUSE mounts in fuse-tests job On shared self-hosted runners, leftover mounts from crashed runs can exhaust the kernel mount_max limit. Lazy-unmount kubo-test and harness temp mounts before and after tests. * chore(deps): bump boxo to merged ipfs/boxo#1134 Picks up flushUp unlinked-entry guard and setNodeData content link preservation. * docs: add build tag comments, normalize tag style Add a one-line comment above every //go:build directive explaining why the constraint exists. Normalize tag style: positive platform constraints first, then feature flags/negations. Simplify redundant expressions. * fix(fuse): add Setattr to directories for chmod and mtime Tools like tar and rsync call utimensat on directories after extraction. Without Setattr on Dir, this returned ENOTSUP. Add Setattr to Dir (MFS) and Directory (IPNS) that handles mode and mtime the same way as the file-level Setattr. When StoreMtime or StoreMode is disabled the call succeeds silently, matching the file-level behavior. * docs: clarify directory support and spec link for StoreMtime/StoreMode - mention that touch and chmod work on both files and directories - note tar and rsync as practical use cases - link to UnixFS spec for optional metadata storage * fix(fuse): use proper mode conversion, document 9-bit limit Use files.UnixPermsToModePerms and files.ModePermsToUnixPerms for converting between FUSE kernel mode (unix 12-bit layout) and Go's os.FileMode (different bit positions for setuid/setgid/sticky). The UnixFS spec supports all 12 permission bits, but boxo's MFS layer (File.Mode, Directory.Mode) exposes only the lower 9. FUSE mounts are always nosuid so the upper 3 bits would have no effect. Add TestSetuidBitsStripped to both mounts confirming the behavior. * feat(fuse): symlink Setattr with mtime persistence Wire the backing mfs.File into the FUSE Symlink struct so Setattr can call SetModTime when StoreMtime is enabled. boxo's File methods (SetModTime, ModTime) already work on TSymlink nodes since they operate on the FSNode protobuf without checking the type. Without Setattr, rsync -a fails with "failed to set times" on symlinks. Every major FUSE filesystem (gocryptfs, rclone, sshfs, s3fs) implements Setattr on symlinks for this reason. Mode is always 0777 per POSIX convention, so chmod requests are silently accepted but not stored. * fix(fuse): return EIO instead of panicking on unknown node type Replace panic with log.Errorf + syscall.EIO in IPNS Directory.Lookup for unexpected MFS node types. Also remove duplicate comment block on File.Flush. * docs: update FUSE docs for go-fuse migration - fuse.md: replace stale OSXFUSE section with macFUSE, remove obsolete go-fuse-version tool, fix broken FreeBSD sudo echo, update xattr example to ipfs.cid with CIDv1, add mode/mtime section, add unixfs-v1-2025 tip, add debug logging section, add TOC, link to hanwen/go-fuse - changelog: refine bullet wording, link to fuse.md - config.md: fix double space, update fuse.md link text - experimental-features.md: fix double space, soften wording - README.md: add FUSE to features list and docs table * refactor(fuse): extract shared writable types and test suite Extract duplicated code from fuse/mfs and fuse/ipns into a shared fuse/writable package, and consolidate duplicated tests into a reusable suite in fuse/fusetest. - fuse/writable: Dir, FileInode, FileHandle, Symlink types with all FUSE interface methods, shared by both mounts - fuse/fusetest: RunWritableSuite with helpers, exercised by both mfs and ipns via mount-specific factories - fix cache invalidation race: NotifyContent in Flush (synchronous) in addition to Release (async), so stat after close sees new size - drop deprecated ipfs_cid xattr, log error guiding users to ipfs.cid - mfs_unix.go: 632 -> 19 lines (thin wrapper over writable.Dir) - ipns_unix.go: 795 -> 170 lines (Root + key resolution only) - mfs_test.go: 1183 -> 95 lines (factory + persistence test) - ipns_test.go: 1309 -> 162 lines (factory + IPNS-specific tests) - tests that were only in one mount now run on both * feat(fuse): add macOS-specific mount options Set volname, noapplexattr, and noappledouble on macOS via PlatformMountOpts, applied in NewMount so all three mounts benefit automatically. - volname: shows mount name in Finder instead of "macfuse Volume 0" - noapplexattr: suppresses Finder's com.apple.* xattr probes - noappledouble: prevents ._ resource fork sidecar files * fix(fuse): detect symlinks in readdir, fix stale refs Readdir on writable mounts now checks the underlying DAG node type for TFile entries, reporting S_IFLNK for symlinks instead of regular file. This makes ls -l and find -type l work correctly. - writable: Readdir checks SymlinkTarget for TFile entries - writablesuite: add SymlinkReaddir regression test - readonly: add TestReaddirSymlink regression test - test/cli/fuse: fix stale bazil.org/fuse reference in doc comment * fix(fuse): normalize deprecated ipfs_cid xattr to ipfs.cid Getxattr for the old "ipfs_cid" name now returns the CID instead of ENOATTR, keeping existing tooling working during the deprecation period. A log error is emitted on each access to nudge migration. * fix(fuse): serialize concurrent reads on readonly file handles The go-fuse server dispatches each FUSE request in its own goroutine. On files larger than 128 KB the kernel issues concurrent readahead Read requests on the same file handle, racing on the shared DagReader's Seek+CtxReadFull sequence and corrupting its internal state. Add sync.Mutex to roFileHandle (matching the existing pattern in writable.FileHandle) and lock in Read and Release. - fuse/readonly/readonly_unix.go: add mu sync.Mutex to roFileHandle - fuse/readonly/ipfs_test.go: add TestConcurrentLargeFileRead - fuse/fusetest/writablesuite.go: add LargeFileConcurrentRead to shared writable suite (exercised by both /mfs and /ipns tests) * fix(fuse): bypass MFS locking for read-only opens MFS uses an RWMutex (desclock) that holds RLock for the lifetime of a read descriptor and requires exclusive Lock for writes. Tools like rsync --inplace open the same file for reading and writing from separate processes, deadlocking on this mutex. For O_RDONLY opens, create a DagReader directly from the current DAG node instead of going through MFS. The reader gets a point-in-time snapshot and never touches desclock, so writers proceed independently. - fuse/writable/writable.go: add roFileHandle with DagReader for read-only opens, add DAG field to Config - fuse/mfs/mfs_unix.go: pass ipfs.DAG to writable Config - fuse/ipns/ipns_unix.go: pass ipfs.Dag() to writable Config - fuse/fusetest/writablesuite.go: add ConcurrentReadWrite test exercising simultaneous read and write on the same file * fix(fuse): support truncate(path, size) without open fd Open a temporary write descriptor in Setattr when the kernel sends a size change without a file handle (the truncate(2) syscall, as opposed to ftruncate(fd) which passes the handle). Previously this returned ENOTSUP. - fuse/writable: open, truncate, flush, close in Setattr else branch - fuse/fusetest: add TruncatePath to the shared writable suite - test/cli/fuse: add end-to-end truncation test covering ftruncate(fd), syscall.Truncate(path), and open(O_TRUNC) through a real daemon * ci(fuse): get stack traces on test hangs The fuse-tests job was being silently cancelled by GitHub at 10min because Go's per-test timeout (5m) was the same order as the job timeout, and GOTRACEBACK=single hid the hung goroutines anyway. - shrink TEST_FUSE_TIMEOUT to 4m so Go's panic fires first - shrink job timeout-minutes to 6 (normal run is ~3min) - set GOTRACEBACK=all so the panic dumps every goroutine, not just the timer * fix(fuse): fill attrs in FileInode.Setattr response Without this, the kernel could cache zero attrs after a chmod, touch, or ftruncate until AttrTimeout (1s) expired. Dir.Setattr and Symlink.Setattr already fill out.Attr; FileInode.Setattr now matches. * docs(config): clarify Mounts.IPNS writability scope Only directories backed by keys the node holds are writable. All other names resolve via IPNS to read-only symlinks into the /ipfs mount. * fuse: review cleanup for go-fuse migration Final pass on #11272 addressing review feedback. - writable: panic in NewDir if Config.DAG is nil. Both call sites already supply it, but a nil value silently fell back to the MFS path in FileInode.Open, re-introducing the rsync --inplace deadlock the read-only fast path was added to fix. - writable: document Dir.Rename non-atomicity. Source unlink happens before destination add, so any failure between the two loses the source. An atomic fix requires changes in boxo/mfs. - writable: add unit test locking in that Symlink.Setattr accepts a mode-only request without erroring and does not store the requested mode (POSIX symlinks have no meaningful permission bits). - docs/config: correct StoreMode default modes; the previous text listed 0666 for files, which the code never uses. * docs(config): list StoreMtime and StoreMode in Mounts TOC * fix(fuse): fill EntryOut attrs in Dir.Create and Dir.Mkdir Without this, fstat on the file handle returned by Create reports mode 0 and size 0 for up to AttrTimeout (1s), because the kernel caches the empty attrs from the Create response. Path-based stat goes through Lookup which already fills attrs, so the bug only shows up via fstat. Mirrors the same fix already applied to FileInode.Setattr. Dir.Mkdir gets the same fillAttr treatment for consistency, plus a TODO noting that boxo's mfs.Directory.Mkdir accepts no mode arg so the caller's mode is dropped on creation. Adds CreateAttrsImmediate and MkdirAttrsImmediate to the shared writable suite to guard both paths against future regressions. * fix(fuse): map context cancellation to EINTR in read paths When a userspace process is killed mid-read (Ctrl-C, SIGKILL on a stuck cat) the kernel sends FUSE_INTERRUPT and go-fuse cancels the per-request context. fs.ToErrno does not recognise context.Canceled and falls through to "function not implemented", which the kernel cannot act on. Map context.Canceled and DeadlineExceeded to EINTR so the syscall is correctly aborted. - mount/errno.go: new ReadErrno helper used by all context-aware read paths in both readonly and writable mounts - readonly: applied to Node.Open, Node.Readdir, roFileHandle.Read - writable: applied to FileInode.Open, FileHandle.Read, roFileHandle.Read - readonly/ipfs_test.go: TestReadCancellationUnblocks guards the contract via a blocking DagReader fake; without ReadErrno the test reports "function not implemented" instead of EINTR * test(fuse): add OExcl, DirRename, SparseWrite, FsyncCrossHandle Coverage gaps in the shared writable suite: - OExcl: lock files and atomic-create patterns rely on the second open with O_CREATE|O_EXCL failing with EEXIST - DirRename: previously only file rename and cross-dir file rename were tested; this exercises Rename on a directory inode - SparseWrite: WriteAt past the end of an empty file must report the correct size and return zeros for the gap - FsyncCrossHandle: a reader on a fresh fd must see data flushed by fsync on the writer fd, not just after close * test(fuse): cover external unmount on /ipns and /mfs Previously TestExternalUnmount only exercised /ipfs, leaving the goroutine that watches fuse.Server.Wait() untested for the other two mounts. Refactor into a table-driven test that runs the same fusermount/umount-then-IsActive flow against all three mounts. Switch to coremock.NewMockNode so the node is online: doMount only attaches the /ipns mount when node.IsOnline is true, and the table needs all three populated. * fix(commands): align 'ipfs mount' output columns MountCmd's LongDescription has "MFS mounted at:" with two spaces so the column lines up with the 4-char "IPFS" and "IPNS" rows above, but the runtime encoder and the daemon's startup print used a single space and produced misaligned output. Bring both runtime sites in line with the help text, and update the two existing test fixtures (test/cli/fuse and the sharness test-lib helper that t0040-add-and-cat.sh still uses) to expect the aligned form. * fix(fuse): invalidate kernel cache on Fsync FileHandle.Fsync only flushed the MFS file descriptor and left the kernel's cached attrs and content for the inode untouched. A fresh reader on the same path then saw the size cached from the original Create response (zero), reading zero bytes regardless of how much the writer had synced. Mirror the cache invalidation already done in Flush via inode.NotifyContent(0, 0) so a writer that fsyncs while another process opens the file (vim then a follow-up cat, IDE then a language server) sees consistent state. Sharpen the FsyncCrossHandle assertion to report the size delta on failure; the bug surfaced as got=0/want=500 only after switching from bytes.Equal to require.Equal. * chore(gitignore): ignore test_fuse_unit and test_fuse_cli json output The new test_fuse_unit and test_fuse_cli make targets emit test/fuse/fuse-unit-tests.json and test/fuse/fuse-cli-tests.json respectively, the same gotestsum --jsonfile pattern that test_unit and test_cli already use. Add them to the same .gitignore section so a local test run does not leave the working tree dirty. * test(fuse): end-to-end coverage with real POSIX tools Adds TestFUSERealWorld in test/cli/fuse/realworld_test.go: a single shared-daemon test with 18 subtests that exercise the writable /mfs mount through the actual binaries users invoke (sh, cat, seq, wc, ls, stat, cp, mv, rm, ln, readlink, find, dd, sha256sum, tar, rsync, vim). Each subtest verifies the result both via the FUSE filesystem and via 'ipfs files read|stat|ls' so both views agree. Synthetic payloads default to 1 MiB + 1 byte so multi-chunk read/write paths are exercised, not just single-chunk fast paths. External tools are required, not optional: a missing binary fails the test loudly so a CI image change cannot silently turn the suite green. The whole-suite TEST_FUSE gate is the only place a developer is allowed to skip. runCmd forces LC_ALL=C so locale-sensitive tool output (date formats in 'ls -l', decimal separators in 'wc', localized error messages, find/ls collation) is deterministic regardless of the runner's locale settings. One shared daemon across all 18 subtests keeps total runtime under two seconds; isolation comes from per-subtest subdirectories under the mount. | 4 个月前 | |
cmdkit -> cmds License: MIT Signed-off-by: Steven Allen <steven@stebalien.com> | 7 年前 | |
ci: add stylecheck to golangci-lint (#9334) | 3 年前 | |
feat(p2p): add --foreground flag to listen and forward commands (#11099) * feat(p2p): add --foreground flag to listen and forward commands adds `-f/--foreground` option that keeps the command running until interrupted (SIGTERM/Ctrl+C) or closed via `ipfs p2p close`. the listener/forwarder is automatically removed when the command exits. useful for systemd services and scripts that need cleanup on exit. * docs: add p2p-tunnels.md with systemd examples - add dedicated docs/p2p-tunnels.md covering: - why p2p tunnels (NAT traversal, no public IP needed) - quick start with netcat - background and foreground modes - systemd integration with path-based activation - security considerations and troubleshooting - document Experimental.Libp2pStreamMounting in docs/config.md - simplify docs/experimental-features.md, link to new doc - add "Learn more" links to ipfs p2p listen/forward --help - update changelog entry with doc link - add cross-reference in misc/README.md * chore: reference kubo#5460 for p2p config Ref. https://github.com/ipfs/kubo/issues/5460 * fix(daemon): write api/gateway files only after HTTP server is ready fixes race condition where $IPFS_PATH/api and $IPFS_PATH/gateway files were written before the HTTP servers were ready to accept connections. this caused issues for tools like systemd path units that immediately try to connect when these files appear. changes: - add corehttp.ServeWithReady() that signals when server is ready - wait for ready signal before writing address files - use sync.WaitGroup.Go() (Go 1.25) for cleaner goroutine management - add TestAddressFileReady to verify both api and gateway files * fix(daemon): buffer errc channel and wait for all listeners - buffer error channel with len(listeners) to prevent deadlock when multiple servers write errors simultaneously - wait for ALL listeners to be ready before writing api/gateway file, not just the first one Feedback-from: https://github.com/ipfs/kubo/pull/11099#pullrequestreview-3593885839 * docs(changelog): improve p2p tunnel section clarity reframe to lead with user benefit and add example output * docs(p2p): remove obsolete race condition caveat the "First launch fails but restarts work" troubleshooting section described a race where the api file was written before the daemon was ready. this was fixed in 80b703a which ensures api/gateway files are only written after HTTP servers are ready to accept connections. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 7 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
feat(rpc): Content-Type headers and IPNS record get/put (#11067) * fix http header when compress enabled for get command Closes #2376 * fix(rpc): set Content-Type for ipfs get based on output format - set application/x-tar when outputting tar (default and --archive) - set application/gzip when compression is enabled (--compress) - update go-ipfs-cmds with Tar encoding type and RFC 6713 compliant MIME types (application/gzip instead of application/x-gzip) * test(rpc): add Content-Type header tests for ipfs get * feat(rpc): add Content-Type headers for binary responses set proper Content-Type headers for RPC endpoints that return binary data: - `dag export`: application/vnd.ipld.car - `block get`: application/vnd.ipld.raw - `diag profile`: application/zip - `get`: application/x-tar or application/gzip (already worked, migrated to new API) uses the new OctetStream encoding type and SetContentType() method from go-ipfs-cmds to specify custom MIME types for binary responses. refs: https://github.com/ipfs/kubo/issues/2376 * feat(rpc): add `ipfs name get` command for IPNS record retrieval add dedicated command to retrieve raw signed IPNS records from the routing system. returns protobuf-encoded IPNS record with Content-Type `application/vnd.ipfs.ipns-record`. this provides a more convenient alternative to `ipfs routing get /ipns/<name>` which returns JSON with base64-encoded data. the raw output can be piped directly to `ipfs name inspect`: ipfs name get <name> | ipfs name inspect spec: https://specs.ipfs.tech/ipns/ipns-record/ * feat(rpc): add `ipfs name put` command for IPNS record storage adds `ipfs name put` to complement `ipfs name get`, allowing users to store IPNS records obtained from external sources without needing the private key. useful for backup, restore, and debugging workflows. the command validates records by default (signature, sequence number). use `--force` to bypass validation for testing how routing handles malformed or outdated records. also reorganizes test/cli files: - rename http_rpc_* -> rpc_* to match existing convention - merge name_get_put_test.go into name_test.go - add file header comments documenting test purposes * chore(deps): update go-ipfs-cmds to latest master includes SetContentType() for dynamic Content-Type headers --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 7 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
feat(pubsub): persistent validation and diagnostic commands (#11110) * feat(pubsub): persistent seqno validation and diagnostic commands - upgrade go-libp2p-pubsub to v0.15.0 - add persistent seqno validator using BasicSeqnoValidator stores max seen seqno per peer at /pubsub/seqno/<peerid> survives daemon restarts, addresses message cycling in large networks (#9665) - add `ipfs pubsub reset` command to clear validator state - add `ipfs diag datastore get/count` commands for datastore inspection requires daemon to be stopped, useful for debugging - change pubsub status from Deprecated to Experimental - add CLI tests for pubsub and diag datastore commands - remove flaky pubsub_msg_seen_cache_test.go (replaced by CLI tests) * fix(pubsub): improve reset command and add deprecation warnings - use batched delete for efficient bulk reset - check key existence before reporting deleted count - sync datastore after deletions to ensure persistence - show "no validator state found" when resetting non-existent peer - log deprecation warnings when using --enable-pubsub-experiment or --enable-namesys-pubsub CLI flags * refactor(test): add datastore helpers to test harness --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 7 个月前 | |
fix(cli/rpc): --cid-base works in all commands (#11239) * fix: --cid-base works in all commands and auto-upgrades CIDv0 Passing --cid-base=base32 now returns CIDv1 in base32 everywhere, including block, dag stat, and object patch which previously ignored it. - cidbase: auto-upgrade CIDv0 when base is not base58btc, deprecate --upgrade-cidv0-in-output, remove GetLowLevelCidEncoder - block stat/put/rm: use GetCidEncoder - dag stat: store CID as pre-encoded string, drop MarshalJSON/UnmarshalJSON - object patch rm-link/add-link: use GetCidEncoder - bitswap: switch to GetCidEncoder * test: add harness tests for --cid-base flag Remove unused DagStat.String() which truncated CIDs. Add CLI tests for --cid-base across block, dag stat, and object patch commands, including the --format=v0 interaction. * fix: respect --cid-base in refs local, object diff, pin remote, files chroot Use GetCidEncoder in commands that were still outputting CIDs via raw .String() calls. - refs local: encode blockstore keys with the requested base - object diff: encode Before/After CIDs in text encoder - pin remote add/ls: pass encoder through toRemotePinOutput - files chroot: encode old/new root CIDs in status message - tests: use base16 to avoid false positives if base32 becomes default * docs: update changelog entry for --cid-base fixes * test: cover --cid-base for add, pin ls, dag import Add harness tests for add, add -Q, pin ls, and dag import. Fix object patch tests broken by upstream UnixFS validation. Use base16 in all tests to avoid false positives. * docs: add metrics and CARv2 highlights to v0.41 changelog | 4 个月前 | |
feat(cli): add --human and --sort-size to ipfs ls (#11408) * feat(cli): add --human and --sort-size to ipfs ls - --human (-H): SI human-readable sizes in text output (humanize.Bytes) - --sort-size (-S): sort directory entries by size, largest first - Validation: --sort-size + --stream and --sort-size + --size=false errors - Unit tests for formatSize and sort helpers - CLI integration tests for both flags - Changelog highlight in v0.44 * test(ls): make sort tests fail when sorting breaks The tests covering --sort-size could not detect a broken feature. The unit tests copied the comparator into the test body and sorted with their own copy, so they passed regardless of what ls.go did. The CLI tests named each fixture after its size, which left alphabetical order and size order in agreement, so every ordering subtest passed even with --sort-size disabled outright. - extract lsLinkByName and lsLinkBySize so the tests exercise shipped code - point the unit tests at those two functions - name fixtures so their alphabetical order disagrees with their sizes - pin the real directory behaviour: UnixFS directories carry no Filesize, so they sort as 0, tie with empty files and break by name rather than landing strictly last * fix(ls): return 400 not 500 for bad flag combos Passing --sort-size together with --stream or --size=false made /api/v0/ls answer 500 Internal Server Error, telling API clients the server had broken when the caller had simply combined flags that cannot work together. Clients that retry on 5xx would retry a request that can never succeed. cmds.ErrClient maps to 400, matching how the rest of core/commands reports caller mistakes. CLI output is unchanged. * docs: fix --human size examples and JSON claims The help for --human advertised "1K 234M 2G", which no kubo command has ever printed. All three use humanize.Bytes, so the real output is SI with a space: 1.2 kB, 234 MB, 2.0 GB. The stale example was copied into 'ipfs ls' from the two commands that already carried it, so correct all three together. - ls, repo stat, bitswap stat: examples now match real output - drop the claim that --enc=json reports bytes. On the CLI, 'ipfs ls' has a PostRun that prints the text table whatever --enc says, so no JSON is produced there at all. Only the /api/v0/ls response is JSON, and that part is true - directories have no UnixFS Filesize, so they sort as 0 and tie with empty files. They do not land strictly last, so stop saying they do - changelog: a #### highlight with a TOC entry, kept short and pointing at 'ipfs ls --help' for the details - format sizes with strconv.FormatUint rather than fmt.Sprintf("%d") --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 个月前 | |
fix(fuse): switch to hanwen/go-fuse (#11272) * test(fuse): consolidate FUSE tests into test/cli/fuse Move FUSE integration tests from sharness shell scripts (t0030, t0031, t0032) and test/cli/fuse_test.go into a dedicated test/cli/fuse/ Go sub-package, ensuring all FUSE test cases run in CI. - git mv test/cli/fuse_test.go to test/cli/fuse/ (package fuse) - convert all sharness FUSE tests to Go subtests under TestFUSE: mount failure, IPNS symlink, IPNS NS map resolution, MFS file/dir creation, xattr (Linux), files write, add --to-files, file removal, nested dirs, publish-while-mounted block, sharded directory reads - add xattr helpers with build tags (linux/other) using unix.Getxattr - split make test_fuse into test_fuse_unit (./fuse/...) and test_fuse_cli (./test/cli/fuse/...) sub-targets - set TEST_FUSE=0 in test_cli so FUSE tests skip in cli-tests CI job - increase fuse-tests CI timeout from 5m to 10m for CLI tests - delete sharness t0030, t0031, t0032 (were always skipped in CI) * docs: document FUSE test split between unit and e2e Add cross-reference comments between the unit tests in fuse/readonly/, fuse/ipns/, fuse/mfs/ and the end-to-end CLI tests in test/cli/fuse/. Also fix AGENTS.md to use a temp dir for fusermount symlink instead of sudo. * ci: prevent stale FUSE mounts from failing fuse-tests On shared self-hosted runners, leftover mount points from previous runs can exhaust the kernel FUSE mount limit. - add job-level concurrency group so only one fuse-tests runs at a time - lazy-unmount stale /tmp/fusetest* mounts before running tests * ci: only symlink fusermount3 when fusermount is missing * fix(fuse): remove goroutine leak in IPNS Flush handler The Flush handler wrapped fi.fi.Flush() in a goroutine so it could return early when the FUSE context was canceled. But the goroutine kept running in the background, and when Release arrived it called Close on the same file descriptor concurrently. The two paths both entered DagModifier.Sync, racing on its internal write buffer and causing a nil pointer panic. The fix is to call Flush directly without a goroutine. The MFS flush cannot be safely canceled mid-operation anyway, so the goroutine only added the illusion of cancellation while leaking work and masking the real error. Also bumps boxo to pick up the matching defense-in-depth fix that serializes FileDescriptor.Flush and Close with a mutex. * fix(fuse): add mutex to IPNS file handle operations bazil/fuse dispatches each FUSE request in its own goroutine. The IPNS File handle had no synchronization, so concurrent Read/Write/Flush/Release calls could overlap on the underlying DagModifier which is not safe for concurrent use. Add sync.Mutex to File, matching the pattern already used by the MFS FileHandler. * refactor(fuse): remove dead File.Forget method bazil/fuse only dispatches Forget to nodes via the NodeForgetter interface. File is a handle, not a node, so this method was never called. The /mfs mount has no equivalent. * fix(fuse): flush IPNS directory after Remove and Rename The /mfs mount flushes the directory after Unlink and Rename so changes propagate to the MFS root immediately. The /ipns mount did not, leaving mutations pending until an unrelated flush. Also add an empty-directory check before removing directories, matching the /mfs mount's safety check. * fix(fuse): inherit CID builder and flush on IPNS Create New files created via the /ipns FUSE mount now inherit the CID builder from their parent directory, preventing CIDv0 nodes from appearing inside a CIDv1 tree. The directory is also flushed after AddChild so the new entry propagates to the MFS root immediately, matching the /mfs mount. * test(fuse): add IPNS Remove and non-empty rmdir tests Cover the file removal path and the empty-directory safety check added in the previous commit. TestRemoveFile verifies a created file can be removed and is gone afterwards. TestRemoveNonEmptyDirectory verifies that rmdir on a directory with children fails, and succeeds once the children are removed first. * feat(fuse): read UnixFS mode/mtime, add StoreMtime/StoreMode config All three FUSE mounts now read mode and mtime from UnixFS metadata when present, falling back to POSIX defaults when absent. Most IPFS data does not include this optional metadata. Writing mode and mtime is opt-in via two new config flags: - Mounts.StoreMtime: persist mtime on file create and open-for-write - Mounts.StoreMode: persist mode on chmod Other changes in this commit: - align default file/dir modes across /ipns and /mfs to 0644/0755 - share mode constants via fuse/mount/mode.go - convert Mounts.FuseAllowOther from bool to Flag for consistency - add Setattr to /ipns FileNode and /mfs File for chmod and touch - move dead File.Setattr from IPNS handle to FileNode (node) - bump boxo for Directory.Mode() and Directory.ModTime() getters * feat(fuse): add ipfs.cid xattr to all mounts All three FUSE mounts now expose the node's CID via the ipfs.cid extended attribute on both files and directories. The /mfs mount also accepts the old ipfs_cid name for backward compatibility. The /ipfs mount previously had a stub that returned nil for all xattrs; it now returns the correct CID. The xattr name follows the convention used by CephFS (ceph.*), Btrfs (btrfs.*), and GlusterFS (glusterfs.*). * feat(fuse): switch from bazil.org/fuse to hanwen/go-fuse v2 Replace the unmaintained bazil.org/fuse (last commit 2020) with hanwen/go-fuse v2.9.0, fixing two architectural issues that could not be solved with the old library. ftruncate now works: hanwen/go-fuse passes the open file handle to NodeSetattrer, so Setattr can truncate through the existing write descriptor instead of trying to open a second one (which deadlocks on MFS's single-writer lock). fsync now works: FileFsyncer runs on the handle directly, flushing the write buffer through the open descriptor. Previously a no-op because bazil dispatched Fsync to the inode only. mount package: - NewMount takes (InodeEmbedder, mountpoint, *fs.Options) instead of (fs.FS, mountpoint, allowOther) - mount/unmount collapses to a single fs.Mount call - fusermount3 tried before fusermount in ForceUnmount all three mounts: - structs embed fs.Inode (hanwen's InodeEmbedder pattern) - Remove split into Unlink + Rmdir (separate FUSE interfaces) - ReadDirAll replaced with Readdir returning DirStream - fillAttr helper shared between Getattr and Lookup responses - kernel cache invalidation via NotifyContent after Flush - 1s entry/attr timeout for writable mounts (matches go-fuse default, gocryptfs, rclone) - O_APPEND tracked on file handle, writes seek to end - build tags standardized to (linux || darwin || freebsd) && !nofuse tests: - replaced bazil fstestutil.MountedT with shared fusetest.TestMount - fixed TestConcurrentRW: channel drain mismatch and missing sync between write Close and read start - added TestFsync, TestFtruncate, TestReadlink, TestSeekRead, TestLargeFile, TestRmdir, TestCrossDirRename, TestUnknownXattr - added StoreMtime disabled/enabled subtests * fix(fuse): close fd on error in Open to prevent leak MFS enforces a single-writer lock, so a leaked write descriptor blocks all subsequent opens of that file until GC. * fix(fuse): detect external unmount via server.Wait Without this, IsActive stays true after `fusermount -u` and Unmount returns nil instead of ErrNotMounted. * fix(fuse): return actual error from Unlink/Rmdir, not ENOENT After confirming the child exists, an Unlink failure could be an IO error. Returning ENOENT would hide the real cause. * fix(fuse): reuse DagReader per open, pass ctx to all reads Readonly Open now returns a file handle holding a DagReader instead of recreating one per Read call. Sequential reads no longer re-traverse the DAG from the root on each kernel request. All three mounts now use CtxReadFull with the kernel's per-request context so killing a process mid-read cancels in-flight block fetches instead of letting them complete uselessly. * chore(fuse): cleanup dead code, add var comments - remove dead `_ = mntDir` in TestXattrCID - comment why immutableAttrCacheTime and mutableCacheTime are var - add TODO for using IPNS record TTL as cache timeout * chore(fuse): replace OSXFUSE 2.x check with macFUSE detection The old check tried to verify OSXFUSE >= 2.7.2 to avoid a kernel panic from 2015. It used sysctl, tried to `go install` a third-party tool at runtime, and referenced paths that no longer exist. Replace with a simple check for the macFUSE mount helper, matching the same paths go-fuse looks for. If neither macFUSE nor OSXFUSE is found, point the user to the install page. Also standardize build tags to (linux || darwin || freebsd) && !nofuse and use strings.ReplaceAll. * fix(fuse): include mountpoint path in mount errors go-fuse's fusermount errors don't include the path, so tools that check error messages for the mountpoint name couldn't tell which mount failed. * chore(ci): remove bazil fusermount workaround go-fuse finds fusermount3 natively, no symlink needed. The stale mount cleanup was for bazil's fstestutil which we no longer use. * docs: update v0.41 changelog for FUSE rewrite * chore(deps): bump boxo for full FileDescriptor serialization boxo@64be0815 extends the mutex from Flush/Close to all FileDescriptor operations (Read, Write, Seek, Truncate, Size), preventing data races on the underlying DagModifier. * chore(deps): bump boxo to merged ipfs/boxo#1133 Picks up full FileDescriptor serialization: the mutex now covers all operations (Read, Write, Seek, Truncate, Size), not just Flush and Close. * feat(fuse): CAP_ATOMIC_O_TRUNC, new integration tests Advertise CAP_ATOMIC_O_TRUNC so the kernel sends O_TRUNC inside Open instead of doing a separate SETATTR(size=0) first. Without this, the kernel's SETATTR needs to open a write descriptor inside Setattr, which deadlocks on MFS's single-writer lock. Move kernel cache invalidation from Flush to Release because mfsFD.Close (in Release) is where the final DAG node is committed. Upgrade go-fuse to latest for ExtraCapabilities support. New tests for both MFS and IPNS: - TestOpenTrunc, TestSeekAndWrite, TestOverwriteExisting - TestTempFileRename, TestVimSavePattern, TestRsyncPattern (skipped pending rename-over-existing and cache fixes) * fix(fuse): rename-over-existing, bump boxo for flushUp race fix IPNS Rename now unlinks the target before AddChild, matching MFS. Without this, renaming onto an existing name returned "directory already has entry". Bump boxo to pick up the flushUp unlinked-entry fix (ipfs/boxo@8ae46d5): when a file descriptor outlives its directory entry (FUSE RELEASE racing with RENAME), flushUp no longer re-adds the stale name. Unskip TestTempFileRename and TestRsyncPattern on both mounts. * fix(fuse): unskip VimSavePattern, bump boxo for setNodeData fix boxo@552d8e7 fixes File.setNodeData dropping content links when updating metadata (mode, mtime). chmod or touch after write no longer makes the file appear empty. Unskip TestVimSavePattern on both mounts. Remove debug logging and temporary test functions added during investigation. * fix(fuse): build tags for cross-compilation go-fuse does not compile on windows/openbsd/netbsd/plan9. Move WritableMountCapabilities (which imports go-fuse) from mode.go (no build tag) to caps.go (platform-gated). Align build tags on fusetest and core/commands/mount stubs so unsupported platforms don't pull in go-fuse transitively. * fix(test): use fusermount3 in CLI FUSE tests The doUnmount helper hardcoded fusermount, but systems with only fuse3 installed have fusermount3. Try fusermount3 first, matching what go-fuse and our ForceUnmount already do. * feat(fuse): symlink support on writable mounts Add NodeSymlinker to MFS and IPNS directories. Symlinks are stored as UnixFS TSymlink nodes in the DAG, the same format used by `ipfs add` for directories containing symlinks. The readonly /ipfs mount already rendered existing symlinks; now /mfs and /ipns can create them too. The target string is cached at Lookup time to avoid re-parsing the DAG node on every Readlink call. Symlink permissions are always 0777 per POSIX convention (access control uses the target's mode). * fix(fuse): checked type assertion in MFS Rename The direct type assertion on newParent could panic if the kernel passed a non-directory inode. Use a checked assertion with EINVAL fallback, matching the type-switch pattern in the IPNS mount. * fix(test): add missing continue in stress test Missing continue after error sends let execution fall through to nil type assertions (read.(files.File)) that would panic on error. Also cancel the context before continuing to avoid leaking it. * fix(fuse): return error from Readdir when DAG.Get fails Abort the directory listing instead of silently omitting the unretrievable entry. Callers get EIO, which is more honest than a partial listing that hides missing blocks. * docs: remove duplicate fsync bullet in changelog * ci: clean up stale FUSE mounts in fuse-tests job On shared self-hosted runners, leftover mounts from crashed runs can exhaust the kernel mount_max limit. Lazy-unmount kubo-test and harness temp mounts before and after tests. * chore(deps): bump boxo to merged ipfs/boxo#1134 Picks up flushUp unlinked-entry guard and setNodeData content link preservation. * docs: add build tag comments, normalize tag style Add a one-line comment above every //go:build directive explaining why the constraint exists. Normalize tag style: positive platform constraints first, then feature flags/negations. Simplify redundant expressions. * fix(fuse): add Setattr to directories for chmod and mtime Tools like tar and rsync call utimensat on directories after extraction. Without Setattr on Dir, this returned ENOTSUP. Add Setattr to Dir (MFS) and Directory (IPNS) that handles mode and mtime the same way as the file-level Setattr. When StoreMtime or StoreMode is disabled the call succeeds silently, matching the file-level behavior. * docs: clarify directory support and spec link for StoreMtime/StoreMode - mention that touch and chmod work on both files and directories - note tar and rsync as practical use cases - link to UnixFS spec for optional metadata storage * fix(fuse): use proper mode conversion, document 9-bit limit Use files.UnixPermsToModePerms and files.ModePermsToUnixPerms for converting between FUSE kernel mode (unix 12-bit layout) and Go's os.FileMode (different bit positions for setuid/setgid/sticky). The UnixFS spec supports all 12 permission bits, but boxo's MFS layer (File.Mode, Directory.Mode) exposes only the lower 9. FUSE mounts are always nosuid so the upper 3 bits would have no effect. Add TestSetuidBitsStripped to both mounts confirming the behavior. * feat(fuse): symlink Setattr with mtime persistence Wire the backing mfs.File into the FUSE Symlink struct so Setattr can call SetModTime when StoreMtime is enabled. boxo's File methods (SetModTime, ModTime) already work on TSymlink nodes since they operate on the FSNode protobuf without checking the type. Without Setattr, rsync -a fails with "failed to set times" on symlinks. Every major FUSE filesystem (gocryptfs, rclone, sshfs, s3fs) implements Setattr on symlinks for this reason. Mode is always 0777 per POSIX convention, so chmod requests are silently accepted but not stored. * fix(fuse): return EIO instead of panicking on unknown node type Replace panic with log.Errorf + syscall.EIO in IPNS Directory.Lookup for unexpected MFS node types. Also remove duplicate comment block on File.Flush. * docs: update FUSE docs for go-fuse migration - fuse.md: replace stale OSXFUSE section with macFUSE, remove obsolete go-fuse-version tool, fix broken FreeBSD sudo echo, update xattr example to ipfs.cid with CIDv1, add mode/mtime section, add unixfs-v1-2025 tip, add debug logging section, add TOC, link to hanwen/go-fuse - changelog: refine bullet wording, link to fuse.md - config.md: fix double space, update fuse.md link text - experimental-features.md: fix double space, soften wording - README.md: add FUSE to features list and docs table * refactor(fuse): extract shared writable types and test suite Extract duplicated code from fuse/mfs and fuse/ipns into a shared fuse/writable package, and consolidate duplicated tests into a reusable suite in fuse/fusetest. - fuse/writable: Dir, FileInode, FileHandle, Symlink types with all FUSE interface methods, shared by both mounts - fuse/fusetest: RunWritableSuite with helpers, exercised by both mfs and ipns via mount-specific factories - fix cache invalidation race: NotifyContent in Flush (synchronous) in addition to Release (async), so stat after close sees new size - drop deprecated ipfs_cid xattr, log error guiding users to ipfs.cid - mfs_unix.go: 632 -> 19 lines (thin wrapper over writable.Dir) - ipns_unix.go: 795 -> 170 lines (Root + key resolution only) - mfs_test.go: 1183 -> 95 lines (factory + persistence test) - ipns_test.go: 1309 -> 162 lines (factory + IPNS-specific tests) - tests that were only in one mount now run on both * feat(fuse): add macOS-specific mount options Set volname, noapplexattr, and noappledouble on macOS via PlatformMountOpts, applied in NewMount so all three mounts benefit automatically. - volname: shows mount name in Finder instead of "macfuse Volume 0" - noapplexattr: suppresses Finder's com.apple.* xattr probes - noappledouble: prevents ._ resource fork sidecar files * fix(fuse): detect symlinks in readdir, fix stale refs Readdir on writable mounts now checks the underlying DAG node type for TFile entries, reporting S_IFLNK for symlinks instead of regular file. This makes ls -l and find -type l work correctly. - writable: Readdir checks SymlinkTarget for TFile entries - writablesuite: add SymlinkReaddir regression test - readonly: add TestReaddirSymlink regression test - test/cli/fuse: fix stale bazil.org/fuse reference in doc comment * fix(fuse): normalize deprecated ipfs_cid xattr to ipfs.cid Getxattr for the old "ipfs_cid" name now returns the CID instead of ENOATTR, keeping existing tooling working during the deprecation period. A log error is emitted on each access to nudge migration. * fix(fuse): serialize concurrent reads on readonly file handles The go-fuse server dispatches each FUSE request in its own goroutine. On files larger than 128 KB the kernel issues concurrent readahead Read requests on the same file handle, racing on the shared DagReader's Seek+CtxReadFull sequence and corrupting its internal state. Add sync.Mutex to roFileHandle (matching the existing pattern in writable.FileHandle) and lock in Read and Release. - fuse/readonly/readonly_unix.go: add mu sync.Mutex to roFileHandle - fuse/readonly/ipfs_test.go: add TestConcurrentLargeFileRead - fuse/fusetest/writablesuite.go: add LargeFileConcurrentRead to shared writable suite (exercised by both /mfs and /ipns tests) * fix(fuse): bypass MFS locking for read-only opens MFS uses an RWMutex (desclock) that holds RLock for the lifetime of a read descriptor and requires exclusive Lock for writes. Tools like rsync --inplace open the same file for reading and writing from separate processes, deadlocking on this mutex. For O_RDONLY opens, create a DagReader directly from the current DAG node instead of going through MFS. The reader gets a point-in-time snapshot and never touches desclock, so writers proceed independently. - fuse/writable/writable.go: add roFileHandle with DagReader for read-only opens, add DAG field to Config - fuse/mfs/mfs_unix.go: pass ipfs.DAG to writable Config - fuse/ipns/ipns_unix.go: pass ipfs.Dag() to writable Config - fuse/fusetest/writablesuite.go: add ConcurrentReadWrite test exercising simultaneous read and write on the same file * fix(fuse): support truncate(path, size) without open fd Open a temporary write descriptor in Setattr when the kernel sends a size change without a file handle (the truncate(2) syscall, as opposed to ftruncate(fd) which passes the handle). Previously this returned ENOTSUP. - fuse/writable: open, truncate, flush, close in Setattr else branch - fuse/fusetest: add TruncatePath to the shared writable suite - test/cli/fuse: add end-to-end truncation test covering ftruncate(fd), syscall.Truncate(path), and open(O_TRUNC) through a real daemon * ci(fuse): get stack traces on test hangs The fuse-tests job was being silently cancelled by GitHub at 10min because Go's per-test timeout (5m) was the same order as the job timeout, and GOTRACEBACK=single hid the hung goroutines anyway. - shrink TEST_FUSE_TIMEOUT to 4m so Go's panic fires first - shrink job timeout-minutes to 6 (normal run is ~3min) - set GOTRACEBACK=all so the panic dumps every goroutine, not just the timer * fix(fuse): fill attrs in FileInode.Setattr response Without this, the kernel could cache zero attrs after a chmod, touch, or ftruncate until AttrTimeout (1s) expired. Dir.Setattr and Symlink.Setattr already fill out.Attr; FileInode.Setattr now matches. * docs(config): clarify Mounts.IPNS writability scope Only directories backed by keys the node holds are writable. All other names resolve via IPNS to read-only symlinks into the /ipfs mount. * fuse: review cleanup for go-fuse migration Final pass on #11272 addressing review feedback. - writable: panic in NewDir if Config.DAG is nil. Both call sites already supply it, but a nil value silently fell back to the MFS path in FileInode.Open, re-introducing the rsync --inplace deadlock the read-only fast path was added to fix. - writable: document Dir.Rename non-atomicity. Source unlink happens before destination add, so any failure between the two loses the source. An atomic fix requires changes in boxo/mfs. - writable: add unit test locking in that Symlink.Setattr accepts a mode-only request without erroring and does not store the requested mode (POSIX symlinks have no meaningful permission bits). - docs/config: correct StoreMode default modes; the previous text listed 0666 for files, which the code never uses. * docs(config): list StoreMtime and StoreMode in Mounts TOC * fix(fuse): fill EntryOut attrs in Dir.Create and Dir.Mkdir Without this, fstat on the file handle returned by Create reports mode 0 and size 0 for up to AttrTimeout (1s), because the kernel caches the empty attrs from the Create response. Path-based stat goes through Lookup which already fills attrs, so the bug only shows up via fstat. Mirrors the same fix already applied to FileInode.Setattr. Dir.Mkdir gets the same fillAttr treatment for consistency, plus a TODO noting that boxo's mfs.Directory.Mkdir accepts no mode arg so the caller's mode is dropped on creation. Adds CreateAttrsImmediate and MkdirAttrsImmediate to the shared writable suite to guard both paths against future regressions. * fix(fuse): map context cancellation to EINTR in read paths When a userspace process is killed mid-read (Ctrl-C, SIGKILL on a stuck cat) the kernel sends FUSE_INTERRUPT and go-fuse cancels the per-request context. fs.ToErrno does not recognise context.Canceled and falls through to "function not implemented", which the kernel cannot act on. Map context.Canceled and DeadlineExceeded to EINTR so the syscall is correctly aborted. - mount/errno.go: new ReadErrno helper used by all context-aware read paths in both readonly and writable mounts - readonly: applied to Node.Open, Node.Readdir, roFileHandle.Read - writable: applied to FileInode.Open, FileHandle.Read, roFileHandle.Read - readonly/ipfs_test.go: TestReadCancellationUnblocks guards the contract via a blocking DagReader fake; without ReadErrno the test reports "function not implemented" instead of EINTR * test(fuse): add OExcl, DirRename, SparseWrite, FsyncCrossHandle Coverage gaps in the shared writable suite: - OExcl: lock files and atomic-create patterns rely on the second open with O_CREATE|O_EXCL failing with EEXIST - DirRename: previously only file rename and cross-dir file rename were tested; this exercises Rename on a directory inode - SparseWrite: WriteAt past the end of an empty file must report the correct size and return zeros for the gap - FsyncCrossHandle: a reader on a fresh fd must see data flushed by fsync on the writer fd, not just after close * test(fuse): cover external unmount on /ipns and /mfs Previously TestExternalUnmount only exercised /ipfs, leaving the goroutine that watches fuse.Server.Wait() untested for the other two mounts. Refactor into a table-driven test that runs the same fusermount/umount-then-IsActive flow against all three mounts. Switch to coremock.NewMockNode so the node is online: doMount only attaches the /ipns mount when node.IsOnline is true, and the table needs all three populated. * fix(commands): align 'ipfs mount' output columns MountCmd's LongDescription has "MFS mounted at:" with two spaces so the column lines up with the 4-char "IPFS" and "IPNS" rows above, but the runtime encoder and the daemon's startup print used a single space and produced misaligned output. Bring both runtime sites in line with the help text, and update the two existing test fixtures (test/cli/fuse and the sharness test-lib helper that t0040-add-and-cat.sh still uses) to expect the aligned form. * fix(fuse): invalidate kernel cache on Fsync FileHandle.Fsync only flushed the MFS file descriptor and left the kernel's cached attrs and content for the inode untouched. A fresh reader on the same path then saw the size cached from the original Create response (zero), reading zero bytes regardless of how much the writer had synced. Mirror the cache invalidation already done in Flush via inode.NotifyContent(0, 0) so a writer that fsyncs while another process opens the file (vim then a follow-up cat, IDE then a language server) sees consistent state. Sharpen the FsyncCrossHandle assertion to report the size delta on failure; the bug surfaced as got=0/want=500 only after switching from bytes.Equal to require.Equal. * chore(gitignore): ignore test_fuse_unit and test_fuse_cli json output The new test_fuse_unit and test_fuse_cli make targets emit test/fuse/fuse-unit-tests.json and test/fuse/fuse-cli-tests.json respectively, the same gotestsum --jsonfile pattern that test_unit and test_cli already use. Add them to the same .gitignore section so a local test run does not leave the working tree dirty. * test(fuse): end-to-end coverage with real POSIX tools Adds TestFUSERealWorld in test/cli/fuse/realworld_test.go: a single shared-daemon test with 18 subtests that exercise the writable /mfs mount through the actual binaries users invoke (sh, cat, seq, wc, ls, stat, cp, mv, rm, ln, readlink, find, dd, sha256sum, tar, rsync, vim). Each subtest verifies the result both via the FUSE filesystem and via 'ipfs files read|stat|ls' so both views agree. Synthetic payloads default to 1 MiB + 1 byte so multi-chunk read/write paths are exercised, not just single-chunk fast paths. External tools are required, not optional: a missing binary fails the test loudly so a CI image change cannot silently turn the suite green. The whole-suite TEST_FUSE gate is the only place a developer is allowed to skip. runCmd forces LC_ALL=C so locale-sensitive tool output (date formats in 'ls -l', decimal separators in 'wc', localized error messages, find/ls collation) is deterministic regardless of the runner's locale settings. One shared daemon across all 18 subtests keeps total runtime under two seconds; isolation comes from per-subtest subdirectories under the mount. | 4 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
feat: add built-in `ipfs update` command (#11203) * feat: add built-in `ipfs update` command adds `ipfs update` command tree that downloads pre-built Kubo binaries from GitHub Releases, verifies SHA-512 checksums, and replaces the running binary in place. subcommands: - `ipfs update check` -- query GitHub for newer versions - `ipfs update versions` -- list available releases - `ipfs update install [version]` -- download, verify, backup, and atomically replace the current binary - `ipfs update revert` -- restore the previously backed up binary from `$IPFS_PATH/old-bin/` read-only subcommands (check, versions) work while the daemon is running. install and revert require the daemon to be stopped first. design decisions: - uses GitHub Releases API instead of dist.ipfs.tech because GitHub is harder to censor in regions that block IPFS infrastructure - honors GITHUB_TOKEN/GH_TOKEN to avoid unauthenticated rate limits - backs up the current binary before replacing, with permission-error fallback that saves to a temp dir with manual `sudo mv` instructions - `KUBO_UPDATE_GITHUB_URL` env var redirects API calls for integration testing; `IPFS_VERSION_FAKE` overrides the reported version - unit tests use mock HTTP servers and the var override; CLI tests use the env vars with a temp binary copy so the real build is never touched resolves https://github.com/ipfs/kubo/issues/10937 * fix(update): harden download and extraction - cap decompressed binary at 1 GB to block zip/tar bombs - propagate tar.gz/zip errors instead of swallowing them - fall back to 1h context timeout when --timeout is not set - warn on stderr when daemon lock check fails - clarify that fetch+verify+extract complete before touching binary * fix(update): resolve binary path on windows The test harness hardcodes the binary path as `cmd/ipfs/ipfs` without the `.exe` suffix. On Windows the built binary is `ipfs.exe`, so copyBuiltBinary needs to append the extension. * fix(update): handle windows binary locking in install test On Windows the OS locks the running executable, so atomicfile cannot rename over it. The install command falls back to saving the new binary to a temp path. Accept both outcomes in TestUpdateInstall: in-place replacement (Unix) or permission-denied fallback (Windows). Also fix stash path to include .exe suffix on Windows. - test/cli/update_test.go: branch on runtime.GOOS for install assertions - test/sharness/t0063-external.sh: remove, tested the old ExternalBinary delegation which is replaced by the built-in update command - .github/workflows/test-migrations.yml: pass GITHUB_TOKEN to avoid rate limits * fix(test): handle windows EINVAL on process signal after wait On Windows, Process.Wait() sets the handle state to "released" rather than "done", so a subsequent Signal() returns syscall.EINVAL instead of os.ErrProcessDone. This caused StopDaemon cleanup to panic on Windows CI. Treat both errors as "process already exited". * feat(update): add 'clean' subcommand Drops every backed-up Kubo binary from $IPFS_PATH/old-bin/ so users can reclaim disk space without hand-deleting files. Safe with the daemon running, only touches the backup directory. - update.go: extract stashDirName const, factor out listStashes() helper, add updateCleanCmd - commands_test.go: register /update/clean - test/cli/update_test.go: TestUpdateClean covers removal, empty dir, json output, and preservation of unrelated files * docs(changelog): tighten ipfs update entry Drop the marketing opener, the duplicate install example, and the revert/versions sentence; all are covered by 'ipfs update --help'. Mention the new 'clean' subcommand in the trailing pointer. * fix(test): skip fuse cli tests on non-unix Both fuse_test.go and realworld_test.go rely on Unix-only APIs (syscall.Truncate, POSIX tools). The sibling xattr_*_test.go files were already gated, but these two compiled everywhere, so any workflow running 'go test ./test/cli/...' on Windows hit 'undefined: syscall.Truncate'. Use the same '(linux || darwin || freebsd) && !nofuse' constraint that the fuse/ packages already use so platform gating is consistent. * fix(test): run install/revert sequentially TestUpdateInstall and TestUpdateRevert write a copy of the ipfs binary and then exec it. When other tests run in parallel, a concurrent fork() can inherit the still-open write fd into its child, leaving the freshly written file 'text file busy' for exec until the sibling child execs. Dropping t.Parallel() on these two tests ensures no other goroutine is mid-fork while the binary is being written, which is the only reliable way to avoid the ETXTBSY race without clever fd tricks. * ci(update): use cloudflare/google DNS on macos GitHub's macOS runners intermittently lose DNS for api.github.com, which fails the real-network subtests in TestUpdate. Point the resolver at 1.1.1.1 and 8.8.8.8 on every active network service and flush the DNS cache before running the update tests. * fix(update): fsync before close in atomicfile and stash * fix(update): use unique temp file in permission fallback The previous fallback wrote to a predictable path (/tmp/ipfs-<ver>), which on shared systems lets a local attacker pre-create the path as a symlink and steer the user's subsequent 'sudo mv' anywhere. Switch to os.CreateTemp so the path is unique and exclusively owned by this process. * refactor(update): rename test env vars to TEST_KUBO_* IPFS_VERSION_FAKE and KUBO_UPDATE_GITHUB_URL are test-only escape hatches with no production use case. The TEST_ prefix signals this clearly and reduces the chance of accidental use in production. - IPFS_VERSION_FAKE -> TEST_KUBO_VERSION - KUBO_UPDATE_GITHUB_URL -> TEST_KUBO_UPDATE_GITHUB_URL * style(update): unshadow err in stashBinary * fix(update): warn when IPFS path can't be resolved silently skipping the daemon lock check on path-resolution failure can mask a misconfigured IPFS_PATH; print a warning so the user notices before the install proceeds. * fix(update): revert atomicfile Sync, use errors.Is for EOF Revert the Sync() addition in atomicfile.Close() to avoid widening the failure surface for existing migration callers that panic on Close errors (Must(out.Close()) in WithBackup). The stashBinary fsync in update.go is kept since that code path is new. - revert repo/fsrepo/migrations/atomicfile/atomicfile.go to master - use errors.Is(err, io.EOF) in extractFromTarGz | 4 个月前 | |
core/corehttp!: remove /api/v0 from gateway port | 2 年前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
refactor: rename to kubo | 4 年前 | |
feat: `Provider.WorkerCount` and `stats reprovide` (#10779) * adjust ipfs stats provide * update boxo dep * bump boxo * fixing tests * docs/chore: mark stat reprovide as experimental * docs: Provider.Strategy explicitly document it is not used - without this legacy users will have it in their config and be very confused --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 年前 | |
feat(cli/rpc/add): fast provide of root CID (#11046) * feat: fast provide * Check error from provideRoot * do not provide if nil router * fix(commands): prevent panic from typed nil DHTClient interface Fixes panic when ipfsNode.DHTClient is a non-nil interface containing a nil pointer value (typed nil). This happened when Routing.Type=delegated or when using HTTP-only routing without DHT. The panic occurred because: - Go interfaces can be non-nil while containing nil pointer values - Simple `if DHTClient == nil` checks pass, but calling methods panics - Example: `(*ddht.DHT)(nil)` stored in interface passes nil check Solution: - Add HasActiveDHTClient() method to check both interface and concrete value - Update all 7 call sites to use proper check before DHT operations - Rename provideRoot → provideCIDSync for clarity - Add structured logging with "fast-provide" prefix for easier filtering - Add tests covering nil cases and valid DHT configurations Fixes: https://github.com/ipfs/kubo/pull/11046#issuecomment-3525313349 * feat(add): split fast-provide into two flags for async/sync control Renames --fast-provide to --fast-provide-root and adds --fast-provide-wait to give users control over synchronous vs asynchronous providing behavior. Changes: - --fast-provide-root (default: true): enables immediate root CID providing - --fast-provide-wait (default: false): controls whether to block until complete - Default behavior: async provide (fast, non-blocking) - Opt-in: --fast-provide-wait for guaranteed discoverability (slower, blocking) - Can disable with --fast-provide-root=false to rely on background reproviding Implementation: - Async mode: launches goroutine with detached context for fire-and-forget - Added 10 second timeout to prevent hanging on network issues - Timeout aligns with other kubo operations (ping, DNS resolve, p2p) - Sufficient for DHT with sweep provider or accelerated client - Sync mode: blocks on provideCIDSync until completion (uses req.Context) - Improved structured logging with "fast-provide-root:" prefix - Removed redundant "root CID" from messages (already in prefix) - Clear async/sync distinction in log messages - Added FAST PROVIDE OPTIMIZATION section to ipfs add --help explaining: - The problem: background queue takes time, content not immediately discoverable - The solution: extra immediate announcement of just the root CID - The benefit: peers can find content right away while queue handles rest - Usage: async by default, --fast-provide-wait for guaranteed completion Changelog: - Added highlight section for fast root CID providing feature - Updated TOC and overview - Included usage examples with clear comments explaining each mode - Emphasized this is extra announcement independent of background queue The feature works best with sweep provider and accelerated DHT client where provide operations are significantly faster. * fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * Update core/commands/add.go --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 9 个月前 | |
refactor: move `ipfs stat provide/reprovide` to `ipfs provide stat` (#10896) - Move `ipfs stat reprovide` to `ipfs provide stat` - Mark `ipfs stat provide` as deprecated and replaces by `ipfs provide stat` - Mark `ipfs stat reprovide` as deprecated and replaces by `ipfs provide stat` - Remove redundant code from deprecated subcommands Closes #10869 | 1 年前 | |
feat(ci): reusable spellcheck from unified CI (#10873) * ci: use spellcheck from unified CI * chore: fix spelling --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 年前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
fix(http-routing): keep browser transports in provider records (#11394) * chore: bump go-libp2p for sorted confirmed addrs Pin the head commit of libp2p/go-libp2p#3526: AutoNAT V2's ConfirmedAddrs returned unsorted buckets, and removeNotInSource silently dropped webrtc-direct from the confirmed set. Switch to a master pseudo-version once the PR merges. * fix: keep browser transports in provider records Provider records sent to HTTP routers were narrowed to the addresses AutoNAT V2 confirmed reachable, which silently dropped the only two transports a browser can dial: the AutoTLS /tls/ws address and webrtc-direct. A publicly reachable node was invisible to browser and Helia clients that found it through a delegated router, even though ipfs id and the DHT both advertised those addresses. AutoNAT only ever sees listen addresses, so the AutoTLS address, which the AddrsFactory synthesizes afterwards, can never reach the confirmed set. webrtc-direct does get confirmed, but go-libp2p loses it again in getConfirmedAddrs, which feeds an unsorted slice to a scan that assumes sorted input; that one is fixed upstream in libp2p/go-libp2p#3526. Announce host.Addrs() instead, the same set identify sends to peers and the DHT already publishes, narrowed to globally routable addresses so loopback and LAN entries stay out of a public index. Nodes with no public address keep announcing what they have, so LAN-only setups pointing at a local router are unaffected. - core/node/libp2p/routingopt.go: drop the ConfirmedAddrs branch from httpRouterAddrFunc, filter host.Addrs() with manet.IsPublicAddr; AppendAnnounce is emitted exactly once and does not count toward the public-addr check - core/commands/swarm_addrs_autonat.go: take over the BasicHost compile-time assertion, now the only ConfirmedAddrs consumer Fixes #11369 * docs: move highlight to v0.43 and scope it The fix ships in v0.43, so the entry moves out of v0.44.md and in next to the other browser-retrieval highlights. - names the config it applies to: Routing.Type=custom with a provide method on an HTTP router. Default auto provides over the DHT alone and is unaffected, since constructDefaultHTTPRouters leaves ProvideRouter as a noop. - cites bitsocial.net, which runs libp2p in the browser and uses delegated routers to find peers, as the app the gap broke | 1 个月前 | |
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
feat: add built-in `ipfs update` command (#11203) * feat: add built-in `ipfs update` command adds `ipfs update` command tree that downloads pre-built Kubo binaries from GitHub Releases, verifies SHA-512 checksums, and replaces the running binary in place. subcommands: - `ipfs update check` -- query GitHub for newer versions - `ipfs update versions` -- list available releases - `ipfs update install [version]` -- download, verify, backup, and atomically replace the current binary - `ipfs update revert` -- restore the previously backed up binary from `$IPFS_PATH/old-bin/` read-only subcommands (check, versions) work while the daemon is running. install and revert require the daemon to be stopped first. design decisions: - uses GitHub Releases API instead of dist.ipfs.tech because GitHub is harder to censor in regions that block IPFS infrastructure - honors GITHUB_TOKEN/GH_TOKEN to avoid unauthenticated rate limits - backs up the current binary before replacing, with permission-error fallback that saves to a temp dir with manual `sudo mv` instructions - `KUBO_UPDATE_GITHUB_URL` env var redirects API calls for integration testing; `IPFS_VERSION_FAKE` overrides the reported version - unit tests use mock HTTP servers and the var override; CLI tests use the env vars with a temp binary copy so the real build is never touched resolves https://github.com/ipfs/kubo/issues/10937 * fix(update): harden download and extraction - cap decompressed binary at 1 GB to block zip/tar bombs - propagate tar.gz/zip errors instead of swallowing them - fall back to 1h context timeout when --timeout is not set - warn on stderr when daemon lock check fails - clarify that fetch+verify+extract complete before touching binary * fix(update): resolve binary path on windows The test harness hardcodes the binary path as `cmd/ipfs/ipfs` without the `.exe` suffix. On Windows the built binary is `ipfs.exe`, so copyBuiltBinary needs to append the extension. * fix(update): handle windows binary locking in install test On Windows the OS locks the running executable, so atomicfile cannot rename over it. The install command falls back to saving the new binary to a temp path. Accept both outcomes in TestUpdateInstall: in-place replacement (Unix) or permission-denied fallback (Windows). Also fix stash path to include .exe suffix on Windows. - test/cli/update_test.go: branch on runtime.GOOS for install assertions - test/sharness/t0063-external.sh: remove, tested the old ExternalBinary delegation which is replaced by the built-in update command - .github/workflows/test-migrations.yml: pass GITHUB_TOKEN to avoid rate limits * fix(test): handle windows EINVAL on process signal after wait On Windows, Process.Wait() sets the handle state to "released" rather than "done", so a subsequent Signal() returns syscall.EINVAL instead of os.ErrProcessDone. This caused StopDaemon cleanup to panic on Windows CI. Treat both errors as "process already exited". * feat(update): add 'clean' subcommand Drops every backed-up Kubo binary from $IPFS_PATH/old-bin/ so users can reclaim disk space without hand-deleting files. Safe with the daemon running, only touches the backup directory. - update.go: extract stashDirName const, factor out listStashes() helper, add updateCleanCmd - commands_test.go: register /update/clean - test/cli/update_test.go: TestUpdateClean covers removal, empty dir, json output, and preservation of unrelated files * docs(changelog): tighten ipfs update entry Drop the marketing opener, the duplicate install example, and the revert/versions sentence; all are covered by 'ipfs update --help'. Mention the new 'clean' subcommand in the trailing pointer. * fix(test): skip fuse cli tests on non-unix Both fuse_test.go and realworld_test.go rely on Unix-only APIs (syscall.Truncate, POSIX tools). The sibling xattr_*_test.go files were already gated, but these two compiled everywhere, so any workflow running 'go test ./test/cli/...' on Windows hit 'undefined: syscall.Truncate'. Use the same '(linux || darwin || freebsd) && !nofuse' constraint that the fuse/ packages already use so platform gating is consistent. * fix(test): run install/revert sequentially TestUpdateInstall and TestUpdateRevert write a copy of the ipfs binary and then exec it. When other tests run in parallel, a concurrent fork() can inherit the still-open write fd into its child, leaving the freshly written file 'text file busy' for exec until the sibling child execs. Dropping t.Parallel() on these two tests ensures no other goroutine is mid-fork while the binary is being written, which is the only reliable way to avoid the ETXTBSY race without clever fd tricks. * ci(update): use cloudflare/google DNS on macos GitHub's macOS runners intermittently lose DNS for api.github.com, which fails the real-network subtests in TestUpdate. Point the resolver at 1.1.1.1 and 8.8.8.8 on every active network service and flush the DNS cache before running the update tests. * fix(update): fsync before close in atomicfile and stash * fix(update): use unique temp file in permission fallback The previous fallback wrote to a predictable path (/tmp/ipfs-<ver>), which on shared systems lets a local attacker pre-create the path as a symlink and steer the user's subsequent 'sudo mv' anywhere. Switch to os.CreateTemp so the path is unique and exclusively owned by this process. * refactor(update): rename test env vars to TEST_KUBO_* IPFS_VERSION_FAKE and KUBO_UPDATE_GITHUB_URL are test-only escape hatches with no production use case. The TEST_ prefix signals this clearly and reduces the chance of accidental use in production. - IPFS_VERSION_FAKE -> TEST_KUBO_VERSION - KUBO_UPDATE_GITHUB_URL -> TEST_KUBO_UPDATE_GITHUB_URL * style(update): unshadow err in stashBinary * fix(update): warn when IPFS path can't be resolved silently skipping the daemon lock check on path-resolution failure can mask a misconfigured IPFS_PATH; print a warning so the user notices before the install proceeds. * fix(update): revert atomicfile Sync, use errors.Is for EOF Revert the Sync() addition in atomicfile.Close() to avoid widening the failure surface for existing migration callers that panic on Close errors (Must(out.Close()) in WithBackup). The stashBinary fsync in update.go is kept since that code path is new. - revert repo/fsrepo/migrations/atomicfile/atomicfile.go to master - use errors.Is(err, io.EOF) in extractFromTarGz | 4 个月前 | |
feat: add built-in `ipfs update` command (#11203) * feat: add built-in `ipfs update` command adds `ipfs update` command tree that downloads pre-built Kubo binaries from GitHub Releases, verifies SHA-512 checksums, and replaces the running binary in place. subcommands: - `ipfs update check` -- query GitHub for newer versions - `ipfs update versions` -- list available releases - `ipfs update install [version]` -- download, verify, backup, and atomically replace the current binary - `ipfs update revert` -- restore the previously backed up binary from `$IPFS_PATH/old-bin/` read-only subcommands (check, versions) work while the daemon is running. install and revert require the daemon to be stopped first. design decisions: - uses GitHub Releases API instead of dist.ipfs.tech because GitHub is harder to censor in regions that block IPFS infrastructure - honors GITHUB_TOKEN/GH_TOKEN to avoid unauthenticated rate limits - backs up the current binary before replacing, with permission-error fallback that saves to a temp dir with manual `sudo mv` instructions - `KUBO_UPDATE_GITHUB_URL` env var redirects API calls for integration testing; `IPFS_VERSION_FAKE` overrides the reported version - unit tests use mock HTTP servers and the var override; CLI tests use the env vars with a temp binary copy so the real build is never touched resolves https://github.com/ipfs/kubo/issues/10937 * fix(update): harden download and extraction - cap decompressed binary at 1 GB to block zip/tar bombs - propagate tar.gz/zip errors instead of swallowing them - fall back to 1h context timeout when --timeout is not set - warn on stderr when daemon lock check fails - clarify that fetch+verify+extract complete before touching binary * fix(update): resolve binary path on windows The test harness hardcodes the binary path as `cmd/ipfs/ipfs` without the `.exe` suffix. On Windows the built binary is `ipfs.exe`, so copyBuiltBinary needs to append the extension. * fix(update): handle windows binary locking in install test On Windows the OS locks the running executable, so atomicfile cannot rename over it. The install command falls back to saving the new binary to a temp path. Accept both outcomes in TestUpdateInstall: in-place replacement (Unix) or permission-denied fallback (Windows). Also fix stash path to include .exe suffix on Windows. - test/cli/update_test.go: branch on runtime.GOOS for install assertions - test/sharness/t0063-external.sh: remove, tested the old ExternalBinary delegation which is replaced by the built-in update command - .github/workflows/test-migrations.yml: pass GITHUB_TOKEN to avoid rate limits * fix(test): handle windows EINVAL on process signal after wait On Windows, Process.Wait() sets the handle state to "released" rather than "done", so a subsequent Signal() returns syscall.EINVAL instead of os.ErrProcessDone. This caused StopDaemon cleanup to panic on Windows CI. Treat both errors as "process already exited". * feat(update): add 'clean' subcommand Drops every backed-up Kubo binary from $IPFS_PATH/old-bin/ so users can reclaim disk space without hand-deleting files. Safe with the daemon running, only touches the backup directory. - update.go: extract stashDirName const, factor out listStashes() helper, add updateCleanCmd - commands_test.go: register /update/clean - test/cli/update_test.go: TestUpdateClean covers removal, empty dir, json output, and preservation of unrelated files * docs(changelog): tighten ipfs update entry Drop the marketing opener, the duplicate install example, and the revert/versions sentence; all are covered by 'ipfs update --help'. Mention the new 'clean' subcommand in the trailing pointer. * fix(test): skip fuse cli tests on non-unix Both fuse_test.go and realworld_test.go rely on Unix-only APIs (syscall.Truncate, POSIX tools). The sibling xattr_*_test.go files were already gated, but these two compiled everywhere, so any workflow running 'go test ./test/cli/...' on Windows hit 'undefined: syscall.Truncate'. Use the same '(linux || darwin || freebsd) && !nofuse' constraint that the fuse/ packages already use so platform gating is consistent. * fix(test): run install/revert sequentially TestUpdateInstall and TestUpdateRevert write a copy of the ipfs binary and then exec it. When other tests run in parallel, a concurrent fork() can inherit the still-open write fd into its child, leaving the freshly written file 'text file busy' for exec until the sibling child execs. Dropping t.Parallel() on these two tests ensures no other goroutine is mid-fork while the binary is being written, which is the only reliable way to avoid the ETXTBSY race without clever fd tricks. * ci(update): use cloudflare/google DNS on macos GitHub's macOS runners intermittently lose DNS for api.github.com, which fails the real-network subtests in TestUpdate. Point the resolver at 1.1.1.1 and 8.8.8.8 on every active network service and flush the DNS cache before running the update tests. * fix(update): fsync before close in atomicfile and stash * fix(update): use unique temp file in permission fallback The previous fallback wrote to a predictable path (/tmp/ipfs-<ver>), which on shared systems lets a local attacker pre-create the path as a symlink and steer the user's subsequent 'sudo mv' anywhere. Switch to os.CreateTemp so the path is unique and exclusively owned by this process. * refactor(update): rename test env vars to TEST_KUBO_* IPFS_VERSION_FAKE and KUBO_UPDATE_GITHUB_URL are test-only escape hatches with no production use case. The TEST_ prefix signals this clearly and reduces the chance of accidental use in production. - IPFS_VERSION_FAKE -> TEST_KUBO_VERSION - KUBO_UPDATE_GITHUB_URL -> TEST_KUBO_UPDATE_GITHUB_URL * style(update): unshadow err in stashBinary * fix(update): warn when IPFS path can't be resolved silently skipping the daemon lock check on path-resolution failure can mask a misconfigured IPFS_PATH; print a warning so the user notices before the install proceeds. * fix(update): revert atomicfile Sync, use errors.Is for EOF Revert the Sync() addition in atomicfile.Close() to avoid widening the failure surface for existing migration callers that panic on Close errors (Must(out.Close()) in WithBackup). The stashBinary fsync in update.go is kept since that code path is new. - revert repo/fsrepo/migrations/atomicfile/atomicfile.go to master - use errors.Is(err, io.EOF) in extractFromTarGz | 4 个月前 | |
feat: add built-in `ipfs update` command (#11203) * feat: add built-in `ipfs update` command adds `ipfs update` command tree that downloads pre-built Kubo binaries from GitHub Releases, verifies SHA-512 checksums, and replaces the running binary in place. subcommands: - `ipfs update check` -- query GitHub for newer versions - `ipfs update versions` -- list available releases - `ipfs update install [version]` -- download, verify, backup, and atomically replace the current binary - `ipfs update revert` -- restore the previously backed up binary from `$IPFS_PATH/old-bin/` read-only subcommands (check, versions) work while the daemon is running. install and revert require the daemon to be stopped first. design decisions: - uses GitHub Releases API instead of dist.ipfs.tech because GitHub is harder to censor in regions that block IPFS infrastructure - honors GITHUB_TOKEN/GH_TOKEN to avoid unauthenticated rate limits - backs up the current binary before replacing, with permission-error fallback that saves to a temp dir with manual `sudo mv` instructions - `KUBO_UPDATE_GITHUB_URL` env var redirects API calls for integration testing; `IPFS_VERSION_FAKE` overrides the reported version - unit tests use mock HTTP servers and the var override; CLI tests use the env vars with a temp binary copy so the real build is never touched resolves https://github.com/ipfs/kubo/issues/10937 * fix(update): harden download and extraction - cap decompressed binary at 1 GB to block zip/tar bombs - propagate tar.gz/zip errors instead of swallowing them - fall back to 1h context timeout when --timeout is not set - warn on stderr when daemon lock check fails - clarify that fetch+verify+extract complete before touching binary * fix(update): resolve binary path on windows The test harness hardcodes the binary path as `cmd/ipfs/ipfs` without the `.exe` suffix. On Windows the built binary is `ipfs.exe`, so copyBuiltBinary needs to append the extension. * fix(update): handle windows binary locking in install test On Windows the OS locks the running executable, so atomicfile cannot rename over it. The install command falls back to saving the new binary to a temp path. Accept both outcomes in TestUpdateInstall: in-place replacement (Unix) or permission-denied fallback (Windows). Also fix stash path to include .exe suffix on Windows. - test/cli/update_test.go: branch on runtime.GOOS for install assertions - test/sharness/t0063-external.sh: remove, tested the old ExternalBinary delegation which is replaced by the built-in update command - .github/workflows/test-migrations.yml: pass GITHUB_TOKEN to avoid rate limits * fix(test): handle windows EINVAL on process signal after wait On Windows, Process.Wait() sets the handle state to "released" rather than "done", so a subsequent Signal() returns syscall.EINVAL instead of os.ErrProcessDone. This caused StopDaemon cleanup to panic on Windows CI. Treat both errors as "process already exited". * feat(update): add 'clean' subcommand Drops every backed-up Kubo binary from $IPFS_PATH/old-bin/ so users can reclaim disk space without hand-deleting files. Safe with the daemon running, only touches the backup directory. - update.go: extract stashDirName const, factor out listStashes() helper, add updateCleanCmd - commands_test.go: register /update/clean - test/cli/update_test.go: TestUpdateClean covers removal, empty dir, json output, and preservation of unrelated files * docs(changelog): tighten ipfs update entry Drop the marketing opener, the duplicate install example, and the revert/versions sentence; all are covered by 'ipfs update --help'. Mention the new 'clean' subcommand in the trailing pointer. * fix(test): skip fuse cli tests on non-unix Both fuse_test.go and realworld_test.go rely on Unix-only APIs (syscall.Truncate, POSIX tools). The sibling xattr_*_test.go files were already gated, but these two compiled everywhere, so any workflow running 'go test ./test/cli/...' on Windows hit 'undefined: syscall.Truncate'. Use the same '(linux || darwin || freebsd) && !nofuse' constraint that the fuse/ packages already use so platform gating is consistent. * fix(test): run install/revert sequentially TestUpdateInstall and TestUpdateRevert write a copy of the ipfs binary and then exec it. When other tests run in parallel, a concurrent fork() can inherit the still-open write fd into its child, leaving the freshly written file 'text file busy' for exec until the sibling child execs. Dropping t.Parallel() on these two tests ensures no other goroutine is mid-fork while the binary is being written, which is the only reliable way to avoid the ETXTBSY race without clever fd tricks. * ci(update): use cloudflare/google DNS on macos GitHub's macOS runners intermittently lose DNS for api.github.com, which fails the real-network subtests in TestUpdate. Point the resolver at 1.1.1.1 and 8.8.8.8 on every active network service and flush the DNS cache before running the update tests. * fix(update): fsync before close in atomicfile and stash * fix(update): use unique temp file in permission fallback The previous fallback wrote to a predictable path (/tmp/ipfs-<ver>), which on shared systems lets a local attacker pre-create the path as a symlink and steer the user's subsequent 'sudo mv' anywhere. Switch to os.CreateTemp so the path is unique and exclusively owned by this process. * refactor(update): rename test env vars to TEST_KUBO_* IPFS_VERSION_FAKE and KUBO_UPDATE_GITHUB_URL are test-only escape hatches with no production use case. The TEST_ prefix signals this clearly and reduces the chance of accidental use in production. - IPFS_VERSION_FAKE -> TEST_KUBO_VERSION - KUBO_UPDATE_GITHUB_URL -> TEST_KUBO_UPDATE_GITHUB_URL * style(update): unshadow err in stashBinary * fix(update): warn when IPFS path can't be resolved silently skipping the daemon lock check on path-resolution failure can mask a misconfigured IPFS_PATH; print a warning so the user notices before the install proceeds. * fix(update): revert atomicfile Sync, use errors.Is for EOF Revert the Sync() addition in atomicfile.Close() to avoid widening the failure surface for existing migration callers that panic on Close errors (Must(out.Close()) in WithBackup). The stashBinary fsync in update.go is kept since that code path is new. - revert repo/fsrepo/migrations/atomicfile/atomicfile.go to master - use errors.Is(err, io.EOF) in extractFromTarGz | 4 个月前 | |
feat(cli/rpc/add): fast provide of root CID (#11046) * feat: fast provide * Check error from provideRoot * do not provide if nil router * fix(commands): prevent panic from typed nil DHTClient interface Fixes panic when ipfsNode.DHTClient is a non-nil interface containing a nil pointer value (typed nil). This happened when Routing.Type=delegated or when using HTTP-only routing without DHT. The panic occurred because: - Go interfaces can be non-nil while containing nil pointer values - Simple `if DHTClient == nil` checks pass, but calling methods panics - Example: `(*ddht.DHT)(nil)` stored in interface passes nil check Solution: - Add HasActiveDHTClient() method to check both interface and concrete value - Update all 7 call sites to use proper check before DHT operations - Rename provideRoot → provideCIDSync for clarity - Add structured logging with "fast-provide" prefix for easier filtering - Add tests covering nil cases and valid DHT configurations Fixes: https://github.com/ipfs/kubo/pull/11046#issuecomment-3525313349 * feat(add): split fast-provide into two flags for async/sync control Renames --fast-provide to --fast-provide-root and adds --fast-provide-wait to give users control over synchronous vs asynchronous providing behavior. Changes: - --fast-provide-root (default: true): enables immediate root CID providing - --fast-provide-wait (default: false): controls whether to block until complete - Default behavior: async provide (fast, non-blocking) - Opt-in: --fast-provide-wait for guaranteed discoverability (slower, blocking) - Can disable with --fast-provide-root=false to rely on background reproviding Implementation: - Async mode: launches goroutine with detached context for fire-and-forget - Added 10 second timeout to prevent hanging on network issues - Timeout aligns with other kubo operations (ping, DNS resolve, p2p) - Sufficient for DHT with sweep provider or accelerated client - Sync mode: blocks on provideCIDSync until completion (uses req.Context) - Improved structured logging with "fast-provide-root:" prefix - Removed redundant "root CID" from messages (already in prefix) - Clear async/sync distinction in log messages - Added FAST PROVIDE OPTIMIZATION section to ipfs add --help explaining: - The problem: background queue takes time, content not immediately discoverable - The solution: extra immediate announcement of just the root CID - The benefit: peers can find content right away while queue handles rest - Usage: async by default, --fast-provide-wait for guaranteed completion Changelog: - Added highlight section for fast root CID providing feature - Updated TOC and overview - Included usage examples with clear comments explaining each mode - Emphasized this is extra announcement independent of background queue The feature works best with sweep provider and accelerated DHT client where provide operations are significantly faster. * fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * Update core/commands/add.go --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 6 个月前 | ||
| 3 年前 | ||
| 2 个月前 | ||
| 4 个月前 | ||
| 2 个月前 | ||
| 1 年前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 4 个月前 | ||
| 1 年前 | ||
| 3 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 1 年前 | ||
| 2 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 2 年前 | ||
| 3 个月前 | ||
| 3 年前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 6 个月前 | ||
| 7 年前 | ||
| 6 个月前 | ||
| 26 天前 | ||
| 26 天前 | ||
| 1 年前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 7 年前 | ||
| 3 年前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 2 个月前 | ||
| 7 个月前 | ||
| 4 个月前 | ||
| 1 个月前 | ||
| 4 个月前 | ||
| 2 个月前 | ||
| 4 个月前 | ||
| 2 年前 | ||
| 2 个月前 | ||
| 4 年前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 1 个月前 | ||
| 6 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 9 个月前 |