| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
refactor: apply go fix modernizers from Go 1.26 (#11190) * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement) | 6 个月前 | |
feat: support optional pin names (#10261) | 2 年前 | |
chore: bump go-libp2p-kad-dht to v0.42 (#11398) * chore: bump go-libp2p-kad-dht to v0.42.0 * bump kad-dht * fix(init): stop publishing an empty-directory IPNS record * bump kad-dht to v0.42.1 * bump boxo * chore: bump boxo, clarify IPNS storage changelog Bumps boxo, bringing the offline router retention change (records are served until their EOL, no receive-age cap by default; ipfs/boxo#1189) and a gateway 304 cache-freshness fix (ipfs/boxo#1188). The v0.43 "Unified IPNS record storage" highlight now spells out how retention works: with the offline router and the DHT sharing the /dht datastore prefix, online nodes drop value records 48h after storage (the DHT's value-store GC) while offline nodes keep them until EOL. * fix: sync DHT purge once after deletion * bump boxo * depend on boxo@master * chore: bump boxo to main, align module pins Pick up boxo main HEAD (4794174d), which includes the merged offline router value-store change (ipfs/boxo#1189). This brings the root module, the kubo-as-a-library example, and test/dependencies onto one boxo pin; they previously pointed at three different commits. - go-libp2p moves to ec408fcc as a transitive floor required by boxo - changelog: note the one-time full-datastore key scan on first start after upgrading, and update the pinned boxo and go-libp2p hashes * chore: require make mod_tidy before commit/push Spell out that the repo has three go.mod files that must stay on the same dependency versions, and that a bare `go mod tidy` only tidies one module and lets pins drift between them. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 个月前 | |
feat: support optional pin names (#10261) | 2 年前 | |
fix(key): restore secp256k1 keygen and add PEM PKCS8 import/export (#11387) * fix(key): restore secp256k1 keygen and add PEM PKCS8 import/export * fix(key): validate --size for fixed-size key types ed25519 and secp256k1 keys have a single valid size, so a --size (--bits for init) that does not match it is now an error instead of being accepted and ignored. core/coreapi Generate and config.CreateIdentity share a new options.CheckKeySize helper, so key gen, key rotate, and init accept --size only when it equals the fixed 256 bits. RSA keeps its variable size. * test(cli): cover key lifecycle for all key types Adds end-to-end CLI coverage of the key commands (gen, list, export, import, rename, rm, rotate) for rsa, ed25519, and secp256k1. This mirrors the sharness keystore and rotate suites and extends them to secp256k1, which they never exercised. - OpenSSL fixtures under testdata/ pin byte-identical PKCS#8 export and import in both directions - rotate checks the previous identity survives, usable, under the backup name - reserved-name ('self') and restricted-type imports assert the specific refusal, not just a non-zero exit * chore(deps): note secp256k1 version alignment go-libp2p/core/crypto's key types alias this package, so the direct and transitive pins must stay on one version to avoid two copies in the build. The go.mod comment flags that for future dependency bumps. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> | 2 个月前 | |
feat(cli): accept native ipfs:// and ipns:// URIs (#11375) * feat: accept native ipfs:// and ipns:// URIs Commands that take a content path or CID now also accept native IPFS URIs (ipfs://cid, ipns://name, and the schemeless ipfs:/ipns: forms), so a URI copied from a browser or another tool works as-is. - cmdutils: PathOrCidPath parses via boxo NewPathFromURI; new CidFromArg for raw-CID commands takes the root CID and rejects sub-paths and mutable IPNS. - files: cp/stat sources and getNodeFromPath accept URIs and content paths; chroot takes its CID via CidFromArg. - resolve and name resolve normalize URIs before the namespace checks; name resolve stays IPNS-only. - routing, provide, filestore, pin remote: raw-CID args via CidFromArg. Depends on boxo NewPathFromURI (ipfs/boxo#1182); go.mod pins the PR commit until it is released. * depend on boxo@main * test: fix telemetry opt-out assertions #11374 made telemetry opt-in and rewrote the explicit "off" mode to no longer log "telemetry disabled via opt-out", but the opt-out subtests still assert that string, so TestTelemetry is red on master. Assert the "telemetry collection skipped: opted out" message the daemon emits whenever telemetry is off. * ci: inject .aegir.js for helia interop @helia/interop v11.0.0+ ships without .aegir.js (ipfs/helia#1049), so aegir test finds no specs and the interop job fails. Inject a minimal config pointing at the prebuilt dist specs when it is missing. Helia's own .aegir.js can't be reused as-is: it globs source .ts specs that Node won't run from node_modules. The same omission regressed before (ipfs/helia#1001, fixed by ipfs/helia#1003); see the comment. * ci: force mocha exit after helia interop run The node interop specs leave kubo daemon and libp2p handles open, so mocha prints "N passing" and then hangs until the job timeout instead of exiting. Pass --exit so mocha quits once the run completes. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> | 2 个月前 | |
fix(rpc): validate UnixFS in `object patch` (#11248) * fix(object): validate UnixFS type in patch add-link Reject adding named links to non-directory nodes in `object patch add-link`, which previously produced invalid DAGs silently. - reject UnixFS File/Symlink/etc nodes (only Directory and HAMTShard support named links per the UnixFS spec) - reject non-UnixFS dag-pb nodes (no UnixFS metadata to validate) - add `--allow-non-unixfs` flag to bypass both checks - pass `allow-non-unixfs` in client/rpc when SkipUnixFSValidation is set - test all three node types: bare dag-pb, UnixFS File, UnixFS Directory - reproduce the exact data-loss scenario from #7190 Fixes: https://github.com/ipfs/kubo/issues/7190 * fix(object): reject HAMTShard in patch add-link dagutils.Editor operates at the dag-pb level and does not update HAMT bitfields, so mutating HAMTShard nodes produces corrupt DAGs. - reject HAMTShard in add-link (was incorrectly allowed) - update help text to note dag-pb limitations and suggest ipfs files - add HAMT test cases to sharness and API tests - expect full error strings in all validation tests - update changelog to cover all rejected node types * fix(object): validate UnixFS type in patch rm-link Same issue as add-link: dagutils.Editor operates at the dag-pb level and cannot update UnixFS metadata, so mutating non-Directory nodes produces corrupt DAGs. - add UnixFS validation to rm-link (Directory allowed, all else rejected) - add --allow-non-unixfs flag to rm-link command - add ObjectRmLinkSettings/ObjectRmLinkOption types - update ObjectAPI.RmLink interface to accept options - pass allow-non-unixfs in client/rpc - update rm-link help text to note dag-pb limitations - add rm-link validation tests for all four node types | 5 个月前 | |
chore: clean migration | 2 年前 | |
fix: Provide according to Reprovider.Strategy (#10886) * Provide according to strategy Updates boxo to a version with the changes from https://github.com/ipfs/boxo/pull/976, which decentralize the providing responsibilities (from a central providing.Exchange to blockstore, pinner, mfs). The changes consist in initializing the Pinner, MFS and the blockstore with the provider.System, which is created first. Since the provider.System is created first, the reproviding KeyChanFunc is set later when we can create it once we have the Pinner, MFS and the blockstore. Some additional work applies to the Add() workflow. Normally, blocks would get provided at the Blockstore or the Pinner, but when adding blocks AND a "pinned" strategy is used, the blockstore does not provide, and the pinner does not traverse the DAG (and thus doesn't provide either), so we need to provide directly from the Adder. This is resolved by wrapping the DAGService in a "providingDAGService" which provides every added block, when using the "pinned" strategy. `ipfs --offline add` when the ONLINE daemon is running will now announce blocks per the chosen strategy, where before it did not announce them. This is documented in the changelog. A couple of releases ago, adding with `ipfs --offline add` was faster, but this is no longer the case so we are not incurring in any penalties by sticking to the fact that the daemon is online and has a providing strategy that we follow. Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> | 1 年前 | |
chore: clean migration | 2 年前 | |
feat: opt-in new Sweep provide system (#10834) * reprovide sweep draft * update reprovider dep * go mod tidy * fix provider type * change router type * dual reprovider * revert to provider.System * back to start * SweepingReprovider test * fix nil pointer deref * noop provider for nil dht * disabled initial network estimation * another iteration * suppress missing self addrs err * silence empty rt err on lan dht * comments * new attempt at integrating * reverting changes in core/node/libp2p/routing.go * removing SweepingProvider * make reprovider optional * add noop reprovider * update KeyChanFunc type alias * restore boxo KeyChanFunc * fix missing KeyChanFunc * test(sharness): PARALLEL=1 and timeout 30m running sequentially to see where timeout occurs * initialize MHStore * revert workflow debug * config * config docs * merged IpfsNode provider and reprovider * move Provider interface to from kad-dht to node * moved Provider interface from kad-dht to kubo/core/node * mod_tidy * Add Clear to Provider interface * use latest kad-dht commit * make linter happy * updated boxo provide interface * boxo PR fix * using latest kad-dht commit * use latest boxo release * fix fx * fx cyclic deps * fix merge issues * extended tests * don't provide LAN DHT * docs * restore dual dht provider * don't start provider before it is online * address linter * dual/provider fix * add delay in provider tests for dht bootstrap * add OfflineDelay parameter to config * remove increase number of workers in test * improved keystore gc process * fix: replace incorrect logger import in coreapi replaced github.com/labstack/gommon/log with the standard github.com/ipfs/go-log/v2 logger used throughout kubo. removed unused labstack dependency from go.mod files. * fix: remove duplicate WithDefault call in provider config * fix: use correct option method for burst workers * fix: improve error messages for experimental sweeping provider updated error messages to clearly indicate when commands are unavailable due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true * docs: remove obsolete KeyStoreGCInterval config removed from config.md as option no longer exists (removed in b540fba1a) updated keystore description to reflect gc happens at reprovide interval * docs: add TODO placeholder changelog for experimental sweeping DHT provider using v0.38-TODO.md name to avoid merge conflicts with master branch and allow CI tests to run. will be renamed to v0.38.md once config migration is added to the PR * fix: provideKeysRec go routine * clear keystore on close * fix: datastore prefix * fix: improve error handling in provideKeysRec - close errCh channel to distinguish between nil and pending errors - check for pending errors when provided.New closes - handle context cancellation during error send - prevent race condition where errors could be silently lost this ensures DAG walk errors are always propagated correctly * address gammazero's review * rename BurstProvider to LegacyProvider * use latest provider/keystore * boxo: make mfs StartProviding async * bump boxo * chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c - updated boxo dependency to specified commit - updated go.mod and go.sum files across all modules * use latest kad-dht/boxo * Buffered SweepingProvider wrapper * use latest kad-dht commit * allow no DHT router * use latest kad-dht & boxo --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> | 11 个月前 | |
chore: clean migration | 2 年前 | |
feat(provide): +unique and +entities strategy modifiers (#11245) * fix(config): harden provide strategy parsing with error returns - config: ParseProvideStrategy returns error, rejects "all" mixed with selective strategies, removes dead strategy==0 check - config: add MustParseProvideStrategy for pre-validated call sites - config: ValidateProvideConfig validates strategy at startup - config: ShouldProvideForStrategy uses bitmask check for ProvideStrategyAll - core/node: downstream callers use MustParseProvideStrategy - core/node: fix Pinning() nil return that caused fx.Provide panic * feat(config): add +unique and +entities strategy modifiers - ProvideStrategyUnique: bloom filter cross-DAG deduplication - ProvideStrategyEntities: entity-aware traversal (implies Unique) - parser: "unique" and "entities" tokens recognized - validation: modifiers must combine with pinned/mfs, incompatible with all/roots - go.mod: update boxo to feat/provide-entity-roots-with-dedup (VisitedTracker, WalkDAG, WalkEntityRoots, NewConcatProvider, NewUniquePinnedProvider, NewPinnedEntityRootsProvider) * refactor(cmd): rename ExecuteFastProvide to ExecuteFastProvideRoot pure rename, no behavior change. prepares for ExecuteFastProvideDAG which will walk the DAG according to Provide.Strategy. * feat(pin): fast-provide root CID after pin add and pin update adds ExecuteFastProvideRoot calls to pin add and pin update, matching the behavior of ipfs add and ipfs dag import. respects Import.FastProvideRoot and Import.FastProvideWait config options. previously, pin add/update did not trigger any immediate providing, leaving pinned content invisible to the DHT until the next reprovide cycle (up to 22h). * feat(provider): wire +unique reprovide cycle with bloom dedup when Provide.Strategy includes +unique, the reprovide cycle uses a shared BloomTracker across all sub-walks (MFS, recursive pins, direct pins). duplicate sub-DAG branches across recursive pins are detected and skipped, reducing traversal from O(pins * total_blocks) to O(unique_blocks). - readLastUniqueCount / persistUniqueCount: persist bloom sizing count between cycles at /reprovideLastUniqueCount - uniqueMFSProvider: MFS walker with shared tracker + locality check - createKeyProvider restructured: +unique bit checked first, non-unique strategies fall through to existing switch unchanged - per-cycle fresh BloomTracker sized from previous cycle's count - channel wrapper persists count on successful cycle completion * feat(provider): wire +entities reprovide cycle with entity root walkers when Provide.Strategy includes +entities (which implies +unique), the reprovide cycle uses WalkEntityRoots instead of WalkDAG, emitting only entity roots (files, directories, HAMT shards) and skipping internal file chunks. - mfsEntityRootsProvider: MFS walk with entity root detection - createKeyProvider: select walker based on +entities flag via function references (makePinProv / makeMFSProv) to avoid duplicating the stream wiring logic - all combinations: pinned+entities, mfs+entities, pinned+mfs+entities * docs: document +unique and +entities strategy modifiers - config.md: document +unique, +entities modifiers with caveats (range request limitation, roots vs entities distinction) - changelog v0.41: add entries for strategy modifiers, pin add/update fast-provide, and hardened strategy parsing * feat: gate providingDagService behind --fast-provide-dag per-block providing during ipfs add is now opt-in via --fast-provide-dag (or Import.FastProvideDAG config, default: false). without it, only the root CID is fast-provided after add, and the reprovide cycle handles the rest. this changes the default for Provide.Strategy=pinned: previously every block was provided during write, now only the root is immediate. use --fast-provide-dag=true to restore the previous behavior. Provide.Strategy=all is unaffected (blockstore hook provides on Put). * feat(pin): expose --fast-provide-root and --fast-provide-wait flags pin add and pin update now accept the same --fast-provide-root and --fast-provide-wait CLI flags as ipfs add and ipfs dag import, with the same config fallbacks (Import.FastProvideRoot, Import.FastProvideWait). previously these were config-only with no CLI override. * feat: wire --fast-provide-dag across all content commands --fast-provide-dag now available on ipfs add, ipfs dag import, ipfs pin add, and ipfs pin update (matching --fast-provide-root). - ExecuteFastProvideDAG accepts []cid.Cid so multiple roots share one bloom tracker (cross-root dedup for dag import and pin add) - --fast-provide-dag supersedes --fast-provide-root (DAG walk includes the root CID as the first emitted via DFS pre-order) - wait parameter: when true blocks until walk completes, when false runs in background goroutine - Import.FastProvideDAG config option (default: false) * docs(config): improve Provide.Strategy docs, add Import.FastProvideDAG - strategy section: clearer trade-offs, suggested configurations, memory comparison with concrete numbers - Import.FastProvideDAG: new config option documentation - Import.FastProvideRoot/Wait: updated to mention pin commands - all three Import.FastProvide* options: consistent "Applies to" lists * chore: gofumpt and gci formatting * chore: update boxo to latest feat/provide-entity-roots-with-dedup * feat: TEST_DHT_STUB with ephemeral DHT peers when TEST_DHT_STUB=1, the CLI test harness creates 20 in-process libp2p hosts on loopback, each running a DHT server with a shared in-memory ProviderStore. kubo daemons bootstrap to them over real TCP, exercising the full DHT code path without public internet. tests opt in via h.SetStubBootstrap(nodes) after Init(). on the daemon side, WAN DHT filters (AddressFilter, QueryFilter, RoutingTableFilter, RoutingTablePeerDiversityFilter) are lifted to accept loopback peers when TEST_DHT_STUB is set. depends on: github.com/libp2p/go-libp2p-kad-dht#1241 * test: harden provider strategy tests add sweep reprovide tests for all strategies (all, pinned, roots, mfs, pinned+mfs). each test waits for two reprovide cycles to confirm the schedule runs repeatedly. sweep uses short Provide.DHT.Interval and polls provide stat --enc=json. harden negative assertions: - roots: test excludes child blocks of a recursive pin (not just unpinned content), using --only-hash to learn the child CID - mfs: test that pinned content outside MFS is not provided fix: ipfs add --only-hash no longer triggers fast-provide or pinning (was providing CIDs for data that was never stored) rename SetStubBootstrap to BootstrapWithStubDHT with lazy-init (ephemeral peers created on first call, not on harness creation) * test: add +unique and +entities strategy tests strategy tests for pinned+mfs+unique and pinned+mfs+entities, covering both provide-at-add-time and reprovide (two cycles). content uses a nested DAG (root/subdir/largefile with 1 MiB chunks) to exercise the walker on multi-level structures. BootstrapWithStubDHT is now self-contained: it always creates 20 ephemeral DHT peers on loopback and sets TEST_DHT_STUB=1 on each node's environment so the daemon lifts WAN DHT filters. no external env var needed. the sweep provider requires >=20 DHT peers to estimate network size (prefix length); without enough peers it stays offline and never provides. TEST_DHT_STUB on the daemon side lifts WAN DHT filters (AddressFilter, QueryFilter, RoutingTableFilter, RoutingTablePeerDiversityFilter) to accept loopback peers. this is set automatically by BootstrapWithStubDHT. other changes: - Provide.DHT.Interval=30s in sweep reprovide tests (was 1m) - uniq() helper for unique CIDs across parallel subtests - ipfs add --only-hash disables fast-provide and pinning * docs: improve help text and changelog accuracy ipfs add --help: rewrite fast-provide section with clear structure (content discoverability, flag defaults, strategy=all behavior) ipfs routing reprovide: mark as deprecated, note it returns an error with sweep provider, log error with actionable guidance changelog: fix missing --fast-provide-dag flag on pin commands, use "routing system" instead of "DHT" where applicable, link to docs/config.md as source of truth for defaults environment-variables.md: note that BootstrapWithStubDHT sets TEST_DHT_STUB automatically, no external env var needed * chore: revert go-libp2p-kad-dht to released v0.39.0 the fork (NoopMessageSender, MsgSenderBuilder) is no longer used. the ephemeral peer pool in BootstrapWithStubDHT replaced the NoopMessageSender approach. * feat: log bloom dedup stats after provide cycles log providedCIDs and skippedBranches after each unique reprovide cycle and fast-provide-dag walk. tests verify exact counts with two dir pins sharing a 10 KiB file (5 KiB chunks): fast-provide-dag asserts 5 provided + 1 skipped branch, reprovide asserts 6 provided + 1 skipped branch (includes empty MFS root pin). both assert bloom tracker created and no autoscale. updates boxo to pick up Deduplicated() counter, bloom creation/autoscale logging, and review feedback fixes. * chore(deps): switch boxo to post-merge commit boxo#1124 landed on master; point to the merge commit instead of the PR branch. * fix(coreapi): drop providingDagService wrap ipfs add --pin --fast-provide-dag wrapped the DAGService with providingDagService, which announced every block as it was written regardless of strategy modifiers. ExecuteFastProvideDAG ran in parallel as the post-add walker. Net effect: - pinned+entities: chunks reached the DHT despite +entities saying they should be skipped (correctness bug) - pinned+unique: every block announced twice; the post-walk bloom only dedups against its own pass - pinned (plain): every block announced twice ExecuteFastProvideDAG already has bloom dedup, entity-roots support, and unbuffered backpressure, so it is now the single mechanism for --fast-provide-dag across ipfs add, dag import, pin add, and pin update. Provide.Strategy=all is untouched: every block is provided at the blockstore level via the blockstore.Provider hook in core/node/storage.go, which is independent of coreapi. The Pinned strategy bit gated providingDagService and the parser rejects combining "all" with other strategies, so "all" never set that bit in the first place. - core/coreapi/unixfs.go: drop the wrap, the providingDagService struct, and the now-unused mh and boxo/provider imports - core/coreiface/options/unixfs.go: drop FastProvideDAG option - core/coreapi/coreapi.go: drop now-dead providingStrategy field - core/commands/add.go: drop the FastProvideDAG option pass-through - test/cli/provider_test.go: regression test using ipfs add --fast-provide-dag with pinned+entities -- fails on the previous code and passes here * feat(config): add Provide.BloomFPRate Operators tuning +unique or +entities strategies on memory-constrained or extra-large repos previously had no way to trade bloom filter memory against false-positive rate -- both the reprovide cycle and fast-provide-dag walks hardcoded walker.DefaultBloomFPRate. Provide.BloomFPRate is the target false positive rate (1/N) for the shared bloom tracker. Has no effect on Provide.Strategy=all or other strategies that do not walk DAGs through the tracker. Validation rejects values below 1_000_000 (~1 in 1M); below that the bloom becomes lossy enough to drop a meaningful fraction of CIDs from each reprovide cycle. The single source of truth for the default value is config.DefaultProvideBloomFPRate; docs reference it descriptively (~1 in 4.75M, ~4 bytes/CID) so the literal lives in exactly one place. - config/provide.go: BloomFPRate field, DefaultProvideBloomFPRate and MinProvideBloomFPRate constants, validation - config/provide_test.go: round-trip + validation cases - core/node/provider.go: plumb fpRate through setReproviderKeyProvider and createKeyProvider - core/commands/cmdenv/env.go: ExecuteFastProvideDAG takes fpRate - core/commands/{add,dag/import,pin/pin}.go: resolve from cfg and pass through to ExecuteFastProvideDAG - docs/config.md: new Provide.BloomFPRate section after Provide.DHT.* with memory tradeoff table and minimum-value note - docs/changelogs/v0.41.md: link to the new option from the +unique/ +entities section * test(node): cover unique count persistence readLastUniqueCount and persistUniqueCount were exercised only indirectly via CLI tests, leaving the 8-byte length check and the "missing key" fallback without direct coverage. - empty datastore returns 0 (no previous cycle) - round trip across the full uint64 range (0, 1, 1k, 1M, 1B, MaxUint64) - overwrite returns the most recent value (matches per-cycle persist) - corrupt length (empty, short, long, single byte) returns 0 instead of panicking * fix(cmdenv): tie async fast-provide to node ctx Background fast-provide goroutines were implicitly bound to req.Context, which go-ipfs-cmds cancels on handler exit, so async --fast-provide-dag (and --fast-provide-root parented on context.Background) aborted or outlived the node. Parent both paths off the IpfsNode lifetime context instead. - ExecuteFastProvideRoot: async goroutine now derives from ipfsNode.Context(), so it cancels on daemon shutdown rather than potentially touching a closed DHT client. - ExecuteFastProvideDAG: takes cmdCtx and nodeCtx; wait=true runs inline under cmdCtx (Ctrl+C still cancels the walk), wait=false runs in a goroutine under nodeCtx so the walk survives command exit but still stops on shutdown. - add, dag import, pin add/update: pass node.Context() as the new nodeCtx argument. - changelog: note the behavior change for opt-in strategies. * test(cli): cover async fast-provide-dag walk Adds TestProviderFastProvideDAGAsyncSurvives: ipfs add with --fast-provide-dag=true but no --fast-provide-wait must walk the full DAG in a background goroutine that outlives the command handler, announce every block, and leave chunk CIDs findable by peers via findprovs. A long Provide.DHT.Interval ensures the scheduled reprovide cycle cannot be the source of the chunk announcements. * docs(changelog): tighten v0.41 provide section | 4 个月前 |