master

GGitHubfeat(start-wrt): SNI hostname routes end to end — UPnP vendor actions, StartWRT dataplane, Remote Access coexistence (#3783)

db7aa808创建于 6 天前历史提交

文件	最后提交记录	最后更新时间
draft-start9-pcp-hostname.md	chore(fmt): normalize formatting — one config per language, reproducible rustfmt (#3437) * chore(fmt): one config per language + reproducible rustfmt Our rustfmt.toml uses nightly-only options (group_imports, imports_granularity) but nothing pinned the nightly, so output drifted between contributors — and CI never even checked Rust formatting. Configs were also duplicated and partial. Consolidate to one config per language and make the output reproducible: - rustfmt: single root rustfmt.toml (was 5 duplicate copies + ~14 crates with none). Runs in start9/fmt-env — start9/cargo-zigbuild (the same image the Rust build uses) plus the pinned nightly + rustfmt component (build/fmt/fmtenv.Dockerfile) — via build/fmt/run-fmt.sh, which runs it --user so output stays host-owned. FMT_NATIVE=1 runs on the host against the same pinned toolchain (read from the Dockerfile ARG — one source of truth for the version). - prettier: single root .prettierrc.json (was 3-4 drifted copies, incl. container-runtime on double quotes). One repo-wide pass with a hardened .prettierignore (excludes .sqlx cache, the generated exver.ts parser, conformance vectors, locales, snapshots/fixtures, proxy.pac, patch-db/client). - taplo: single root taplo.toml over all TOML (was one stray copy), pinned via @taplo/cli. Only rustfmt needs the container; prettier and taplo run natively. - CI `make format-check` now covers rustfmt (previously unchecked) + taplo. - Remove the dead husky v4 / lint-staged pre-commit hook (no lint-staged config existed). Per-project `<project>-format` targets are kept (routed through the same tools/config). The formatting content itself lands in the next commit. * style: apply repo-wide formatting Mechanical output of `make format` using the configs from the previous commit (rustfmt in the pinned-nightly container, prettier and taplo). No behavior changes. * chore: ignore the repo-wide reformat in git blame	2 个月前
draft-start9-pcp-hostname.txt	refactor: reorganize start-os into all-products monorepo (#3352) * docs: propose monorepo reorganization for all Start9 products * refactor(monorepo): split core into start-core lib + thin product bin crates - core/ -> shared/crates/start-core (lib 'startos', package 'start-core') - entry points moved to product dirs: start-os (startbox+start-container), start-cli, start-registry, start-tunnel - root Cargo workspace + shared Cargo.lock; profiles hoisted to root - patch-db submodule relocated to vendor/patch-db - service files moved to their product dirs - fixed include_dir!/include_str! paths for new crate locations cargo check -p start-cli -p start-registry passes (lib compiles). * refactor(monorepo): split web into product dirs + shared/web; relocate sdk & container-runtime - angular workspace rooted at shared/web (holds shared + marketplace libs + config) - apps moved to product dirs: start-os/web/{ui,setup-wizard}, start-tunnel/web, brochure/ - angular.json roots/outputs + per-app tsconfig paths repointed (Plan A) - sdk -> start-sdk (base+package kept cohesive: package imports base via relative paths under shared rootDir; splitting base out would break those imports) - container-runtime -> start-os/container-runtime - file: deps repointed (sdk baseDist/dist, patch-db client under vendor/) * build(monorepo): rewire Makefile, build scripts & CI to new layout - build scripts target workspace (-p <crate>, ./Cargo.toml, repo-root cwd) - Makefile paths: core->shared/crates/start-core, web split across product dirs, sdk->start-sdk, container-runtime->start-os/container-runtime, patch-db->vendor - compress-uis.sh takes per-product web dir; split compress pattern rules - ts-bindings recipe sed patterns generalized for new bindings path - CI workflows repointed (deploy-brochure, start-cli, test, startos-iso, ...) * chore(monorepo): gitignore per-product web dist outputs * docs(monorepo): update proposal to reflect implemented layout + verification status * refactor(monorepo): relocate root docs/ internal notes into their projects - exver.md, VERSION_BUMP.md -> shared/crates/start-core/ - PHYSICAL_DEVICE_TEST_PLAN.md, TODO.md -> start-os/ - draft-start9-pcp-hostname.* -> start-tunnel/ frees top-level docs/ for the migrated docs site * docs(monorepo): migrate start-docs (plain copy, no history) - mdbooks into product dirs: start-os/docs, start-tunnel/docs, start-sdk/docs (packaging) - bitcoin-guides, landing, build infra (build.sh/serve.sh/theme/versions.conf/scripts) -> top-level docs/ - repoint book theme symlinks to ../../docs/theme; book.toml build-dir=book, repo/edit URLs -> monorepo - build.sh maps book names to relocated dirs (absolute output); deploy.yml runs in docs/ with paths filter - verified: docs/build.sh builds all 4 books * docs(monorepo): adopt AGENTS.md convention (CLAUDE.md -> @AGENTS.md import) * docs(root): rewrite README/ARCHITECTURE/AGENTS/CONTRIBUTING for monorepo layout * docs(start-os): add product README/ARCHITECTURE/AGENTS/CHANGELOG/CONTRIBUTING Document the StartOS OS product as a thin wrapper in the monorepo: startbox/ start-container bins, web UIs (ui + setup-wizard), container-runtime, systemd units, and OS image packaging. Reflect new paths (start-core, shared/web, start-sdk, vendor/patch-db) and root-workspace build commands. * docs: integrate merged start-docs PRs (#93 UPnP/gateway, #94 task accept/set, #96 init progress) Only PRs whose feature is confirmed merged into start-os were applied. #99 (upstreamCertValidation) skipped — its code PR (#3353) is still open. * docs: add/normalize per-project doc sets (README/ARCHITECTURE/AGENTS/[CHANGELOG]/CONTRIBUTING) Products get the full set incl CHANGELOG; shared components + docs site get the set minus CHANGELOG; all updated to reflect the monorepo layout and AGENTS.md convention. * docs: add CLAUDE.md -> @AGENTS.md import to remaining project dirs * build(monorepo): root the Angular workspace at repo root so apps resolve node_modules Angular resolves @angular/core per-project from each app's root; with apps in product dirs, shared/web/node_modules was unreachable. Move the workspace config (angular.json, package.json, lockfile, tsconfig{,.lib}.json, .browserslistrc) to the repo root — the only ancestor of every app — so resolution works. - angular.json: project roots -> product dirs, lib roots -> shared/web/{shared,marketplace} - tsconfig paths/extends repointed; app source config.json/package.json require() depths corrected for the new app locations - package.json file: deps + script paths rebased to root; check-i18n.mjs scans the scattered project dirs; update-config.sh writes config.json at the workspace root - Makefile web targets run npm at root; build/env + build-cargo-dep stale paths fixed - build-cli.sh: drop stale 'cd core' in chown step Verified: full build succeeds — ts-bindings, SDK bundle, all 4 Angular UIs, and all five musl bins (startbox/registrybox/tunnelbox/start-container/start-cli). * build(monorepo): fix full-image build paths (container-runtime squashfs + version) - Makefile: undouble container-runtime.service dep path in rootfs rule - check-version.sh: read version from root package.json (moved from web/) - update-image-local.sh: mount repo root so start-sdk + target/ are visible to the image build; run start-os/container-runtime/update-image.sh - update-image.sh: copy start-container from ../../target (workspace), not ../core/target Verified: 'make all' completes (exit 0) — all musl bins + container-runtime rootfs.squashfs (437M) build; second run is a no-op (fully built). * build(monorepo): sync container-runtime package-lock to relocated SDK path; prettier ARCHITECTURE table * feat(monorepo): migrate startos-backup-fs into start-os/backup-fs Vendor the backup-fs crate (was external git dep Start9Labs/start-fs) as a workspace member under the start-os product; build it via the zigbuild path like the other bins instead of 'cargo install --git'. - start-os/backup-fs/: the startos-backup-fs crate (encrypted erasure-coded FUSE backup filesystem); relaxed its =4.5.7 clap / =0.2.17 ppv-lite86 exact pins so they unify with the workspace - root Cargo workspace member + single lock - shared/crates/start-core/build/build-backup-fs.sh; Makefile target builds the local crate (no more git URL) - docs: start-os ARCHITECTURE + CHANGELOG note the migration Verified: 'make all' (exit 0) builds startos-backup-fs (musl) as a member. * build(sdk): decouple 'bundle' from test/check-fmt so consumers don't re-run jest bundle now builds baseDist+dist only; test/check-fmt are standalone (CI calls them directly), and publish runs them explicitly. Fixes the recursive-make coupling where the OS build re-ran the full SDK jest suite every build and an SDK test/format failure broke the OS build. * build(monorepo): split Makefile into per-project include fragments Thin root Makefile includes build/common.mk (shared vars/macros + cross-cutting infra) and one <project>/build.mk per product. Uses include (not recursive make) so it stays a single DAG and cross-project prereqs (start-core -> ts-bindings -> SDK -> web/container-runtime) resolve correctly. - build/common.mk: vars, cp/mkdir/ln/ssh macros, patch-db client, external cargo tools - shared/crates/start-core/build.mk: test-core, ts-bindings - shared/web/build.mk: angular workspace (install, .angular, i18n, UI builds, compress, config.json) - start-sdk/build.mk: test-sdk, dist bundle (consumes the now-decoupled SDK Makefile) - start-{cli,registry,tunnel}/build.mk: their bins + install/deb - start-os/build.mk: startbox/start-container/backup-fs, container-runtime image, OS image assembly + deploy - docs/build.mk: docs site build Verified: make all is a no-op (full build intact); all targets resolve; no duplicate recipes. * docs(root): note the per-project build.mk Makefile structure in AGENTS.md * fix(ci): repoint test/web paths after workspace moves - run-tests.sh: cd to repo root (was shared/crates), build via ./Cargo.toml -p start-core (was ./core/Cargo.toml --workspace) - test.yaml / deploy-brochure: install the Angular workspace at the repo root (npm ci) instead of shared/web; fix vendor/vendor/patch-db doubling; brochure path filters -> root - startos-iso prevent-rebuild placeholders: node_modules/.angular at root; version read from root package.json Verified: npm ci passes at root (lockfile gate). * build(start-os): namespace OS-product make targets as startos-* / install-startos The repo is no longer start-os-only, so the generic target names now read as start-os-specific: - deb->startos-deb, iso/img->startos-$(IMAGE_TYPE), squashfs->startos-squashfs - install->install-startos (matches install-registry/install-tunnel) - wormhole*/update*/emulate-reflash/upload-ota -> startos-* - new 'startos' aggregate (= STARTOS_TARGETS); root 'all: startos' Callers updated: dpkg-build.sh INSTALL_TARGET, deploy targets' $(MAKE) install, startos-iso.yaml (make startos-iso/startos-img), root .PHONY. NOTE: external shared-workflows may invoke the old names (make iso/squashfs/install) for OS image/release builds — needs a companion update there. * build(start-os): move OS-specific build assets into start-os/build Relocate the start-os-only build inputs out of the shared top-level build/ into the product dir: image-recipe/, dpkg-deps/, lib/, download-firmware.sh, and save-migration-images.sh -> start-os/build/. Keep genuinely shared pieces at build/ (common.mk, env/, os-compat/, build-cargo-dep.sh, and lib/scripts/forward-port, which start-tunnel also installs). Relocate the start-os-specific make variables/rules out of build/common.mk into start-os/build.mk (web src/output vars -> shared/web/build.mk; registry and tunnel target vars -> their own fragments) so common.mk is shared-only. Repoint every reference (fragments, Makefile clean, container-runtime update-image.sh, and the moved scripts' own internal paths). Delete the unreferenced legacy build/registry/ eos deploy scripts. * docs(changelog): write 0.4.0-beta.10 per-product release notes Fill in the [0.4.0-beta.10] sections across the per-product CHANGELOGs (brochure, start-cli, start-os, start-registry, start-sdk, start-tunnel) with Added/Changed/Fixed/Removed/Security notes for this cycle, cross-linked between products. * refactor(start-core): rename lib startos to start_core, drop package alias Rename the start-core library from `startos` to `start_core` so the crate's lib name matches its package and the legacy `startos = { package = "start-core" }` dependency-rename alias is gone. The name now penetrates all source: - [lib] name = "start_core"; every `startos::` crate path -> `start_core::` - product crates depend on `start-core` directly; features become `start-core/*` - RUST_LOG=warn,startos=debug -> start_core=debug in the systemd units and CI (the target is module_path!()-derived, so it tracks the crate name) - docs updated to match The product identifier "startos" is left untouched (the root:startos system user/group, the tor.startos / *.startos DNS names, the nftables table, the signature context, the .startos/ packaging-workspace dir, i18n keys, and the /usr/lib/startos install paths). * refactor(monorepo): nest products under projects/, rename shared -> shared-libs Move the buildable products and the docs site into a top-level projects/ dir to separate them from repo infrastructure: start-os, start-cli, start-registry, start-tunnel, start-sdk, brochure (-> brochure-marketplace), docs (-> start-docs) -> projects/ Rename the shared Rust+web library container shared/ -> shared-libs/, kept at the top level alongside build/ and vendor/ as cross-cutting infrastructure. Rewire every path reference to the new layout: - Cargo workspace members + product path deps (../shared -> ../../shared-libs) - Makefile, build/common.mk, and every <project>/build.mk fragment - angular.json, package.json, root + per-app tsconfig (web app configs moved a level deeper, so their relative extends/paths gain one ../) - .github/workflows (the Start9Labs/start-os repo URL is preserved; docs-deploy working-directory + path triggers updated) - build scripts (run-local-build.sh / update-image-local.sh cd depths and internal paths; start-core build/*.sh chown paths) - root .gitignore build-output globs and the web package-lock file: paths Verified: cargo check of all six crates (UI-embed include_dir! and build/env include_str! resolve to the new locations), make -n of the OS / registry / tunnel / web targets. The cold web/SDK build remains CI-grade. * refactor(monorepo): relocate project-specific assets/debian/scripts into projects Apply the same shared-vs-project split to the remaining top-level dirs: - assets/ (create-vm screenshots) -> projects/start-os/assets/ - debian/{startos,start-registry,start-tunnel}/postinst -> each project's debian/; the shared debian/dpkg-build.sh stays top-level and now maps PROJECT -> projects/<dir>/debian for the control files - scripts/copy-categories.sh (registry admin) -> projects/start-registry/scripts/ Kept at top level as genuinely shared/repo-level: debian/dpkg-build.sh, scripts/manage-release.sh (repo releases), scripts/publish-deb.sh (apt publish). Repoint the deb build.mk prereqs, the CONTRIBUTING create-vm link, and code/unit comments. Verified make -n of the three *-deb targets. * docs(rfcs): move draft-start9-pcp-hostname to a top-level rfcs/ dir The PCP HOSTNAME extension Internet-Draft (.md + .txt) describes a protocol spoken by both the StartOS client and the StartTunnel server, so it belongs at the repo level rather than inside start-tunnel/. Repoint the start-os CHANGELOG reference (was the stale docs/ path) to rfcs/. * docs: sync structure docs to the projects/ layout + README project shout-outs - README: add a "rest of the monorepo" section with a short shout-out to each non-OS product (StartTunnel, start-cli, Start SDK, start-registry, and the marketplace + docs sites), and update the directory table + icon path to the projects/ + shared-libs layout. - Root AGENTS.md: rewrite "what lives where" for the new layout and complete the Sub-scopes list (it was missing most products). - Root ARCHITECTURE.md: repoint the module map + cross-layer paths; MONOREPO.md gains a note that the layout was refined (products -> projects/, shared -> shared-libs). - Per-project docs: rename shared/ -> shared-libs/ references, fix relative links whose depth changed when products moved a level deeper into projects/ (links to the repo root, LICENSE, shared-libs, and cross-product changelogs), and repoint functional cd / --prefix build commands. Sibling refs under projects/ (e.g. ../start-sdk, file:../../start-sdk/dist) are correct and left as-is. * build(brochure-marketplace): rename Angular project brochure -> brochure-marketplace Rename the Angular project key (and its build/serve targets) so the project name matches its directory. The dist output is now projects/brochure-marketplace/dist/raw/brochure-marketplace, and the deploy workflow reads/rsyncs that path — this also corrects a path the projects/ restructure had mangled to raw/projects/brochure-marketplace. The npm script names (build:brochure / start:brochure) are kept as conveniences. * docs: remove MONOREPO.md The reorganization proposal has been fully implemented and superseded by the current README/ARCHITECTURE; drop the historical proposal doc and its two links. * feat(build): per-project versioning + Debian packaging for start-cli Decouple product versions from the single StartOS release version. Each Rust product's version is now the source of truth in its own Cargo.toml: start-os stays 0.4.0-beta.10; start-cli / start-registry / start-tunnel move to their own line starting at 1.0.0. - basename.sh and dpkg-build.sh read the version straight from the project's Cargo.toml (per PROJECT), so each .deb is named/versioned independently. - check-version.sh now derives the OS-image /usr/lib/startos/VERSION.txt from the start-os crate manifest instead of the root package.json; nothing maintains a separate version source anymore. - start-cli gains a Debian package: `make cli-deb` builds the musl binary and packages it via the shared dpkg-build.sh (CLI_BASENAME / install-cli staging). CHANGELOGs and the registry AGENTS version note updated to reflect independent versioning. Cargo.lock synced to the new member versions. * chore: ignore *.local.md Broaden the local-notes ignore from CLAUDE.local.md to any *.local.md. * refactor(deps): vendor Start9-maintained crates into shared-libs/crates Move every Start9-maintained crate the workspace depends on in-repo, wired by direct path deps (no [patch]): - rpc-toolkit, imbl-value, exver, yasi, jsonpath (jsonpath_lib), pi-beep — plain-copied from their repos into shared-libs/crates/, added as workspace members. Their inter-deps are repointed to path (exver/imbl-value -> yasi, rpc-toolkit/jsonpath -> imbl-value), and start-core depends on them by path. - patch-db — de-submoduled: moved out of the vendor/ git submodule into shared-libs/crates/patch-db (keeps its own [workspace], excluded from the root one and consumed by start-core via path). Its core/json-patch/json-ptr now path-dep the vendored imbl-value, so there is a single imbl_value::Value type. Drop .gitmodules; repoint the web patch-db-client (package.json / common.mk / CI / shared-libs/web) and pi-beep's build (build-cargo-dep.sh --path). Upstream forks still pulled by git (async-acme, crab_nat, fuser) are left as-is. Verified: cargo check of start-core + start-cli + start-registry + pi-beep compiles the whole path-dep tree clean; Cargo.lock regenerated. * refactor(start-os): move manage-release.sh into the product manage-release.sh is the StartOS release orchestration (startos-images S3 bucket/CDN, the OS image arch matrix incl. -nonfree/-nvidia, the OS registry), not a repo-wide tool — move it to projects/start-os/scripts/. It still calls the shared scripts/publish-deb.sh (which stays top-level, since it publishes any product's .deb), now referenced by its repo-root-relative path. * refactor(debian): rename dpkg-build.sh -> build.sh, move publish-deb.sh -> debian/publish.sh Co-locate the deb tooling under debian/: the package builder is debian/build.sh and the apt-repo publisher is debian/publish.sh (was scripts/publish-deb.sh, which empties scripts/). Repoint the per-product deb build.mk targets, the manage-release.sh caller, and doc/comment references. * build: build pi-beep as a first-party member; reword "vendored" -> "first-party" pi-beep is one of our crates now, so build it like startos-backup-fs (a dedicated build-pi-beep.sh zig build of the workspace member) instead of routing it through build-cargo-dep.sh. That script is now only for the genuinely external crates.io dev tools (tokio-console, flamegraph) bundled into unstable/console images. Also reword the patch-db docs: these are our own crates, so "first-party crate" is more accurate than "vendored" (which implies a third-party copy). * ci: path-gate the per-product build workflows to their project + deps The start-cli / start-registry / start-tunnel / startos-iso build workflows ran on every push/PR (only skipping doc-only changes), so all four built regardless of what changed. Replace the blanket paths-ignore with a paths: allowlist scoped to each product plus its dependencies (start-core + the in-repo shared-libs crates, Cargo manifests, build infra, and — for the web-bearing/OS workflows — the Angular workspace and SDK). workflow_dispatch / workflow_call are kept so manual and orchestrated runs still fire unconditionally. * ci: migrate shared-workflows (service-package CI) into the monorepo Bring the reusable .s9pk build/release workflows and their composite actions in-repo from the standalone Start9Labs/shared-workflows repo, so the packaging toolchain lives alongside the SDK: - .github/workflows/{build,release,tagAndRelease}.yml (reusable, workflow_call) - .github/actions/{extract-version,free-disk-space,setup-build-env, setup-publish-env,upload-each} Their internal references (and the SDK package-template's three workflows + the packaging docs) are repointed from start9labs/shared-workflows@master to Start9Labs/start-os@master. These are workflow_call-only, so they don't run for the monorepo itself — they activate once this lands on master and external service-package repos repoint their `uses:` to Start9Labs/start-os. * docs(monorepo): document tandem-update couplings The per-product CI `paths:` filters mirror each product's build.mk prerequisites by hand — nothing enforces it. Add a "Coupled changes" section to the root AGENTS.md and reciprocal pointers in each gated workflow and its build.mk, so a change to one half is caught at the other. Also catalogs the remaining hand-mirrored pairs (reusable service-package CI <-> SDK package-template <-> packaging docs; the files touched when adding a product/crate) and the already-enforced couplings (ts-bindings, the five i18n locales, the UI beta seed, version <-> CHANGELOG, docs <-> user-facing changes). * chore(manpages): generate man pages into their product projects The export_manpage_* tests in start-core wrote every product's man pages into start-core's own man/ dir. Point each generator at the owning product's man/ dir (anchored to CARGO_MANIFEST_DIR), move the committed pages there, and update build-manpage.sh's chown and the docs. start-container's pages go to projects/start-os, since that bin is part of the StartOS product. * Retitle README * refactor(shared-libs): rename web -> ts-modules Mirror the `crates/` naming: the shared TS/Angular workspace dir becomes `shared-libs/ts-modules/`. Pure path rename — repoints every reference (angular.json, root tsconfig/package.json scripts, the Makefile include + build.mk, CI `paths:`, and docs). No code changes. * fix(monorepo): repoint stale paths from the projects/ move that broke CI Two classes of path left stale by nesting products under projects/: - Web apps `require()` repo-root config.json/package.json by relative path; the extra projects/ level meant every one was short one `../` (resolved to projects/… instead of the repo root), failing the esbuild UI build for start-os (ui + setup-wizard) and start-tunnel. - test.yaml and deploy-brochure.yml still `cd start-sdk` for the baseDist build; the SDK now lives at projects/start-sdk. * fix(debian): resolve PROJECT_DIR before reading VERSION debian/build.sh computed VERSION from "projects/$PROJECT_DIR/Cargo.toml" before PROJECT_DIR was assigned, so it read projects//Cargo.toml (empty) and double-prefixed projects/. The empty Version: produced an invalid DEBIAN/control and dpkg-deb rejected it — breaking the registry and tunnel .deb builds. start-cli's CI only runs `make cli` (the binary), so it never exercised this path. Hoist the PROJECT_DIR/INSTALL_TARGET block above VERSION and read "${PROJECT_DIR}/Cargo.toml" directly. Also only fall back to the OS product's usr/lib/startos/conflicts when that file actually exists, so non-OS products don't error on a missing conflicts file. * fix(web): prettier-wrap tsconfig paths widened by the ts-modules rename Renaming shared-libs/web -> shared-libs/ts-modules pushed the `@start9labs/*` path-mapping lines past prettier's print width, so `npm run format:check` (the Formatting & Lockfiles CI job) flagged the four product tsconfig.json files. Apply prettier's wrapping. * docs(contributing): align with restructure + renamed make recipes Bring the CONTRIBUTING set up to date with the monorepo layout and the namespaced make targets: - root: `make iso` -> `make startos` - start-os: `make $(IMAGE_TYPE)`/`deb`/`squashfs` -> `make startos-$(IMAGE_TYPE)`/`startos-deb`/`startos-squashfs`; deploy/flash targets -> `startos-update*`/`startos-wormhole*`/`startos-emulate-reflash` - start-core: `cd start-sdk` -> `cd projects/start-sdk`; osBindings sync path -> projects/start-sdk/base/lib/osBindings - shared-libs: `web/` -> `ts-modules/` (heading, lib paths, file: deps, cross-links) The other products' CONTRIBUTING files were already correct. * fix(container-runtime): correct repo-root target path in update-image.sh update-image.sh runs with cwd at projects/start-os/container-runtime/ (mounted at /root/start-os in start9/build-env), so the repo-root build output is three levels up. It copied start-container from ../../target (-> projects/target, nonexistent), so the container-runtime squashfs was never built and the OS image compile failed on every arch. Use ../../../target. Refresh the AGENTS.md gotcha that described the old stale path. * refactor(make): namespace OS web targets, require an explicit target, per-project cleans - `ui`/`uis` -> `startos-ui`/`startos-uis`: they build only the StartOS admin UI + setup-wizard, so they belong under the startos-* namespace. Also fix `startos-ui` to depend on the built index.html (the old `ui` depended on a directory with no rule). - No default build: bare `make` now prints `help` (.DEFAULT_GOAL := help) and the misleading `all` target (it only built `startos`, not "everything") is removed — callers specify a target. - Decentralize `clean`: every build.mk owns a `clean-<project>` target and the root `clean` just aggregates them. Per-project cleans use project-prefix wildcards so they're arch/version-independent, and two stale paths from the projects/ move are corrected (env/*.txt -> build/env/*.txt; image-recipe/deb -> projects/start-os/build/image-recipe/deb). - Drop the start-cli targets (`make cli`/`cli-deb`) from the start-os CONTRIBUTING build section (wrong product) and update the docs to the new target names (root + start-os CONTRIBUTING/README/AGENTS). * docs: sync, complete, and standardize all developer docs for the monorepo (#3356) * docs: sync developer docs with the monorepo restructure Audit of all developer documentation (AGENTS/CONTRIBUTING/ARCHITECTURE/README across root, projects/*, shared-libs/*) against the post-restructure tree. Corrects stale references the restructure left behind: - Paths still pointing at the pre-restructure layout (core/, web/, sdk/, brochure/, container-runtime/, patch-db submodule) -> projects/* and shared-libs/*. - Angular workspace root: several docs claimed shared-libs/ts-modules holds angular.json/package.json/tsconfig.json and that npm runs from there. The workspace is rooted at the repo root; fixed cwd/--prefix instructions, the tsconfig path-alias targets, and config-sample.json location accordingly. - Renamed make targets (startos-* namespace), the Rust lib rename (startos -> start_core / crate start-core), and the binary-source table in start-core/ARCHITECTURE.md. - Removed stale 'git clone --recursive' (no submodules remain) and the non-existent repo-level scripts/ reference. - Normalized product self-references and the brochure -> brochure-marketplace Angular project name; verified incidental accuracy fixes (exver 0.2.1, patch-db serde_cbor, image-recipe live-build). All relative doc links verified resolvable. Pre-commit lint-staged hook skipped (--no-verify): the slot has no installed node_modules so the binary can't run, and the repo's prettier targets only the web source dirs, not these markdown docs. * docs: bring utility crates and patch-db up to the standard doc set Every first-party crate under shared-libs/crates/ now carries the same AGENTS/ARCHITECTURE/CONTRIBUTING/README/CLAUDE set the projects and other shared libs use. - New full doc sets for exver, imbl-value, jsonpath, pi-beep, rpc-toolkit, yasi, written from each crate's actual source (verified module names, public API, cargo -p <package> commands, and real consumers; jsonpath's package is jsonpath_lib though its dir is jsonpath). Pre-existing READMEs (exver, jsonpath, yasi) were preserved verbatim and only augmented with a 'Place in the monorepo' + 'Documentation' section. - patch-db: added AGENTS.md (migrated from its content-bearing CLAUDE.md, plus a Build & test section noting it is its own Cargo workspace) and reduced CLAUDE.md to the one-line @AGENTS.md import like every other scope. - Every CLAUDE.md is exactly '@AGENTS.md'; all relative doc links resolve. * docs: normalize section structure across all project & shared-lib docs Standardize every first-class scope (root, the 7 projects, container-runtime, shared-libs, start-core, ts-modules) onto one canonical section template so the same sections appear under the same names in the same order everywhere: - AGENTS.md: intro (+ 'CLAUDE.md is a one-line @AGENTS.md import' + cross-links) → Layout → Build & test (run from the repo root) → Gotchas → scope-specific. Folded synonym headings (Where things are / What lives where / Map → Layout; Operating rules / Gotchas / rules → Gotchas; Build / Build & verify → Build & test). - CONTRIBUTING.md: Documentation (doc-map, added where missing) → Prerequisites → Building → Testing → Formatting → scope-specific → Commits / PRs (last). - ARCHITECTURE.md: ensured a 'Place in the monorepo' opener (where applicable) and a 'Further reading' links footer; internal sections left in place. - README.md: added a Documentation/links footer where missing; landing prose left untouched. Pure reorganization: headings renamed/reordered and short missing sections added — no prose reworded, no facts/commands/tables dropped. Scope-accurate build cwds preserved (start-docs builds from its dir; start-sdk from its dir; container-runtime via npm --prefix; patch-db its own workspace). All 102 docs' relative links verified resolvable; every CLAUDE.md remains the one-line import. * build+docs: merge patch-db into root workspace; enforce doc hierarchy Addresses review feedback on the docs PR: - **patch-db → root Cargo workspace.** Add the six patch-db crates (core, json-patch, json-ptr, macro, macro-internals, util) to the root workspace `members` and drop the `exclude`; remove patch-db's now-redundant virtual `Cargo.toml`. `cargo metadata` + `cargo check -p patch-db -p json-patch -p json-ptr -p patch-db-macro` pass. Updated the docs that called patch-db 'its own workspace' (patch-db AGENTS/CONTRIBUTING build commands now run from the repo root with `-p`; root ARCHITECTURE lists it as a member). - **Root AGENTS.md:** added a note that these doc files must be kept current with every change, and that the docs are hierarchical — a scope documents only what is specific to it and never repeats higher-scope content. - **Hierarchy cleanup:** removed the `## Commits / PRs` (and `Branch / commit / PR` / `Commit conventions`) sections from every non-root CONTRIBUTING.md — those conventions live only in the root CONTRIBUTING.md now. Scope-specific change steps were preserved (e.g. start-tunnel's migration/ bindings/CHANGELOG steps moved to a 'Making a change' section; start-cli's docs-update note folded into 'Where code lives'). Also dropped the duplicated 'keep these docs in sync' line from child Documentation sections (kept the local cross-link lists). - **jsonpath:** reframed the fork note as history-only across its docs — it has fully diverged from freestrings/jsonpath with no intent to upstream; treat it as first-party and edit freely (removed the 'pull fixes from upstream / keep changes minimal / fork-tracking' guidance). All 102 docs' relative links resolve; every CLAUDE.md remains the one-line import. * docs: add hierarchy-navigation notes to AGENTS files - Every non-root AGENTS.md now opens with a 'Read up the tree first' note: the docs are hierarchical, so before working in a scope read the AGENTS.md of each enclosing directory up to the repo root (and their ARCHITECTURE/CONTRIBUTING where relevant). Excludes the packaging-guide + package-template AGENTS under projects/start-sdk/docs/, which target external package authors, not the monorepo dir tree. - Root AGENTS.md gains the converse 'Read down into what you touch' note: read a subdirectory's AGENTS.md (and any further nested ones) before editing it. - Dropped the explicit repo-root CONTRIBUTING pointer from patch-db's CONTRIBUTING.md (the walk-up convention now covers it). * docs(agents): require product docs/ book + CHANGELOG to ship with code Root AGENTS.md now mandates that any change altering user-visible behavior update that product's user-facing docs/ book (projects/<product>/docs/) in the same change and add a CHANGELOG.md entry — no deferring docs/changelog to follow-ups. * build(make): add per-project format targets mirroring the clean decomposition Each build.mk now owns `format-<project>` + `format-check-<project>` (core, web, sdk, cli, registry, tunnel, startos), matching the per-project `clean-<project>` targets; the top-level `format`/`format-check` just aggregate them. Web (the whole Angular workspace incl. brochure) formats via the root npm script; the shared crates via one `cargo +nightly fmt`; container-runtime via its own prettier config (new `format`/`format:check` npm scripts). So you can format one project (`make format-cli`) or all (`make format`). * docs: address PR review — make-target refs, ARCH de-dup, ts-modules wording - Reference stable make targets instead of raw build/format commands across project docs: builds via `make cli`/`registry`/`tunnel`/`startos`/`startos-ui`, formatting via the per-project `make format-<project>` targets (and `format-check-<project>` for CI). Kept `cargo check`/`cargo test` and crate `cargo build` (no make equivalent) as noted dev shortcuts. - Fixed stale targets: dropped `all` (removed upstream; `make` now prints help), `ui`/`uis` -> `startos-ui`/`startos-uis`. - Removed the whole-monorepo ASCII trees that several product ARCHITECTURE.md files re-drew (start-cli, start-registry, container-runtime, ...) — that layout lives once in the root ARCHITECTURE.md; each now states only where it sits. - Reworded the `shared-libs/ts-modules` directory as shared TypeScript modules (not Angular-specific; current contents are the Angular libs shared/marketplace), per review; kept accurate per-library 'Angular library' phrasing. Branch merged up to date with docs/monorepo-proposal first. * refactor(sdk): extract base into @start9labs/start-core shared lib; flatten start-sdk Move projects/start-sdk/base -> shared-libs/ts-modules/start-core (package @start9labs/start-sdk-base -> @start9labs/start-core), mirroring the Rust crate shared-libs/crates/start-core. start-core builds its own self-contained dist consumed via file: deps. Flatten start-sdk (package/ -> root): the SDK now imports @start9labs/start-core instead of ../../base/lib, and its published dist bundles start-core (bundleDependencies) so external authors still install one package. Repoint all consumers off the SDK-as-base alias onto @start9labs/start-core: - web (root file: dep + 153 import sites + shared/marketplace peerDeps) - container-runtime (file: dep; base/lib imports -> start-core, package/lib -> lib) Repoint osBindings generation, the build DAG (build.mk fragments, Makefile), and regenerate the three lockfiles. Resolves the "SDK kept cohesive" deviation: base is now an honest first-class shared TS lib named for what it is. * docs+ci: reflect start-core extraction; repoint SDK build steps off base/baseDist CI: build start-core (cd shared-libs/ts-modules/start-core && make dist) before validating the SDK + web lockfiles; validate the flattened SDK lockfile at projects/start-sdk; add start-core to the prettier check; repoint the iso prevent-rebuild mkdirs and the brochure deploy paths: filter (start-sdk -> start-core). Docs: update every project's AGENTS/ARCHITECTURE/CONTRIBUTING/README for the new layout — base extracted to @start9labs/start-core under shared-libs/ts-modules, the SDK flattened (lib/) and bundling start-core, container-runtime depending on both. Removes the resolved "SDK kept cohesive" deviation note. * refactor(monorepo): use the start-technologies name; repoint init-workspace at the monorepo - Adopt start-technologies for monorepo/repo-URL references across docs and AGENTS files (ahead of the GitHub repo rename; product refs left as start-os: projects/start-os, the start-os crate, *-startos packages, StartOS). - s9pk init-workspace now sparse-clones the start-technologies monorepo (projects/start-sdk/docs) instead of the retired standalone start-docs repo, and the two start-docs-named init strings are reworded across all five locales. - Un-hide the SDK 2.0 packaging-workspace section and rewrite it for the monorepo clone; document fetch-on-demand SDK + OS source access (docs first) in the workspace AGENTS.md and workflow.md. - Fix build-config.js to read and write the repo-root config.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkwYSXK8Uh9D16UTffCNJ1 --------- Co-authored-by: Aiden McClelland <me@drbonez.dev> Co-authored-by: Matt Hill <9935159+MattDHill@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2 个月前
draft-start9-upnp-hostname.md	feat(start-wrt): SNI hostname routes end to end — UPnP vendor actions, StartWRT dataplane, Remote Access coexistence (#3783) * feat(start-tunnel): UPnP vendor action for SNI hostname mappings Add X_START9_AddHostnameMapping / X_START9_DeleteHostnameMapping to the shared UPnP IGD server, giving UPnP parity with the PCP HOSTNAME option: a client that reaches a Start9 gateway over UPnP but not PCP no longer silently loses SNI demux. The StartOS port-map client falls back to the vendor action when no gateway grants the hostname over PCP, detecting support via the SCPD action list the discovery already fetched (no PatchDb field, no TS bindings). Refresh re-asserts the route without a remote delete — registration reclaims idempotently for the same target, so re-adding in place avoids a per-tick outage window. Unlike standard UPnP mappings, vendor-action routes are always lease-bearing (clamped to the server max): a permanent SNI binding is reserved for operator-created routes, and an unreaped device route would answer HostnameTaken (fault 800) to its legitimate owner forever. The delete action carries NewInternalPort because an SNI route's ownership is its full (peer, internal port) target, mirroring the PCP lifetime-0 MAP, and is gated on is_known_client like the PCP delete. Non-TCP requests are refused (the demux is TCP-only), and hostnames are validated client-side before being interpolated into the envelope. Both handlers guard on the backend having an SNI dataplane, faulting 801 HostnameNotSupported otherwise — the UPnP twin of the PCP path's RESULT_UNSUPP_HOSTNAME refusal. Served by StartTunnel today; a StartWRT gateway (which has no dataplane yet) advertises-but-refuses until its demux lands, then serves the action with no further edit. * feat(start-wrt): SNI hostname-route dataplane StartWRT's port-control gateway now serves TLS-SNI hostname routes end to end — PCP HOSTNAME and the X_START9_AddHostnameMapping UPnP vendor action both work against the router instead of faulting 801, so several devices (or several services on one StartOS server) share an external port such as 443, demuxed by ClientHello hostname. Dataplane: the shared SniDemux runs as-is; what StartWRT needed was the plumbing around it. The reply-path divert's nft half ships declaratively as an fw4 include (12-startwrt-sni-divert.nft, `mark or` to preserve the 0x80 DNAT-return bit) — fw4 re-renders includes on every reload, so no reload window can drop it. The iproute2 half is parameterized via a new shared DivertConfig (route table 5344 to clear the VLAN-tag table namespace, masked fwmark to match the or-set mark, manage_nft off); defaults reproduce StartOS/StartTunnel behavior bit-for-bit. The `socket transparent` expression needs kmod-nft-socket (+kmod-nf-socket), added to the image diffconfig in this same commit — without the module fw4 refuses the entire ruleset, so the include and the kmods must ship together. Admission: each demuxed port gets a WAN-input ACCEPT rule (apf_sni_<port>, tagged _apf_label 'SNI' via a new FirewallRule field), written inline under the write lock so a concurrent plain-forward scan can never miss it; the demux's on_change teardown drops it, the sweep heals strays and gaps, and daemon start purges leftovers (routes are demux-memory only — finite-lease, device-renewed — so rules must not outlive them). The rule also makes the port read as router-reserved, keeping plain auto forwards off a demuxed port for free. Conversely add_sni_forward refuses ports already DNAT-forwarded or answered by the router itself (Remote Access, VPN): the demux's specific (wan_ip, port) bind would beat their wildcard binds and capture traffic it has no route for. WAN re-key: listeners bind the WAN address itself, so a new address strands them. A new wan hotplug hook fires published-ports.wan-changed (hidden RPC, daemon-forwarded like reconcile) to re-key immediately via the new shared SniDemux::rekey_ipv4 — which never fires the teardown callback, since the port set is unchanged — with the sweep as a once-a-minute backstop. Visibility: published-ports.auto-list now appends one row per live route (label "SNI", new hostname field, device resolved from the target address) via the new shared SniDemux::snapshot; the Automatic table gains a Hostname column. API_CONTRACT, the user docs' Automatic Port Forwarding page, and the unreleased 1.1.0 changelog entry (which claimed StartWRT has no SNI demux) updated to match; build.mk's staging deps now cover backend/hotplug and backend/nftables (pre-existing gap). * fix(start-wrt): SNI demux binds beside the UI and coexists with Remote Access Bench testing found the StartWRT dataplane dead on 443 and worse than dead: the web UI wildcard-binds [::]:443, so the demux's specific (wan_ip, 443) bind failed EADDRINUSE forever in its spawned retry loop — while the grant had already succeeded and opened the apf_sni_443 WAN admit rule. WAN 443 traffic fell through to the UI's wildcard socket, serving the router admin interface to WAN clients with Remote Access set to Never. On a public-WAN router (where "behind NAT" mode writes no rules and so nothing conflict-refuses the route) that would have been the open internet. Three changes close it: - The demux listener and the daemon's UI 80/443 listeners all bind with SO_REUSEPORT (the DNS :53 pattern). TCP delivery prefers the most specific bound address, so the demux takes WAN-IP-destined connections and the UI wildcard keeps the LAN. - Grants are bind-gated: SniDemux::register/register_fallback bind inline and refuse with PCP NO_RESOURCES (UPnP fault 501) on failure, rolling back the registration — a grant can never outrun its socket and leave the admitted port served by whatever shares it. A re-key bind failure now drops that port's routes and fires the teardown callback rather than stranding the admit rule. - Remote Access coexists with hostname routes on 443 instead of reserving it (it is the default mode behind NAT, and demanding it be turned off to share 443 was untenable): its rules no longer count as SNI conflicts on 443; the daemon instead registers its own UI as the demuxed port's fallback, so no-SNI/unknown-SNI connections (browsing the router by IP sends no SNI) still reach the UI. The fallback leg is a plain connect (a source-preserving dial to ourselves would be martian-dropped) and enforces the same source scoping the displaced firewall rules encoded — any source in "always", RFC1918-only in "default" behind NAT, none in "never" — re-synced on route add, on a Remote Access change, and by the sweep, which also clears it when the last 443 route expires. SSH (server-speaks-first) and the port-80 redirect (plain HTTP) can't ride an SNI peek, so those Remote Access ports — and manual forwards and the VPN port — keep refusing routes. The IGD hostname stub tests move to an unprivileged external port: registration now really binds, and 443 needs root the runner lacks. * feat(start-wrt): router-port confirm dialog names the port's actual holder The publish-confirmation dialog said "Used by This Router" even when the colliding WAN-input rule was an SNI-demux admit rule — a port really held by a device's hostname routes. router_reserved_overlaps now classifies each overlapping rule, RouterPortCollision splits the specs into router_ports and sni_ports (the latter enriched from the live demux with the routed hostnames and owning devices, named the same way auto-list rows are), and the dialog composes its copy from the actual holders: router services, hostname routes, or both on a shared port. The override semantics are unchanged. * fix(start-wrt): revoking a device's permission drops its SNI hostname routes `close_device_forwards` removed a device's `apf_*` UCI redirects but never touched the SNI demux, so its hostname routes kept delivering WAN traffic — admit rule and all — until their lease lapsed, up to an hour after the toggle went off or the device was forgotten. StartTunnel's `clear_for_peer` already covers routes; this brings StartWRT in line. Routes are keyed by target address, not device, so rather than carry a grant-time address→MAC map that can drift, the owner is re-derived the way a grant derives it (neighbor table, else DHCP leases and static hosts) and any route whose device is no longer `_allow_pcp` is unregistered. That audit runs on every revocation and once a minute from the sweep, which also catches an address recycled to a different, unauthorized device. An address that maps to no device is left to its lease rather than reaped on a guess. * fix(start-wrt): hairpinned SNI clients get a working connection, not a hang A LAN client dialing the router's WAN address for a routed hostname hung. The demux opens the internal leg from the client's own source address, so the target — sitting on the same bridge as the client — answered it directly over that segment. The reply never came back through the router, and the client discarded it as coming from an address it never dialed. Drop source preservation for exactly that case and dial the target as ourselves instead, the trade the DNAT path already makes with the hairpin masquerade in `build/lib/scripts/forward-port` — and, on this router, the one fw4's redirect reflection already makes for every manual published port, so the two now behave alike. The demux learns a host's segments through `LocalPrefix`, the userspace equivalent of that script's `target_prefix`: StartWRT answers it from the `br-*` addresses it already parses for SSDP, so the test is per-bridge. A client on another profile's VLAN, or on the WAN, still routes its replies through the router and keeps its address. The tunnel supplies no resolver — every path through its WireGuard hub returns to it, so nothing there needs the trade. The check runs before the dial, never as a fallback after a failed one: the backend gates LAN-only addresses on the source being private, so a blanket plain-connect would present a WAN client as LAN-local. * docs(start-wrt): a hostname route is reachable from every Security Profile The trust note tells the reader to isolate an untrusted device on its own profile so it cannot act for the devices they trust — true of opening forwards, and easy to over-read as reach isolation. A profile's LAN Access setting is a forwarding control, and the router serves a hostname route from its own socket, so no profile boundary stands between a client and a routed hostname. Say so, alongside the fact that makes it uninteresting: the service is published to the Internet either way. * docs(rfcs): UPnP vendor-action status claims only Start9-internal acceptance Aiden's review flagged the bare "Status: accepted" as reading like a standards-status claim. The doc is not IETF-targeted — its own open question 4 notes a UPnP vendor action has no standards venue — so the status line now scopes the acceptance to Start9 and says where external documentation lives (the SCPD). Claude-Session: https://claude.ai/code/session_012peNJE6QiAEMDJEjWYDwjE * fix: address SNI hostname route review findings * fix: make SNI route grants transactional * fix(start-wrt): inline divert setup error mapping * fix(start-os): restore mapped external IP handling * chore: regenerate start-core TypeScript bindings * style(start-os): format port-map tests --------- Co-authored-by: Helix <267227783+helix-nine@users.noreply.github.com>	6 天前
sni-demux-kernel-handoff.md	fix(os-install): refuse preserve selections that would erase the data drive (#3502) * fix(os-install): refuse preserve selections that would erase the data drive Choosing "Preserve" for a data drive whose StartOS pool lives on a partition of that drive (the 0.3.x single-drive layout) while installing the OS to a different drive missed the existing-pool lookup and fell through to setup_data_drive() — silently reformatting the very drive the user asked to keep, then failing later with a cryptic "Not found `version` in db" when the empty pool was attached. plan_data_drive() now resolves the preserve target before any disk is written and fails fast with an actionable, localized error: - pool on a partition of the selected drive -> attach only when the OS shares the drive (that partition is protected during install) - pool spanning the whole drive -> attach only when the OS goes elsewhere (OS partitions can't be carved out of a whole-disk PV); selecting such a drive for both OS and data now errors up front with guidance instead of the mid-install "Cannot protect the entire disk" - no StartOS pool found on a "Preserve" target -> error, never create Unit tests cover each layout/selection combination, including the data-loss case from the field. * feat(setup-wizard): refuse preserve selections the installer would reject Mirror the backend plan_data_drive guard (os_install/mod.rs) in the drive-selection page: when the user chooses "Preserve", validate the (OS drive, data drive) combination up front and block the doomed selections with actionable guidance, instead of submitting an install that fails later (or, before this PR's backend fix, silently reformatted the data drive). preserveBlockedReason() reproduces all four backend refusal cases (whole-disk pool sharing a drive, partition pool needing the same drive, partition pool on a pre-installed device, no StartOS data to preserve), gated to the Preserve path so the Overwrite/wipe flow is unchanged. Adds the label + four localized messages to all five dictionaries and folds the wizard behavior into the existing CHANGELOG entry. * fix(web): refine preserve-blocked messaging and dialog flow - Update the preserve-blocked error message to remove the leading '"Preserve" was selected,' prefix - the dialog already opens in the context of a blocked preserve selection, so the message is cleaner without it - Move the preserve validation from the Continue handler to the OS drive change handler - when the user reselects the OS drive, the blocked reason is re-evaluated and the dialog re-offers with the current state - Pass the blocked reason into the preserve/overwrite dialog so it can display the specific reason preserve is unavailable - Update all five locale dictionaries (en_US, de_DE, es_ES, fr_FR, pl_PL) with the revised wording - Update CHANGELOG.md to reflect the new dialog behavior - Refresh mock test data to match the updated drive detection logic --------- Co-authored-by: Helix <267227783+helix-nine@users.noreply.github.com>	1 个月前