AAiden McClellandadd encryption subkey to gpg key
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore(ci): continuously deploy master to alpha, add integration/ and live-docs branches (#3652) * chore(ci): continuously deploy master to alpha, add integration/ and live-docs branches master becomes the always-shippable branch and publishes on every push: StartOS images into the alpha registry, .debs into a new `alpha` apt suite, and StartWRT images once STARTWRT_ALPHA_REGISTRY is set. The SDK stays out — an npm version can never be republished, so burning one per commit would block the real release. Work that can't land piecemeal goes on integration/<topic>; the PR merging one into master builds the full flashable image matrix regardless of paths touched. next/* is retired from every trigger. debian/publish.sh gains per-suite pools and suite-scoped sync, so an alpha publish can neither index nor delete anything in stable. alpha is signed by a CI-only key with its own keyring, never the production release key. docs.start9.com now serves live-docs rather than master: a book goes live when its product is tagged, and fixes to already-published docs are PR'd against live-docs and landed on master automatically. Both docs workflows authenticate as a GitHub App whose key is an environment secret scoped to the ref each one triggers on — granting the bypass to github-actions[bot] instead would let any workflow on any branch write to master. The backport commit is created through the GraphQL API so GitHub signs it, and expectedHeadOid makes the write an atomic compare-and-swap against master's head. Adds scripts/update-from-gha.sh (make start-os-update-from-gha) to put a CI build on a server without compiling locally. * fix(ci): address review — apt suite isolation, fork gating, and docs sync guards Alpha debs lived at pool/alpha/, inside stable's pool root, so a later stable publish scanned them into dists/stable/Packages as Filename: pool/alpha/... Suite pool roots are now siblings (pool/ and pool-<suite>/), which makes the isolation structural rather than a property of the scan path. The download sync no longer swallows failures. Paired with --delete-removed on upload, a partial mirror would have pruned whatever it was missing; an empty prefix is now detected explicitly instead of being inferred from an error. The three deb publishers share one apt-publish-alpha concurrency group. They write the same pool and dists, each syncing down and uploading with --delete-removed, so concurrent runs would drop each other's packages. Alpha OS and StartWRT images move under an alpha/ key prefix, keeping per-commit builds out of the namespace manage-release.sh signs and publishes. The integration/* image bypass now requires the PR head to live in this repo: head_ref is only a branch name, so a fork branch named integration/anything could spend the full image matrix. start-wrt's image job also gains the draft gate the other expensive jobs carry. docs-deploy pins its checkout to live-docs. The Run workflow UI defaults to master, and the checkout had no ref, so a default dispatch would have published master's unreleased books to docs.start9.com. [skip-backport] is now honoured only on the docs bot's own commits, so a docs page that merely contains the string cannot silently skip its own backport. docs-sync-on-tag gains two guards on the shared projects/start-docs tree: the tagged commit must be an ancestor of master, and a tag predating the published infrastructure syncs only its book. Each sync records Source-Commit, which the next run compares against. Without these a backfilled tag would roll the whole site back to old infrastructure, with [skip-backport] preventing reconciliation. upgrade compared the image hash only when given a second positional argument, which no caller passed — so updates never verified their checksum and a corrupt but mountable image installed silently. It now keys off CHECKSUM; callers pass it positionally too, since servers on older images carry the old script. * fix(ci): build release-equivalent artifacts on master, revert alpha s3 prefix Alpha is the source registry beta and production are promoted from, and os promote (registry/os/promote.rs) copies each asset's URL and signed commitment through verbatim. So whatever a master push builds and uploads is what eventually ships, from the same S3 key. Two corrections follow. ENVIRONMENT is now empty on a push to master rather than defaulting to dev, so continuous deployment stops publishing dev-flavoured artifacts (password SSH before setup, uncompressed frontends). Pull requests and integration/* branches still default to dev, which is where that build is wanted. And the alpha/ S3 key prefix added in the previous commit is reverted: because promotion carries urls[0] through unchanged, prefixing alpha uploads would leave promoted production assets pointing inside an alpha namespace, which is both wrong and a hazard if those objects are ever lifecycle-expired. * feat(ci): refuse to overwrite already-published S3 objects without force Basenames are commit-derived, so a key that already exists means that commit was published before. Because os promote copies asset URLs through verbatim, a beta or production commitment may already point at those exact bytes — replacing them would leave a valid signature validating against content that no longer matches. Both deploy jobs now check each key before uploading and fail if it exists, listing every conflict rather than one per re-run. A workflow_dispatch input, force-overwrite, replaces them deliberately; it is unavailable to push events, so continuous deployment can never take that path. * fix(ci): serialize deploys, keep the infra watermark, carry every backport commit The deploy jobs get a per-channel concurrency group shared across event types. The workflow-level group is scoped by github.event_name, so a dispatched alpha deploy could run alongside a master-push one and interleave their remove -> add -> asset-add sequences against the same registry version. The Source-Commit watermark now walks back to the newest commit that actually carries a trailer. Reading only the newest commit touching projects/start-docs meant a hand PR to live-docs fixing the shared infrastructure blanked the watermark, re-opening the rollback the guard exists to prevent. The backport's PR fallback cherry-picks one commit at a time. Picking the whole range stopped at the first conflict, and a single --continue advanced past only that one, so a range conflicting in two places pushed a branch missing its tail while the PR claimed the whole range. Commits that cannot be applied at all are now named in the PR body rather than silently absent. Also documents a limitation of the shared apt-publish-alpha group: GitHub keeps only one pending run per group, so three simultaneous publishes supersede the middle one and that product's alpha deb stays at its previous version until its next push. That costs freshness, never consistency. * refactor(ci): publish the alpha apt suite from a single writer The three deb products share pool-alpha/ and dists/alpha/, and every publish syncs the suite down, replaces its own package, and uploads with --delete-removed. Serializing the per-product publishers on a shared concurrency group stopped them corrupting each other but not going stale: GitHub keeps only one pending run per group, so three simultaneous publishes supersede the middle one and that product's package sits at its previous version until its next push. Replaces them with one apt-publish-alpha workflow triggered on the completion of any deb product's master build. It collects each product's latest successful master build rather than the artifacts of the run that triggered it, so a superseded run costs nothing — whichever run survives republishes the current state of all three. A product whose last successful build has aged past artifact retention is skipped with a warning rather than failing the publish, leaving its existing package in place. The trigger matches on workflow name rather than filename, so it is recorded in the root AGENTS.md coupled-changes list. * fix(ci): only collect same-repo master pushes into the alpha apt suite workflow_run.branches filters the triggering run's head branch, and a pull_request run reports the PR's own head branch — so a fork PR opened from a branch named master satisfied both that trigger and the `gh run list -b master` query. The publisher would then collect fork-built .deb artifacts and sign them into alpha with the CI key. The job now requires the triggering event to be a push, and the query passes --event push and re-asserts it on the returned row, so only same-repo master pushes are collectible. That also excludes dispatch builds, which may be dev-flavoured or partial-arch. The docs watermark walk no longer pipes sed into `head -n1`. head closes the pipe after one line, and once enough trailers exist for sed to flush a second time it takes SIGPIPE and pipefail aborts the tag sync — reproducible at ~120 published tags. It captures the filtered lines and slices the first in shell. The deploy jobs take a single lock rather than one per channel. alpha and beta write the same v$VERSION/ S3 keys for a given commit, so they are not independent: two channels deploying one commit at once could both pass the overwrite check and then race the same object. * fix(ci): divert symlink and mode changes out of the signed-commit backport path createCommitOnBranch can only express regular files: additions carry base64 content and land as mode 100644. Every book's docs/theme is a symlink (mode 120000) to the shared start-docs theme, and base64 reads through a symlink to its target — where that target is a directory it fails, the command substitution swallows the failure, and the mutation writes contents:"" over the symlink, replacing it with an empty regular file. Reproduced end to end with exit 0. The backport now inspects the staged diff's modes and diverts anything that is not a plain 100644 change to the git-based PR fallback, which represents modes natively. The fallback output carries a reason so the PR body can say whether a human needs to resolve conflicts or merely review — a symlink diversion applies cleanly and should not be described as a conflict. Also corrects projects/start-docs/ARCHITECTURE.md, which still documented deployment from master and omitted the start-wrt docs path. * fix(ci): represent renames as delete + add in the signed backport commit Rename detection is on by default, so a renamed page came back as one R row carrying two tab-separated paths. The additions filter emitted only the new path and the deletions filter emitted nothing, so the commit added the moved page and left the old one behind — and the mode check read the old path out of the wrong field. --no-renames on all three diffs renders a rename as a delete plus an add, which createCommitOnBranch expresses exactly, and gives every raw row a single path. * feat(release): promote debs to stable from alpha instead of a CI run The OS and StartWRT already release by promotion — CI publishes to a channel and the release moves that exact artifact onward — but the debs still rebuilt trust from a GitHub Actions run, so the packages reaching stable were not necessarily the ones testers had been running. `release start-cli` / `start-tunnel` / `start-registry` now stage from the alpha suite. The apt pool cannot identify a build on its own: dpkg-name rewrites each file to <package>_<version>_<arch>.deb, dropping the hash the build basename carries, and every master build of a version publishes under the same Version. So debian/build.sh records a Git-Hash control field, which dpkg-scanpackages carries into the Packages index — the right build is selected and verified before anything is downloaded. Alpha holds only the newest master build per package, so a hash mismatch means master moved past the commit being tagged. That fails rather than shipping bytes nobody soaked, which also enforces the existing rule to release from an up-to-date master. pull-gha stays as the repair path. start-cli's per-triple binaries are unchanged: they are published only as release assets, so there is no channel to promote them from and that half still needs the run that built the tagged commit. Both halves are pinned to it, so the two sources cannot disagree. * fix(release): verify the alpha suite's signature chain before promoting The promotion read an unsigned Packages index over HTTPS, trusted a self-asserted Git-Hash field, and downloaded the pool object without checking it. Two ways that fails: the pool key is stable across builds, so an alpha republish between the index read and the download swaps the bytes after the commit check has passed; and anyone able to write the bucket without holding the signing key could forge a stanza naming the target commit and have a releaser sign those bytes into stable. Alpha is signed with the CI key precisely so its consumers can verify it, and a releaser promoting into stable is one. The chain is now anchored end to end: InRelease against apt/start9-alpha.gpg, Packages against the hash the signed Release commits to, and each .deb against the hash that index commits to. Verified against a locally built signed fixture — a valid suite promotes, a tampered pool object is rejected, and a forged stanza is rejected because the attacker cannot re-sign Release. Staging is no longer partial. The marker is written and stale debs cleared before any fetch, and downloads land in a temp dir that is only moved into the release directory once every arch has verified, so an interrupted promotion cannot leave a mixed set beside a marker that disagrees with it. The mismatch error no longer prescribes an impossible remedy. Product workflows are path-filtered, so master routinely advances without producing a new build of a given product and "cut from an up-to-date master" cannot help. It now prints the COMMIT=<hash> invocation that tags the commit which actually produced the bytes. * feat(release): tag the commit alpha built instead of demanding one at HEAD A tag claims that a commit produced the artifact, so when the release promotes, alpha's build is what decides which commit to tag. Product workflows are path-filtered, so master routinely advances without rebuilding a given product; insisting on HEAD demanded a build that will never exist and left the operator to notice the mismatch and re-run with COMMIT= by hand. pull-alpha now adopts that commit, reports how far HEAD has moved on, and prints the git checkout for putting a tree on it. A new alpha-commit subcommand prints the same hash for scripting. Adoption is confined to a commit already in the current branch's history, and an explicit COMMIT is never overridden — that is someone asserting a different intent, so it fails instead. Every architecture must also report the same commit, so a half-published suite cannot produce a release assembled from two builds. * fix(release): fetch the alpha suite from the origin, not through a cache A promotion has to see the suite as it is now, and signature verification does not give that: a cached InRelease is still a validly signed InRelease, so every signature and hash check would pass while quietly promoting whatever build the edge happened to be holding. Signatures prove authenticity, not freshness. APT_BASE_URL already pointed at the S3 origin rather than the *.cdn.* host that apt/start9*.list gives clients, but nothing said so — it read as an inconsistency waiting to be tidied up. It is now documented as deliberate at the definition and in the root AGENTS.md, and every fetch in the promotion path (including pull_apt_debs) sends no-cache headers for any intermediary between the releaser and the origin. End users keep the CDN: apt is built to tolerate a stale mirror, a releaser signing bytes into stable is not. * fix(release): pin both halves to the adopted commit, and bound metadata age For start-cli the run was resolved and its binaries downloaded before promote_alpha_debs adopted alpha's commit. With HEAD past start-cli's last path-filtered build, a run for HEAD validated fine, adoption then moved COMMIT to the older commit and rewrote the marker to match, and cmd_tag tagged there — binaries from one commit, debs and tag from another, every check passing. The alpha commit is now determined and adopted first, so the run is validated against the commit actually being tagged. Release notes are read from the working tree, which an adopted commit can be behind. Where the changelog is identical the distinction is immaterial and nothing changes; where it differs the notes would describe a tree the tag does not point at, so the promotion stops and asks for the checkout. alpha-commit inspected only the first architecture and accepted a missing Git-Hash, so it could print a commit pull-alpha would then refuse. Both now go through one alpha_collect that verifies every architecture and requires them to agree. Finally, a signature proves authenticity but never freshness: someone able to write the bucket without the key could restore an older, validly signed Release plus pool and have it promoted. publish.sh now emits Valid-Until (30 days, configurable) and the promotion refuses metadata that is expired or carries no bound at all. * revert(release): drop Valid-Until, and fix portability and staging holes Valid-Until was the wrong answer and I should not have implemented it. It was applied to every suite including stable, so any month without a start-cli, start-tunnel or start-registry release would have failed `apt update` for every user. Even scoped to alpha it expires a rolling channel on a timer unrelated to release cadence, to close a narrow replay of a previously legitimate build. What actually guards that is showing the operator which commit is being tagged, where a replay appears as an unexpectedly old hash. Reverted from both files, with the reasoning recorded so it does not get reintroduced. The promotion also used `date -d` and bare `sha256sum`, both GNU-only, in a script that checks for bash >= 4.4 with a macOS hint and already carries a sha256sum/shasum fallback in checksum_block. Every alpha promotion would have failed closed on macOS. The date parse is gone with the revert; hashing now goes through a shared sha256_of helper. Staging cleared only *.deb, so a failed start-cli binary download left a fresh marker beside a previous attempt's debs, and the reverse partial set was equally reachable. Both halves are now cleared, and the marker written, before either is fetched. APT_BASE_URL follows the other endpoints in taking an override, which is what allows the promotion to be exercised end to end against a locally built signed suite rather than only through extracted functions. Usage text updated: it still described the pre-adoption failure mode. | 30 天前 | |
chore(ci): continuously deploy master to alpha, add integration/ and live-docs branches (#3652) * chore(ci): continuously deploy master to alpha, add integration/ and live-docs branches master becomes the always-shippable branch and publishes on every push: StartOS images into the alpha registry, .debs into a new `alpha` apt suite, and StartWRT images once STARTWRT_ALPHA_REGISTRY is set. The SDK stays out — an npm version can never be republished, so burning one per commit would block the real release. Work that can't land piecemeal goes on integration/<topic>; the PR merging one into master builds the full flashable image matrix regardless of paths touched. next/* is retired from every trigger. debian/publish.sh gains per-suite pools and suite-scoped sync, so an alpha publish can neither index nor delete anything in stable. alpha is signed by a CI-only key with its own keyring, never the production release key. docs.start9.com now serves live-docs rather than master: a book goes live when its product is tagged, and fixes to already-published docs are PR'd against live-docs and landed on master automatically. Both docs workflows authenticate as a GitHub App whose key is an environment secret scoped to the ref each one triggers on — granting the bypass to github-actions[bot] instead would let any workflow on any branch write to master. The backport commit is created through the GraphQL API so GitHub signs it, and expectedHeadOid makes the write an atomic compare-and-swap against master's head. Adds scripts/update-from-gha.sh (make start-os-update-from-gha) to put a CI build on a server without compiling locally. * fix(ci): address review — apt suite isolation, fork gating, and docs sync guards Alpha debs lived at pool/alpha/, inside stable's pool root, so a later stable publish scanned them into dists/stable/Packages as Filename: pool/alpha/... Suite pool roots are now siblings (pool/ and pool-<suite>/), which makes the isolation structural rather than a property of the scan path. The download sync no longer swallows failures. Paired with --delete-removed on upload, a partial mirror would have pruned whatever it was missing; an empty prefix is now detected explicitly instead of being inferred from an error. The three deb publishers share one apt-publish-alpha concurrency group. They write the same pool and dists, each syncing down and uploading with --delete-removed, so concurrent runs would drop each other's packages. Alpha OS and StartWRT images move under an alpha/ key prefix, keeping per-commit builds out of the namespace manage-release.sh signs and publishes. The integration/* image bypass now requires the PR head to live in this repo: head_ref is only a branch name, so a fork branch named integration/anything could spend the full image matrix. start-wrt's image job also gains the draft gate the other expensive jobs carry. docs-deploy pins its checkout to live-docs. The Run workflow UI defaults to master, and the checkout had no ref, so a default dispatch would have published master's unreleased books to docs.start9.com. [skip-backport] is now honoured only on the docs bot's own commits, so a docs page that merely contains the string cannot silently skip its own backport. docs-sync-on-tag gains two guards on the shared projects/start-docs tree: the tagged commit must be an ancestor of master, and a tag predating the published infrastructure syncs only its book. Each sync records Source-Commit, which the next run compares against. Without these a backfilled tag would roll the whole site back to old infrastructure, with [skip-backport] preventing reconciliation. upgrade compared the image hash only when given a second positional argument, which no caller passed — so updates never verified their checksum and a corrupt but mountable image installed silently. It now keys off CHECKSUM; callers pass it positionally too, since servers on older images carry the old script. * fix(ci): build release-equivalent artifacts on master, revert alpha s3 prefix Alpha is the source registry beta and production are promoted from, and os promote (registry/os/promote.rs) copies each asset's URL and signed commitment through verbatim. So whatever a master push builds and uploads is what eventually ships, from the same S3 key. Two corrections follow. ENVIRONMENT is now empty on a push to master rather than defaulting to dev, so continuous deployment stops publishing dev-flavoured artifacts (password SSH before setup, uncompressed frontends). Pull requests and integration/* branches still default to dev, which is where that build is wanted. And the alpha/ S3 key prefix added in the previous commit is reverted: because promotion carries urls[0] through unchanged, prefixing alpha uploads would leave promoted production assets pointing inside an alpha namespace, which is both wrong and a hazard if those objects are ever lifecycle-expired. * feat(ci): refuse to overwrite already-published S3 objects without force Basenames are commit-derived, so a key that already exists means that commit was published before. Because os promote copies asset URLs through verbatim, a beta or production commitment may already point at those exact bytes — replacing them would leave a valid signature validating against content that no longer matches. Both deploy jobs now check each key before uploading and fail if it exists, listing every conflict rather than one per re-run. A workflow_dispatch input, force-overwrite, replaces them deliberately; it is unavailable to push events, so continuous deployment can never take that path. * fix(ci): serialize deploys, keep the infra watermark, carry every backport commit The deploy jobs get a per-channel concurrency group shared across event types. The workflow-level group is scoped by github.event_name, so a dispatched alpha deploy could run alongside a master-push one and interleave their remove -> add -> asset-add sequences against the same registry version. The Source-Commit watermark now walks back to the newest commit that actually carries a trailer. Reading only the newest commit touching projects/start-docs meant a hand PR to live-docs fixing the shared infrastructure blanked the watermark, re-opening the rollback the guard exists to prevent. The backport's PR fallback cherry-picks one commit at a time. Picking the whole range stopped at the first conflict, and a single --continue advanced past only that one, so a range conflicting in two places pushed a branch missing its tail while the PR claimed the whole range. Commits that cannot be applied at all are now named in the PR body rather than silently absent. Also documents a limitation of the shared apt-publish-alpha group: GitHub keeps only one pending run per group, so three simultaneous publishes supersede the middle one and that product's alpha deb stays at its previous version until its next push. That costs freshness, never consistency. * refactor(ci): publish the alpha apt suite from a single writer The three deb products share pool-alpha/ and dists/alpha/, and every publish syncs the suite down, replaces its own package, and uploads with --delete-removed. Serializing the per-product publishers on a shared concurrency group stopped them corrupting each other but not going stale: GitHub keeps only one pending run per group, so three simultaneous publishes supersede the middle one and that product's package sits at its previous version until its next push. Replaces them with one apt-publish-alpha workflow triggered on the completion of any deb product's master build. It collects each product's latest successful master build rather than the artifacts of the run that triggered it, so a superseded run costs nothing — whichever run survives republishes the current state of all three. A product whose last successful build has aged past artifact retention is skipped with a warning rather than failing the publish, leaving its existing package in place. The trigger matches on workflow name rather than filename, so it is recorded in the root AGENTS.md coupled-changes list. * fix(ci): only collect same-repo master pushes into the alpha apt suite workflow_run.branches filters the triggering run's head branch, and a pull_request run reports the PR's own head branch — so a fork PR opened from a branch named master satisfied both that trigger and the `gh run list -b master` query. The publisher would then collect fork-built .deb artifacts and sign them into alpha with the CI key. The job now requires the triggering event to be a push, and the query passes --event push and re-asserts it on the returned row, so only same-repo master pushes are collectible. That also excludes dispatch builds, which may be dev-flavoured or partial-arch. The docs watermark walk no longer pipes sed into `head -n1`. head closes the pipe after one line, and once enough trailers exist for sed to flush a second time it takes SIGPIPE and pipefail aborts the tag sync — reproducible at ~120 published tags. It captures the filtered lines and slices the first in shell. The deploy jobs take a single lock rather than one per channel. alpha and beta write the same v$VERSION/ S3 keys for a given commit, so they are not independent: two channels deploying one commit at once could both pass the overwrite check and then race the same object. * fix(ci): divert symlink and mode changes out of the signed-commit backport path createCommitOnBranch can only express regular files: additions carry base64 content and land as mode 100644. Every book's docs/theme is a symlink (mode 120000) to the shared start-docs theme, and base64 reads through a symlink to its target — where that target is a directory it fails, the command substitution swallows the failure, and the mutation writes contents:"" over the symlink, replacing it with an empty regular file. Reproduced end to end with exit 0. The backport now inspects the staged diff's modes and diverts anything that is not a plain 100644 change to the git-based PR fallback, which represents modes natively. The fallback output carries a reason so the PR body can say whether a human needs to resolve conflicts or merely review — a symlink diversion applies cleanly and should not be described as a conflict. Also corrects projects/start-docs/ARCHITECTURE.md, which still documented deployment from master and omitted the start-wrt docs path. * fix(ci): represent renames as delete + add in the signed backport commit Rename detection is on by default, so a renamed page came back as one R row carrying two tab-separated paths. The additions filter emitted only the new path and the deletions filter emitted nothing, so the commit added the moved page and left the old one behind — and the mode check read the old path out of the wrong field. --no-renames on all three diffs renders a rename as a delete plus an add, which createCommitOnBranch expresses exactly, and gives every raw row a single path. * feat(release): promote debs to stable from alpha instead of a CI run The OS and StartWRT already release by promotion — CI publishes to a channel and the release moves that exact artifact onward — but the debs still rebuilt trust from a GitHub Actions run, so the packages reaching stable were not necessarily the ones testers had been running. `release start-cli` / `start-tunnel` / `start-registry` now stage from the alpha suite. The apt pool cannot identify a build on its own: dpkg-name rewrites each file to <package>_<version>_<arch>.deb, dropping the hash the build basename carries, and every master build of a version publishes under the same Version. So debian/build.sh records a Git-Hash control field, which dpkg-scanpackages carries into the Packages index — the right build is selected and verified before anything is downloaded. Alpha holds only the newest master build per package, so a hash mismatch means master moved past the commit being tagged. That fails rather than shipping bytes nobody soaked, which also enforces the existing rule to release from an up-to-date master. pull-gha stays as the repair path. start-cli's per-triple binaries are unchanged: they are published only as release assets, so there is no channel to promote them from and that half still needs the run that built the tagged commit. Both halves are pinned to it, so the two sources cannot disagree. * fix(release): verify the alpha suite's signature chain before promoting The promotion read an unsigned Packages index over HTTPS, trusted a self-asserted Git-Hash field, and downloaded the pool object without checking it. Two ways that fails: the pool key is stable across builds, so an alpha republish between the index read and the download swaps the bytes after the commit check has passed; and anyone able to write the bucket without holding the signing key could forge a stanza naming the target commit and have a releaser sign those bytes into stable. Alpha is signed with the CI key precisely so its consumers can verify it, and a releaser promoting into stable is one. The chain is now anchored end to end: InRelease against apt/start9-alpha.gpg, Packages against the hash the signed Release commits to, and each .deb against the hash that index commits to. Verified against a locally built signed fixture — a valid suite promotes, a tampered pool object is rejected, and a forged stanza is rejected because the attacker cannot re-sign Release. Staging is no longer partial. The marker is written and stale debs cleared before any fetch, and downloads land in a temp dir that is only moved into the release directory once every arch has verified, so an interrupted promotion cannot leave a mixed set beside a marker that disagrees with it. The mismatch error no longer prescribes an impossible remedy. Product workflows are path-filtered, so master routinely advances without producing a new build of a given product and "cut from an up-to-date master" cannot help. It now prints the COMMIT=<hash> invocation that tags the commit which actually produced the bytes. * feat(release): tag the commit alpha built instead of demanding one at HEAD A tag claims that a commit produced the artifact, so when the release promotes, alpha's build is what decides which commit to tag. Product workflows are path-filtered, so master routinely advances without rebuilding a given product; insisting on HEAD demanded a build that will never exist and left the operator to notice the mismatch and re-run with COMMIT= by hand. pull-alpha now adopts that commit, reports how far HEAD has moved on, and prints the git checkout for putting a tree on it. A new alpha-commit subcommand prints the same hash for scripting. Adoption is confined to a commit already in the current branch's history, and an explicit COMMIT is never overridden — that is someone asserting a different intent, so it fails instead. Every architecture must also report the same commit, so a half-published suite cannot produce a release assembled from two builds. * fix(release): fetch the alpha suite from the origin, not through a cache A promotion has to see the suite as it is now, and signature verification does not give that: a cached InRelease is still a validly signed InRelease, so every signature and hash check would pass while quietly promoting whatever build the edge happened to be holding. Signatures prove authenticity, not freshness. APT_BASE_URL already pointed at the S3 origin rather than the *.cdn.* host that apt/start9*.list gives clients, but nothing said so — it read as an inconsistency waiting to be tidied up. It is now documented as deliberate at the definition and in the root AGENTS.md, and every fetch in the promotion path (including pull_apt_debs) sends no-cache headers for any intermediary between the releaser and the origin. End users keep the CDN: apt is built to tolerate a stale mirror, a releaser signing bytes into stable is not. * fix(release): pin both halves to the adopted commit, and bound metadata age For start-cli the run was resolved and its binaries downloaded before promote_alpha_debs adopted alpha's commit. With HEAD past start-cli's last path-filtered build, a run for HEAD validated fine, adoption then moved COMMIT to the older commit and rewrote the marker to match, and cmd_tag tagged there — binaries from one commit, debs and tag from another, every check passing. The alpha commit is now determined and adopted first, so the run is validated against the commit actually being tagged. Release notes are read from the working tree, which an adopted commit can be behind. Where the changelog is identical the distinction is immaterial and nothing changes; where it differs the notes would describe a tree the tag does not point at, so the promotion stops and asks for the checkout. alpha-commit inspected only the first architecture and accepted a missing Git-Hash, so it could print a commit pull-alpha would then refuse. Both now go through one alpha_collect that verifies every architecture and requires them to agree. Finally, a signature proves authenticity but never freshness: someone able to write the bucket without the key could restore an older, validly signed Release plus pool and have it promoted. publish.sh now emits Valid-Until (30 days, configurable) and the promotion refuses metadata that is expired or carries no bound at all. * revert(release): drop Valid-Until, and fix portability and staging holes Valid-Until was the wrong answer and I should not have implemented it. It was applied to every suite including stable, so any month without a start-cli, start-tunnel or start-registry release would have failed `apt update` for every user. Even scoped to alpha it expires a rolling channel on a timer unrelated to release cadence, to close a narrow replay of a previously legitimate build. What actually guards that is showing the operator which commit is being tagged, where a replay appears as an unexpectedly old hash. Reverted from both files, with the reasoning recorded so it does not get reintroduced. The promotion also used `date -d` and bare `sha256sum`, both GNU-only, in a script that checks for bash >= 4.4 with a macOS hint and already carries a sha256sum/shasum fallback in checksum_block. Every alpha promotion would have failed closed on macOS. The date parse is gone with the revert; hashing now goes through a shared sha256_of helper. Staging cleared only *.deb, so a failed start-cli binary download left a fresh marker beside a previous attempt's debs, and the reverse partial set was equally reachable. Both halves are now cleared, and the marker written, before either is fetched. APT_BASE_URL follows the other endpoints in taking an override, which is what allows the promotion to be exercised end to end against a locally built signed suite rather than only through extracted functions. Usage text updated: it still described the pre-adoption failure mode. | 30 天前 | |
add encryption subkey to gpg key | 27 天前 | |
feat: support preferred external ports besides 443 (#3117) * docs: update preferred external port design in TODO * docs: add user-controlled public/private and port forward mapping to design * docs: overhaul interfaces page design with view/manage split and per-address controls * docs: move address enable/disable to overflow menu, add SSL indicator, defer UI placement decisions * chore: remove tor from startos core Tor is being moved from a built-in OS feature to a service. This removes the Arti-based Tor client, onion address management, hidden service creation, and all related code from the core backend, frontend, and SDK. - Delete core/src/net/tor/ module (~2060 lines) - Remove OnionAddress, TorSecretKey, TorController from all consumers - Remove HostnameInfo::Onion and HostAddress::Onion variants - Remove onion CRUD RPC endpoints and tor subcommand - Remove tor key handling from account and backup/restore - Remove ~12 tor-related Cargo dependencies (arti-client, torut, etc.) - Remove tor UI components, API methods, mock data, and routes - Remove OnionHostname and tor patterns/regexes from SDK - Add v0_4_0_alpha_20 database migration to strip onion data - Bump version to 0.4.0-alpha.20 * chore: flatten HostnameInfo from enum to struct HostnameInfo only had one variant (Ip) after removing Tor. Flatten it into a plain struct with fields gateway, public, hostname. Remove all kind === 'ip' type guards and narrowing across SDK, frontend, and container runtime. Update DB migration to strip the kind field. * chore: format RPCSpec.md markdown table * docs: update TODO.md with DerivedAddressInfo design, remove completed tor task * feat: implement preferred port allocation and per-address enable/disable - Add AvailablePorts::try_alloc() with SSL tracking (BTreeMap<u16, bool>) - Add DerivedAddressInfo on BindInfo with private_disabled/public_enabled/possible sets - Add Bindings wrapper with Map impl for patchdb indexed access - Flatten HostAddress from single-variant enum to struct - Replace set-gateway-enabled RPC with set-address-enabled - Remove hostname_info from Host; computed addresses now in BindInfo.addresses.possible - Compute possible addresses inline in NetServiceData::update() - Update DB migration, SDK types, frontend, and container-runtime * feat: replace InterfaceFilter with ForwardRequirements, add WildcardListener, complete alpha.20 bump - Replace DynInterfaceFilter with ForwardRequirements for per-IP forward precision with source-subnet iptables filtering for private forwards - Add WildcardListener (binds [::]:port) to replace the per-gateway NetworkInterfaceListener/SelfContainedNetworkInterfaceListener/ UpgradableListener infrastructure - Update forward-port script with src_subnet and excluded_src env vars - Remove unused filter types and listener infrastructure from gateway.rs - Add availablePorts migration (IdPool -> BTreeMap<u16, bool>) to alpha.20 - Complete version bump to 0.4.0-alpha.20 in SDK and web * outbound gateway support (#3120) * Multiple (#3111) * fix alerts i18n, fix status display, better, remove usb media, hide shutdown for install complete * trigger chnage detection for localize pipe and round out implementing localize pipe for consistency even though not needed * Fix PackageInfoShort to handle LocaleString on releaseNotes (#3112) * Fix PackageInfoShort to handle LocaleString on releaseNotes * fix: filter by target_version in get_matching_models and pass otherVersions from install * chore: add exver documentation for ai agents * frontend plus some be types --------- Co-authored-by: Aiden McClelland <3732071+dr-bonez@users.noreply.github.com> * feat: replace SourceFilter with IpNet, add policy routing, remove MASQUERADE * build ts types and fix i18n * fix license display in marketplace * wip refactor * chore: update ts bindings for preferred port design * feat: refactor NetService to watch DB and reconcile network state - NetService sync task now uses PatchDB DbWatch instead of being called directly after DB mutations - Read gateways from DB instead of network interface context when updating host addresses - gateway sync updates all host addresses in the DB - Add Watch<u64> channel for callers to wait on sync completion - Fix ts-rs codegen bug with #[ts(skip)] on flattened Plugin field - Update SDK getServiceInterface.ts for new HostnameInfo shape - Remove unnecessary HTTPS redirect in static_server.rs - Fix tunnel/api.rs to filter for WAN IPv4 address * re-arrange (#3123) * new service interfacee page * feat: add mdns hostname metadata variant and fix vhost routing - Add HostnameMetadata::Mdns variant to distinguish mDNS from private domains - Mark mDNS addresses as private (public: false) since mDNS is local-only - Fall back to null SNI entry when hostname not found in vhost mapping - Simplify public detection in ProxyTarget filter - Pass hostname to update_addresses for mDNS domain name generation * looking good * feat: add port_forwards field to Host for tracking gateway forwarding rules * update bindings for API types, add ARCHITECTURE (#3124) * update binding for API types, add ARCHITECTURE * translations * fix: add CONNMARK restore-mark to mangle OUTPUT chain The CONNMARK --restore-mark rule was only in PREROUTING, which handles forwarded packets. Locally-bound listeners (e.g. vhost) generate replies through the OUTPUT chain, where the fwmark was never restored. This caused response packets to route via the default table instead of back through the originating interface. * chore: reserialize db on equal version, update bindings and docs - Run de/ser roundtrip in pre_init even when db version matches, ensuring all #[serde(default)] fields are populated before any typed access - Add patchdb.md documentation for TypedDbWatch patterns - Update TS bindings for CheckPortParams, CheckPortRes, ifconfigUrl - Update CLAUDE.md docs with patchdb and component-level references * fix: include public gateways for IP-based addresses in vhost targets The server hostname vhost construction only collected private IPs, always setting public to empty. Public IP addresses (Ipv4/Ipv6 metadata with public=true) were never added to the vhost target's public gateway set, causing the vhost filter to reject public traffic for IP-based addresses. * fix: add TLS handshake timeout and fix accept loop deadlock Two issues in TlsListener::poll_accept: 1. No timeout on TLS handshakes: LazyConfigAcceptor waits indefinitely for ClientHello. Attackers that complete TCP handshake but never send TLS data create zombie futures in `in_progress` that never complete. Fix: wrap the entire handshake in tokio::time::timeout(15s). 2. Missing waker on new-connection pending path: when a TCP connection is accepted and the TLS handshake is pending, poll_accept returned Pending without calling wake_by_ref(). Since the TcpListener returned Ready (not Pending), no waker was registered for it. With edge- triggered epoll and no other wakeup source, the task sleeps forever and remaining connections in the kernel accept queue are never drained. Fix: add cx.waker().wake_by_ref() so the task immediately re-polls and continues draining the accept queue. * fix: switch BackgroundJobRunner from Vec to FuturesUnordered BackgroundJobRunner stored active jobs in a Vec<BoxFuture> and polled ALL of them on every wakeup — O(n) per poll. Since this runs in the same tokio::select! as the WebServer accept loop, polling overhead from active connections directly delayed acceptance of new connections. FuturesUnordered only polls woken futures — O(woken) instead of O(n). * chore: update bindings and use typed params for outbound gateway API * feat: per-service and default outbound gateway routing Add set-outbound-gateway RPC for packages and set-default-outbound RPC for the server, with policy routing enforcement via ip rules. Fix connmark restore to skip packets with existing fwmarks, add bridge subnet routes to per-interface tables, and fix squashfs path in update-image-local.sh. * refactor: manifest wraps PackageMetadata, move dependency_metadata to PackageVersionInfo Manifest now embeds PackageMetadata via #[serde(flatten)] instead of duplicating ~14 fields. icon and dependency_metadata moved from PackageMetadata to PackageVersionInfo since they are registry-enrichment data loaded from the S9PK archive. merge_with now returns errors on metadata/icon/dependency_metadata mismatches instead of silently ignoring them. * fix: replace .status() with .invoke() for iptables/ip commands Using .status() leaks stderr directly to system logs, causing noisy iptables error messages. Switch all networking CLI invocations to use .invoke() which captures stderr properly. For check-then-act patterns (iptables -C), use .invoke().await.is_err() instead of .status().await.map_or(false, |s| s.success()). * feat: add check-dns gateway endpoint and fix per-interface routing tables Add a `check-dns` RPC endpoint that verifies whether a gateway's DNS is properly configured for private domain resolution. Uses a three-tier check: direct match (DNS == server IP), TXT challenge probe (DNS on LAN), or failure (DNS off-subnet). Fix per-interface routing tables to clone all non-default routes from the main table instead of only the interface's own subnets. This preserves LAN reachability when the priority-75 catch-all overrides default routing. Filter out status-only flags (linkdown, dead) that are invalid for `ip route add`. * refactor: rename manifest metadata fields and improve error display Rename wrapperRepo→packageRepo, marketingSite→marketingUrl, docsUrl→docsUrls (array), remove supportSite. Add display_src/display_dbg helpers to Error. Fix DepInfo description type to LocaleString. Update web UI, SDK bindings, tests, and fixtures to match. Clean up cli_attach error handling and remove dead commented code. * chore: bump sdk version to 0.4.0-beta.49 * chore: add createTask decoupling TODO * chore: add TODO to clear service error state on install/update * round out dns check, dns server check, port forward check, and gateway port forwards * chore: add TODOs for URL plugins, NAT hairpinning, and start-tunnel OTA updates * version instead of os query param * interface row clickable again, bu now with a chevron! * feat: implement URL plugins with table/row actions and prefill support - Add URL plugin effects (register, export_url, clear_urls) in core - Add PluginHostnameInfo, HostnameMetadata::Plugin, and plugin registration types - Implement plugin URL table in web UI with tableAction button and rowAction overflow menus - Thread urlPluginMetadata (packageId, hostId, interfaceId, internalPort) as prefill to actions - Add prefill support to PackageActionData so metadata passes through form dialogs - Add i18n translations for plugin error messages - Clean up plugin URLs on package uninstall * feat: split row_actions into remove_action and overflow_actions for URL plugins * touch up URL plugins table * show table even when no addresses * feat: NAT hairpinning, DNS static servers, clear service error on install - Add POSTROUTING MASQUERADE rules for container and host hairpin NAT - Allow bridge subnet containers to reach private forwards via LAN IPs - Pass bridge_subnet env var from forward.rs to forward-port script - Use DB-configured static DNS servers in resolver with DB watcher - Fall back to resolv.conf servers when no static servers configured - Clear service error state when install/update completes successfully - Remove completed TODO items * feat: builder-style InputSpec API, prefill plumbing, and port forward fix - Add addKey() and add() builder methods to InputSpec with InputSpecTools - Move OuterType to last generic param on Value, List, and all dynamic methods - Plumb prefill through getActionInput end-to-end (core → container-runtime → SDK) - Filter port_forwards to enabled addresses only - Bump SDK to 0.4.0-beta.50 * fix: propagate host locale into LXC containers and write locale.conf * chore: remove completed URL plugins TODO * feat: OTA updates for start-tunnel via apt repository (untested) - Add apt repo publish script (build/apt/publish-deb.sh) for S3-hosted repo - Add apt source config and GPG key placeholder (apt/) - Add tunnel.update.check and tunnel.update.apply RPC endpoints - Wire up update API in tunnel frontend (api service + mock) - Uses systemd-run --scope to survive service restart during update * fix: publish script dpkg-name, s3cfg fallback, and --reinstall for apply * chore: replace OTA updates TODO with UI TODO for MattDHill * feat: add getOutboundGateway effect and simplify VersionGraph init/uninit Add getOutboundGateway effect across core, container-runtime, and SDK to let services query their effective outbound gateway with callback support. Remove preInstall/uninstall hooks from VersionGraph as they are no longer needed. * frontend start-tunnel updates * chore: remove completed TODO * feat: tor hidden service key migration * chore: migrate from ts-matches to zod across all TypeScript packages * feat(core): allow setting server hostname * send prefill for tasks and hide operations to hidden fields * fix(core): preserve plugin URLs across binding updates BindInfo::update was replacing addresses with a new DerivedAddressInfo that cleared the available set, wiping plugin-exported URLs whenever bind() was called. Also simplify update_addresses plugin preservation to use retain in place rather than collecting into a separate set. * minor cleanup from patch-db audit * clean up prefill flow * frontend support for setting and changing hostname * feat(core): refactor hostname to ServerHostnameInfo with name/hostname pair - Rename Hostname to ServerHostnameInfo, add name + hostname fields - Add set_hostname_rpc for changing hostname at runtime - Migrate alpha_20: generate serverInfo.name from hostname, delete ui.name - Extract gateway.rs helpers to fix rustfmt nesting depth issue - Add i18n key for hostname validation error - Update SDK bindings * add comments to everything potentially consumer facing (#3127) * add comments to everything potentially consumer facing * rework smtp --------- Co-authored-by: Aiden McClelland <3732071+dr-bonez@users.noreply.github.com> * implement server name * setup changes * clean up copy around addresses table * feat: add zod-deep-partial, partialValidator on InputSpec, and z.deepPartial re-export * fix: header color in zoom (#3128) * fix: merge version ranges when adding existing package signer (#3125) * fix: merge version ranges when adding existing package signer Previously, add_package_signer unconditionally inserted the new version range, overwriting any existing authorization for that signer. Now it OR-merges the new range with the existing one, so running signer add multiple times accumulates permissions rather than replacing them. * add --merge flag to registry package signer add Default behavior remains overwrite. When --merge is passed, the new version range is OR-merged with the existing one, allowing admins to accumulate permissions incrementally. * add missing attribute to TS type * make merge optional * upsert instead of insert * VersionRange::None on upsert * fix: header color in zoom --------- Co-authored-by: Dominion5254 <musashidisciple@proton.me> * update snake and add about this server to system general * chore: bump sdk to beta.53, wrap z.deepPartial with passthrough * reset instead of reset defaults * action failure show dialog * chore: bump sdk to beta.54, add device-info RPC, improve SDK abort handling and InputSpec filtering - Bump SDK version to 0.4.0-beta.54 - Add `server.device-info` RPC endpoint and `s9pk select` CLI command - Extract `HardwareRequirements::is_compatible()` method, reuse in registry filtering - Add `AbortedError` class with `muteUnhandled` flag, replace generic abort errors - Handle unhandled promise rejections in container-runtime with mute support - Improve `InputSpec.filter()` with `keepByDefault` param and boolean filter values - Accept readonly tuples in `CommandType` and `splitCommand` - Remove `sync_host` calls from host API handlers (binding/address changes) - Filter mDNS hostnames by secure gateway availability - Derive mDNS enabled state from LAN IPs in web UI - Add "Open UI" action to address table, disable mDNS toggle - Hide debug details in service error component - Update rpc-toolkit docs for no-params handlers * fix: add --no-nvram to efi grub-install to preserve built-in boot order * update snake * diable actions when in error state * chore: split out nvidia variant * misc bugfixes * create manage-release script (untested) * fix: preserve z namespace types for sdk consumers * sdk version bump * new checkPort types * multiple bugs and better port forward ux * fix link * chore: todos and formatting * fix build --------- Co-authored-by: Matt Hill <MattDHill@users.noreply.github.com> Co-authored-by: Matt Hill <mattnine@protonmail.com> Co-authored-by: Alex Inkin <alexander@inkin.ru> Co-authored-by: Dominion5254 <musashidisciple@proton.me> | 6 个月前 |