| feat(ui): add capture-ID consolidation to call flow Groups flow items that represent the same SIP message seen by multiple capture agents into a single row with an expandable sub-item list, reducing visual noise in multi-sensor deployments. - Compute runtime fingerprint (FNV-1a-32 of src/dst/payload) so no ingest-side changes are required - New filter toggle "Consolidate by fingerprint" with configurable time threshold; only enabled when the current flow has fingerprintable rows - Consolidated parent rows show a expand/collapse control and +N badge; sub-items render inline when expanded; one open at a time - Capture ID field added to the message detail dialog; shows "multiple" when clicking a consolidated parent row - Devcontainer added (devbase Dockerfile target, postCreate/postStart hooks, contributor guide) - Vite dev proxy target configurable via HOMER_PROXY_TARGET in .env.local - Safari compat: -webkit-backdrop-filter added to all four backdrop-filter declarations in CallFlow.css | 27 天前 |
| chore(git): block Cursor co-author trailers and track Cursor rules Ignore .cursor workspace junk but keep .cursor/rules in git; optional commit-msg hook rejects cursoragent@cursor.com in messages. Co-authored-by: Cursor <cursoragent@cursor.com> | 3 个月前 |
| fix(ci): build polyfill-glibc from a pinned corsix revision (#963) Stop running the unpinned ghcr.io/lmangani/polyfill-glibc-action:latest image against the release binary. CI now compiles corsix/polyfill-glibc at a fixed commit. | 12 天前 |
| Add homer project files | 3 个月前 |
| feat(storage): add Azure Blob Storage as a DuckLake backend (#983) * feat(storage): add Azure Blob Storage as a DuckLake backend Adds "azure" as a third storage type alongside "local" and "s3", usable both as a single-volume backend (storage.ducklake.data_path + azure block) and as a cold tier in storage_policy.volumes, following the existing S3 integration pattern (DuckDB's azure extension + CREATE SECRET, mirroring the aws extension). Auth precedence: connection_string > account_name+account_key (assembled into a connection string internally — DuckDB's azure secret has no direct account-key parameter) > credential_chain (resolves Managed Identity on an Azure VM automatically, no static credentials needed). Native move_engine "native" gets a real azureCopier using azure-sdk-for-go, wired into the existing mover.Copier abstraction alongside the S3 copier. Live-tested end-to-end against Azurite (examples/docker/docker-compose_azuredirect.yaml, new fixture) — single-volume write/read, tiered hot/cold attach, and both the duckdb and native move engines all verified with real data landing in the emulated blob container. That testing caught and fixed two latent path-handling bugs exposed by Azure's stricter blob semantics (one also affecting S3, previously masked by more forgiving path handling there): - ducklake.go: NewMultiTableWriter's mkdir guard checked isS3Path instead of the general IsRemoteLakeDataPath, so a remote data_path fell through to os.MkdirAll and failed. - cli_cmd.go: glob path construction used naive string concatenation instead of the existing JoinLakeDataPath helper, producing a double slash that broke Azure's blob listing. Managed Identity verified separately on a real Azure VM (system-assigned identity, Storage Blob Data Contributor role, zero static credentials), using Homer's actual duckdb-go binding against a real storage account. That testing surfaced an upstream duckdb-azure bug: its bundled libcurl only checks the RedHat-family CA bundle path, which Debian/Ubuntu never creates, so every HTTPS request to *.blob.core.windows.net fails with "Problem with the SSL CA cert" — for every auth method, not just Managed Identity (Azurite testing never caught this since it runs over plain HTTP). Filed upstream: https://github.com/duckdb/duckdb-azure/issues/185. Worked around here with EnsureAzureCACertPath (called everywhere Homer loads the azure extension) plus the same symlink baked into the Dockerfile as defense-in-depth; both point at the upstream issue and should be removed once it's fixed and released. * fix(storage): refresh Azure credential_chain secret on the writer path Addresses PR review must-fix #1 (github.com/sipcapture/homer/pull/983): Azure secrets have no REFRESH auto, so a PROVIDER credential_chain secret's Managed Identity token (~1h IMDS lifetime) was fixed at CREATE SECRET time on the single-volume writer and node attachVolume paths and never refreshed — same class of bug as #980, but for Azure. The tiered storage path already handled this correctly. - Export BuildAzureSecretSQL / UsesAzureCredentialChain (were already shared package-private helpers in tiered_storage.go) so the writer and node paths can stop duplicating the CREATE SECRET SQL construction. - EnsureWriterAzureSecret (tuning.go) and node.go's attachVolume now both delegate to BuildAzureSecretSQL instead of re-implementing the connection-string synthesis inline. - New CompactionAzureClient + CompactionService.ensureAzureClientSettings, mirroring the existing S3 CompactionS3Client / ensureS3ClientSettings, called at the same three maintenance call sites. Only recreates the secret when it is actually a credential_chain one; static-key secrets are left untouched since they do not expire. - MultiTableWriter.flushLoop also recreates the secret on its own 20-minute cadence (ensureAzureSecretFresh), independent of compaction. Compaction-only refresh left a gap flagged in review: with compaction.enable: false, flush is the only periodic operation left on a single-volume Azure writer, so it needs its own refresh rather than relying on compaction's cycle. Scoped to the single-volume writer path only, per review discussion: the node's own read-path attach has the identical missing-refresh gap (and S3 has the same gap there too, pre-existing and not introduced by Azure support) — left alone for now as a separate follow-up rather than widened here, and commented as such at the call site. * fix(cli): stop loading Azure extension on S3-only data paths Addresses PR review must-fix #3 (github.com/sipcapture/homer/pull/983): openDuckLakeReadOnly gated both the S3 secret setup and the Azure LOAD+secret setup on the same IsRemoteLakeDataPath check, so any S3-only homer cli invocation also attempted LOAD azure and printed a spurious "failed to load azure extension" warning. The writer path already branched isS3Path vs isAzurePath correctly; the CLI did not. - Export IsS3Path / IsAzurePath (were already package-private in ducklake.go) so cli_cmd.go can branch the two setups independently. - openDuckLakeReadOnly now gates S3 secret creation on IsS3Path and Azure LOAD+secret creation on IsAzurePath, instead of both on the generic IsRemoteLakeDataPath check. - New regression tests exercising openDuckLakeReadOnly end-to-end for both an S3-only and an Azure-only data_path, asserting via duckdb_secrets() that only the matching secret gets created. The Azure-side test intentionally uses a fake CONNECTION_STRING rather than a bare account_name: a credential_chain secret in a sandboxed test environment (no IMDS route) makes the immediately-following parquet-view glob() attempt Managed Identity resolution and hang for ~10 minutes instead of failing fast — found by writing the test the other way first and watching it time out. * fix(storage): honor a custom Azure Blob endpoint for account_key and credential_chain Addresses PR review must-fix #2 (github.com/sipcapture/homer/pull/983): account_key and the native mover's credential_chain branch both hardcoded the public-cloud Blob endpoint, so neither could reach Azurite, Gov/China cloud, or any custom endpoint — only a raw connection_string escaped this. - New azure_endpoint config field, threaded through the writer/node/ tiered/compaction secret builders and mover.AzureConfig. - BuildAzureConnectionString (extracted from BuildAzureSecretSQL) injects BlobEndpoint when set, else the existing public-cloud default. credential_chain now also passes ENDPOINT when configured (verified directly against DuckDB v1.5.5 that both are accepted). - mover/azure.go's account_key branch now builds a connection string and uses NewClientFromConnectionString instead of a hardcoded URL, via a small self-contained copy of the same helper (mover can't import ducklake — ducklake already imports mover). - Tests cover three distinct endpoint URL shapes: Azurite (account name as a path segment on a fixed host:port), Gov/China cloud, and the standard public cloud's own DNS-subdomain shape passed explicitly. Docs updated. * fix(storage): treat CA cert symlink EEXIST as success, document manual fallback Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): EnsureAzureCACertPath warned whenever os.Symlink failed for any reason, including the symlink already existing — treat that specific case as success instead, per review ask. - Check os.IsExist on the symlink error and return silently. - Documented the manual fallback for deployments where Homer can't create /etc/pki/tls/certs itself (bare host package, docker run as non-root). - Fixed comments/docs claiming this only affects blob.core.windows.net — it's any Azure Blob HTTPS endpoint (Gov/China cloud, custom too). * chore(deps): go mod tidy — azblob/azidentity as direct dependencies Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): mover/azure.go imports azidentity and azblob directly, but go.mod listed both as // indirect. go mod tidy also dropped a few now-unused go.sum entries. * test(storage): cover az:// and azure:// in JoinLakeDataPath/joinLake Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): only the S3 scheme was tested for the Unix filepath.Join collapse bug ("s3://" -> "s3:/") that this PR's mkdir/glob fixes guard against. Adds the same coverage for both Azure scheme aliases, in both the ducklake and mover packages. * docs: add Azure examples to ENVIRONMENT_VARIABLES.md Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): the doc only showed S3 field examples for the dots-to-underscores env var naming mechanism. Viper maps HOMER_STORAGE_DUCKLAKE_AZURE_* correctly regardless, but an operator reading the doc had nothing to go on for the Azure field names. Adds the same style of example for azure_account_name/account_key/ connection_string/endpoint. * test(writer,cli): cover Azure config wiring when only account_name is set Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): no test proved Azure config is copied through when only account_name is set (Managed Identity — no key, no connection string). A wrong gate could silently drop it, leaving the writer with no Azure secret. Extracted applyAzureDuckLakeConfig out of writer.go's New() so the gate is testable without real DuckLake I/O. cli_cmd.go's duckLakeConfigFromModular was already pure — just needed the test case. * fix(storage): verify size and set upload parity options in azureCopier.Copy Addresses PR review nit (github.com/sipcapture/homer/pull/983): azureCopier.Copy ignored the size parameter entirely, unlike LocalCopier which verifies the copied byte count matches the catalog's recorded size. Also had no BlockSize/Concurrency set, unlike the S3 uploader. Unlike s3Copier, which declares size via PutObjectInput.ContentLength, UploadFileOptions has no equivalent field. - Check the source file's on-disk size against the catalog's recorded size before uploading instead, catching a truncated/resized source up front. - Set BlockSize/Concurrency to the same values as s3Copier's manager.Uploader (8 MiB, concurrency 3). * fix(storage): fail fast when az:// data_path has no Azure config Addresses PR review nit (github.com/sipcapture/homer/pull/983): EnsureWriterAzureSecret silently no-opped when all Azure fields were empty, so a misconfigured az:// data_path failed later with an opaque ATTACH error instead of a clear message. Every caller already gates on Azure being meant to be configured, so this fixes all 4 call sites at once. * fix(config): copy AzureEndpoint in EnsureNodeDuckLakeVolumes The legacy flat-config synthesis path copied account name, key, and connection string into the default volume but dropped Endpoint, unlike the S3 branch one block above it. Azurite and Gov/China cloud setups relying on the legacy data_path/azure config (not storage_policy.volumes) silently lost their endpoint override. * fix(storage): lock catalogMu around Azure secret refresh flushLoop calls ensureAzureSecretFresh before flushAll, while flushQueueWorker can be mid-INSERT on the same DuckDB connection. DROP+CREATE SECRET now runs under catalogMu, matching how compaction already refreshes credential_chain secrets under the same lock. * fix(node): periodically refresh Azure credential_chain secrets A standalone node attaches volumes once and only revisits them via refreshCatalog, which only runs on a timer when FlightSQL is enabled. An Azure credential_chain secret (Managed Identity via IMDS) is only valid for its token's lifetime, so without FlightSQL it would go stale and queries would start failing after the node had been running for a while. Adds an independent 20-minute ticker, gated to standalone nodes with at least one credential_chain Azure volume, that recreates just the Azure secrets without a full catalog reconnect. | 5 天前 |
| feat(storage): add Azure Blob Storage as a DuckLake backend (#983) * feat(storage): add Azure Blob Storage as a DuckLake backend Adds "azure" as a third storage type alongside "local" and "s3", usable both as a single-volume backend (storage.ducklake.data_path + azure block) and as a cold tier in storage_policy.volumes, following the existing S3 integration pattern (DuckDB's azure extension + CREATE SECRET, mirroring the aws extension). Auth precedence: connection_string > account_name+account_key (assembled into a connection string internally — DuckDB's azure secret has no direct account-key parameter) > credential_chain (resolves Managed Identity on an Azure VM automatically, no static credentials needed). Native move_engine "native" gets a real azureCopier using azure-sdk-for-go, wired into the existing mover.Copier abstraction alongside the S3 copier. Live-tested end-to-end against Azurite (examples/docker/docker-compose_azuredirect.yaml, new fixture) — single-volume write/read, tiered hot/cold attach, and both the duckdb and native move engines all verified with real data landing in the emulated blob container. That testing caught and fixed two latent path-handling bugs exposed by Azure's stricter blob semantics (one also affecting S3, previously masked by more forgiving path handling there): - ducklake.go: NewMultiTableWriter's mkdir guard checked isS3Path instead of the general IsRemoteLakeDataPath, so a remote data_path fell through to os.MkdirAll and failed. - cli_cmd.go: glob path construction used naive string concatenation instead of the existing JoinLakeDataPath helper, producing a double slash that broke Azure's blob listing. Managed Identity verified separately on a real Azure VM (system-assigned identity, Storage Blob Data Contributor role, zero static credentials), using Homer's actual duckdb-go binding against a real storage account. That testing surfaced an upstream duckdb-azure bug: its bundled libcurl only checks the RedHat-family CA bundle path, which Debian/Ubuntu never creates, so every HTTPS request to *.blob.core.windows.net fails with "Problem with the SSL CA cert" — for every auth method, not just Managed Identity (Azurite testing never caught this since it runs over plain HTTP). Filed upstream: https://github.com/duckdb/duckdb-azure/issues/185. Worked around here with EnsureAzureCACertPath (called everywhere Homer loads the azure extension) plus the same symlink baked into the Dockerfile as defense-in-depth; both point at the upstream issue and should be removed once it's fixed and released. * fix(storage): refresh Azure credential_chain secret on the writer path Addresses PR review must-fix #1 (github.com/sipcapture/homer/pull/983): Azure secrets have no REFRESH auto, so a PROVIDER credential_chain secret's Managed Identity token (~1h IMDS lifetime) was fixed at CREATE SECRET time on the single-volume writer and node attachVolume paths and never refreshed — same class of bug as #980, but for Azure. The tiered storage path already handled this correctly. - Export BuildAzureSecretSQL / UsesAzureCredentialChain (were already shared package-private helpers in tiered_storage.go) so the writer and node paths can stop duplicating the CREATE SECRET SQL construction. - EnsureWriterAzureSecret (tuning.go) and node.go's attachVolume now both delegate to BuildAzureSecretSQL instead of re-implementing the connection-string synthesis inline. - New CompactionAzureClient + CompactionService.ensureAzureClientSettings, mirroring the existing S3 CompactionS3Client / ensureS3ClientSettings, called at the same three maintenance call sites. Only recreates the secret when it is actually a credential_chain one; static-key secrets are left untouched since they do not expire. - MultiTableWriter.flushLoop also recreates the secret on its own 20-minute cadence (ensureAzureSecretFresh), independent of compaction. Compaction-only refresh left a gap flagged in review: with compaction.enable: false, flush is the only periodic operation left on a single-volume Azure writer, so it needs its own refresh rather than relying on compaction's cycle. Scoped to the single-volume writer path only, per review discussion: the node's own read-path attach has the identical missing-refresh gap (and S3 has the same gap there too, pre-existing and not introduced by Azure support) — left alone for now as a separate follow-up rather than widened here, and commented as such at the call site. * fix(cli): stop loading Azure extension on S3-only data paths Addresses PR review must-fix #3 (github.com/sipcapture/homer/pull/983): openDuckLakeReadOnly gated both the S3 secret setup and the Azure LOAD+secret setup on the same IsRemoteLakeDataPath check, so any S3-only homer cli invocation also attempted LOAD azure and printed a spurious "failed to load azure extension" warning. The writer path already branched isS3Path vs isAzurePath correctly; the CLI did not. - Export IsS3Path / IsAzurePath (were already package-private in ducklake.go) so cli_cmd.go can branch the two setups independently. - openDuckLakeReadOnly now gates S3 secret creation on IsS3Path and Azure LOAD+secret creation on IsAzurePath, instead of both on the generic IsRemoteLakeDataPath check. - New regression tests exercising openDuckLakeReadOnly end-to-end for both an S3-only and an Azure-only data_path, asserting via duckdb_secrets() that only the matching secret gets created. The Azure-side test intentionally uses a fake CONNECTION_STRING rather than a bare account_name: a credential_chain secret in a sandboxed test environment (no IMDS route) makes the immediately-following parquet-view glob() attempt Managed Identity resolution and hang for ~10 minutes instead of failing fast — found by writing the test the other way first and watching it time out. * fix(storage): honor a custom Azure Blob endpoint for account_key and credential_chain Addresses PR review must-fix #2 (github.com/sipcapture/homer/pull/983): account_key and the native mover's credential_chain branch both hardcoded the public-cloud Blob endpoint, so neither could reach Azurite, Gov/China cloud, or any custom endpoint — only a raw connection_string escaped this. - New azure_endpoint config field, threaded through the writer/node/ tiered/compaction secret builders and mover.AzureConfig. - BuildAzureConnectionString (extracted from BuildAzureSecretSQL) injects BlobEndpoint when set, else the existing public-cloud default. credential_chain now also passes ENDPOINT when configured (verified directly against DuckDB v1.5.5 that both are accepted). - mover/azure.go's account_key branch now builds a connection string and uses NewClientFromConnectionString instead of a hardcoded URL, via a small self-contained copy of the same helper (mover can't import ducklake — ducklake already imports mover). - Tests cover three distinct endpoint URL shapes: Azurite (account name as a path segment on a fixed host:port), Gov/China cloud, and the standard public cloud's own DNS-subdomain shape passed explicitly. Docs updated. * fix(storage): treat CA cert symlink EEXIST as success, document manual fallback Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): EnsureAzureCACertPath warned whenever os.Symlink failed for any reason, including the symlink already existing — treat that specific case as success instead, per review ask. - Check os.IsExist on the symlink error and return silently. - Documented the manual fallback for deployments where Homer can't create /etc/pki/tls/certs itself (bare host package, docker run as non-root). - Fixed comments/docs claiming this only affects blob.core.windows.net — it's any Azure Blob HTTPS endpoint (Gov/China cloud, custom too). * chore(deps): go mod tidy — azblob/azidentity as direct dependencies Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): mover/azure.go imports azidentity and azblob directly, but go.mod listed both as // indirect. go mod tidy also dropped a few now-unused go.sum entries. * test(storage): cover az:// and azure:// in JoinLakeDataPath/joinLake Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): only the S3 scheme was tested for the Unix filepath.Join collapse bug ("s3://" -> "s3:/") that this PR's mkdir/glob fixes guard against. Adds the same coverage for both Azure scheme aliases, in both the ducklake and mover packages. * docs: add Azure examples to ENVIRONMENT_VARIABLES.md Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): the doc only showed S3 field examples for the dots-to-underscores env var naming mechanism. Viper maps HOMER_STORAGE_DUCKLAKE_AZURE_* correctly regardless, but an operator reading the doc had nothing to go on for the Azure field names. Adds the same style of example for azure_account_name/account_key/ connection_string/endpoint. * test(writer,cli): cover Azure config wiring when only account_name is set Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): no test proved Azure config is copied through when only account_name is set (Managed Identity — no key, no connection string). A wrong gate could silently drop it, leaving the writer with no Azure secret. Extracted applyAzureDuckLakeConfig out of writer.go's New() so the gate is testable without real DuckLake I/O. cli_cmd.go's duckLakeConfigFromModular was already pure — just needed the test case. * fix(storage): verify size and set upload parity options in azureCopier.Copy Addresses PR review nit (github.com/sipcapture/homer/pull/983): azureCopier.Copy ignored the size parameter entirely, unlike LocalCopier which verifies the copied byte count matches the catalog's recorded size. Also had no BlockSize/Concurrency set, unlike the S3 uploader. Unlike s3Copier, which declares size via PutObjectInput.ContentLength, UploadFileOptions has no equivalent field. - Check the source file's on-disk size against the catalog's recorded size before uploading instead, catching a truncated/resized source up front. - Set BlockSize/Concurrency to the same values as s3Copier's manager.Uploader (8 MiB, concurrency 3). * fix(storage): fail fast when az:// data_path has no Azure config Addresses PR review nit (github.com/sipcapture/homer/pull/983): EnsureWriterAzureSecret silently no-opped when all Azure fields were empty, so a misconfigured az:// data_path failed later with an opaque ATTACH error instead of a clear message. Every caller already gates on Azure being meant to be configured, so this fixes all 4 call sites at once. * fix(config): copy AzureEndpoint in EnsureNodeDuckLakeVolumes The legacy flat-config synthesis path copied account name, key, and connection string into the default volume but dropped Endpoint, unlike the S3 branch one block above it. Azurite and Gov/China cloud setups relying on the legacy data_path/azure config (not storage_policy.volumes) silently lost their endpoint override. * fix(storage): lock catalogMu around Azure secret refresh flushLoop calls ensureAzureSecretFresh before flushAll, while flushQueueWorker can be mid-INSERT on the same DuckDB connection. DROP+CREATE SECRET now runs under catalogMu, matching how compaction already refreshes credential_chain secrets under the same lock. * fix(node): periodically refresh Azure credential_chain secrets A standalone node attaches volumes once and only revisits them via refreshCatalog, which only runs on a timer when FlightSQL is enabled. An Azure credential_chain secret (Managed Identity via IMDS) is only valid for its token's lifetime, so without FlightSQL it would go stale and queries would start failing after the node had been running for a while. Adds an independent 20-minute ticker, gated to standalone nodes with at least one credential_chain Azure volume, that recreates just the Azure secrets without a full catalog reconnect. | 5 天前 |
| feat(storage): add Azure Blob Storage as a DuckLake backend (#983) * feat(storage): add Azure Blob Storage as a DuckLake backend Adds "azure" as a third storage type alongside "local" and "s3", usable both as a single-volume backend (storage.ducklake.data_path + azure block) and as a cold tier in storage_policy.volumes, following the existing S3 integration pattern (DuckDB's azure extension + CREATE SECRET, mirroring the aws extension). Auth precedence: connection_string > account_name+account_key (assembled into a connection string internally — DuckDB's azure secret has no direct account-key parameter) > credential_chain (resolves Managed Identity on an Azure VM automatically, no static credentials needed). Native move_engine "native" gets a real azureCopier using azure-sdk-for-go, wired into the existing mover.Copier abstraction alongside the S3 copier. Live-tested end-to-end against Azurite (examples/docker/docker-compose_azuredirect.yaml, new fixture) — single-volume write/read, tiered hot/cold attach, and both the duckdb and native move engines all verified with real data landing in the emulated blob container. That testing caught and fixed two latent path-handling bugs exposed by Azure's stricter blob semantics (one also affecting S3, previously masked by more forgiving path handling there): - ducklake.go: NewMultiTableWriter's mkdir guard checked isS3Path instead of the general IsRemoteLakeDataPath, so a remote data_path fell through to os.MkdirAll and failed. - cli_cmd.go: glob path construction used naive string concatenation instead of the existing JoinLakeDataPath helper, producing a double slash that broke Azure's blob listing. Managed Identity verified separately on a real Azure VM (system-assigned identity, Storage Blob Data Contributor role, zero static credentials), using Homer's actual duckdb-go binding against a real storage account. That testing surfaced an upstream duckdb-azure bug: its bundled libcurl only checks the RedHat-family CA bundle path, which Debian/Ubuntu never creates, so every HTTPS request to *.blob.core.windows.net fails with "Problem with the SSL CA cert" — for every auth method, not just Managed Identity (Azurite testing never caught this since it runs over plain HTTP). Filed upstream: https://github.com/duckdb/duckdb-azure/issues/185. Worked around here with EnsureAzureCACertPath (called everywhere Homer loads the azure extension) plus the same symlink baked into the Dockerfile as defense-in-depth; both point at the upstream issue and should be removed once it's fixed and released. * fix(storage): refresh Azure credential_chain secret on the writer path Addresses PR review must-fix #1 (github.com/sipcapture/homer/pull/983): Azure secrets have no REFRESH auto, so a PROVIDER credential_chain secret's Managed Identity token (~1h IMDS lifetime) was fixed at CREATE SECRET time on the single-volume writer and node attachVolume paths and never refreshed — same class of bug as #980, but for Azure. The tiered storage path already handled this correctly. - Export BuildAzureSecretSQL / UsesAzureCredentialChain (were already shared package-private helpers in tiered_storage.go) so the writer and node paths can stop duplicating the CREATE SECRET SQL construction. - EnsureWriterAzureSecret (tuning.go) and node.go's attachVolume now both delegate to BuildAzureSecretSQL instead of re-implementing the connection-string synthesis inline. - New CompactionAzureClient + CompactionService.ensureAzureClientSettings, mirroring the existing S3 CompactionS3Client / ensureS3ClientSettings, called at the same three maintenance call sites. Only recreates the secret when it is actually a credential_chain one; static-key secrets are left untouched since they do not expire. - MultiTableWriter.flushLoop also recreates the secret on its own 20-minute cadence (ensureAzureSecretFresh), independent of compaction. Compaction-only refresh left a gap flagged in review: with compaction.enable: false, flush is the only periodic operation left on a single-volume Azure writer, so it needs its own refresh rather than relying on compaction's cycle. Scoped to the single-volume writer path only, per review discussion: the node's own read-path attach has the identical missing-refresh gap (and S3 has the same gap there too, pre-existing and not introduced by Azure support) — left alone for now as a separate follow-up rather than widened here, and commented as such at the call site. * fix(cli): stop loading Azure extension on S3-only data paths Addresses PR review must-fix #3 (github.com/sipcapture/homer/pull/983): openDuckLakeReadOnly gated both the S3 secret setup and the Azure LOAD+secret setup on the same IsRemoteLakeDataPath check, so any S3-only homer cli invocation also attempted LOAD azure and printed a spurious "failed to load azure extension" warning. The writer path already branched isS3Path vs isAzurePath correctly; the CLI did not. - Export IsS3Path / IsAzurePath (were already package-private in ducklake.go) so cli_cmd.go can branch the two setups independently. - openDuckLakeReadOnly now gates S3 secret creation on IsS3Path and Azure LOAD+secret creation on IsAzurePath, instead of both on the generic IsRemoteLakeDataPath check. - New regression tests exercising openDuckLakeReadOnly end-to-end for both an S3-only and an Azure-only data_path, asserting via duckdb_secrets() that only the matching secret gets created. The Azure-side test intentionally uses a fake CONNECTION_STRING rather than a bare account_name: a credential_chain secret in a sandboxed test environment (no IMDS route) makes the immediately-following parquet-view glob() attempt Managed Identity resolution and hang for ~10 minutes instead of failing fast — found by writing the test the other way first and watching it time out. * fix(storage): honor a custom Azure Blob endpoint for account_key and credential_chain Addresses PR review must-fix #2 (github.com/sipcapture/homer/pull/983): account_key and the native mover's credential_chain branch both hardcoded the public-cloud Blob endpoint, so neither could reach Azurite, Gov/China cloud, or any custom endpoint — only a raw connection_string escaped this. - New azure_endpoint config field, threaded through the writer/node/ tiered/compaction secret builders and mover.AzureConfig. - BuildAzureConnectionString (extracted from BuildAzureSecretSQL) injects BlobEndpoint when set, else the existing public-cloud default. credential_chain now also passes ENDPOINT when configured (verified directly against DuckDB v1.5.5 that both are accepted). - mover/azure.go's account_key branch now builds a connection string and uses NewClientFromConnectionString instead of a hardcoded URL, via a small self-contained copy of the same helper (mover can't import ducklake — ducklake already imports mover). - Tests cover three distinct endpoint URL shapes: Azurite (account name as a path segment on a fixed host:port), Gov/China cloud, and the standard public cloud's own DNS-subdomain shape passed explicitly. Docs updated. * fix(storage): treat CA cert symlink EEXIST as success, document manual fallback Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): EnsureAzureCACertPath warned whenever os.Symlink failed for any reason, including the symlink already existing — treat that specific case as success instead, per review ask. - Check os.IsExist on the symlink error and return silently. - Documented the manual fallback for deployments where Homer can't create /etc/pki/tls/certs itself (bare host package, docker run as non-root). - Fixed comments/docs claiming this only affects blob.core.windows.net — it's any Azure Blob HTTPS endpoint (Gov/China cloud, custom too). * chore(deps): go mod tidy — azblob/azidentity as direct dependencies Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): mover/azure.go imports azidentity and azblob directly, but go.mod listed both as // indirect. go mod tidy also dropped a few now-unused go.sum entries. * test(storage): cover az:// and azure:// in JoinLakeDataPath/joinLake Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): only the S3 scheme was tested for the Unix filepath.Join collapse bug ("s3://" -> "s3:/") that this PR's mkdir/glob fixes guard against. Adds the same coverage for both Azure scheme aliases, in both the ducklake and mover packages. * docs: add Azure examples to ENVIRONMENT_VARIABLES.md Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): the doc only showed S3 field examples for the dots-to-underscores env var naming mechanism. Viper maps HOMER_STORAGE_DUCKLAKE_AZURE_* correctly regardless, but an operator reading the doc had nothing to go on for the Azure field names. Adds the same style of example for azure_account_name/account_key/ connection_string/endpoint. * test(writer,cli): cover Azure config wiring when only account_name is set Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): no test proved Azure config is copied through when only account_name is set (Managed Identity — no key, no connection string). A wrong gate could silently drop it, leaving the writer with no Azure secret. Extracted applyAzureDuckLakeConfig out of writer.go's New() so the gate is testable without real DuckLake I/O. cli_cmd.go's duckLakeConfigFromModular was already pure — just needed the test case. * fix(storage): verify size and set upload parity options in azureCopier.Copy Addresses PR review nit (github.com/sipcapture/homer/pull/983): azureCopier.Copy ignored the size parameter entirely, unlike LocalCopier which verifies the copied byte count matches the catalog's recorded size. Also had no BlockSize/Concurrency set, unlike the S3 uploader. Unlike s3Copier, which declares size via PutObjectInput.ContentLength, UploadFileOptions has no equivalent field. - Check the source file's on-disk size against the catalog's recorded size before uploading instead, catching a truncated/resized source up front. - Set BlockSize/Concurrency to the same values as s3Copier's manager.Uploader (8 MiB, concurrency 3). * fix(storage): fail fast when az:// data_path has no Azure config Addresses PR review nit (github.com/sipcapture/homer/pull/983): EnsureWriterAzureSecret silently no-opped when all Azure fields were empty, so a misconfigured az:// data_path failed later with an opaque ATTACH error instead of a clear message. Every caller already gates on Azure being meant to be configured, so this fixes all 4 call sites at once. * fix(config): copy AzureEndpoint in EnsureNodeDuckLakeVolumes The legacy flat-config synthesis path copied account name, key, and connection string into the default volume but dropped Endpoint, unlike the S3 branch one block above it. Azurite and Gov/China cloud setups relying on the legacy data_path/azure config (not storage_policy.volumes) silently lost their endpoint override. * fix(storage): lock catalogMu around Azure secret refresh flushLoop calls ensureAzureSecretFresh before flushAll, while flushQueueWorker can be mid-INSERT on the same DuckDB connection. DROP+CREATE SECRET now runs under catalogMu, matching how compaction already refreshes credential_chain secrets under the same lock. * fix(node): periodically refresh Azure credential_chain secrets A standalone node attaches volumes once and only revisits them via refreshCatalog, which only runs on a timer when FlightSQL is enabled. An Azure credential_chain secret (Managed Identity via IMDS) is only valid for its token's lifetime, so without FlightSQL it would go stale and queries would start failing after the node had been running for a while. Adds an independent 20-minute ticker, gated to standalone nodes with at least one credential_chain Azure volume, that recreates just the Azure secrets without a full catalog reconnect. | 5 天前 |
| fix(ui): stop Events tab throwing on every open (11.0.336) Rename shadowed `items` in loadEvents so the Events tab no longer throws ReferenceError (TDZ) on every open. Regressed in 11.0.312 (cb0e356). | 5 天前 |
| feat(ui): add capture-ID consolidation to call flow Groups flow items that represent the same SIP message seen by multiple capture agents into a single row with an expandable sub-item list, reducing visual noise in multi-sensor deployments. - Compute runtime fingerprint (FNV-1a-32 of src/dst/payload) so no ingest-side changes are required - New filter toggle "Consolidate by fingerprint" with configurable time threshold; only enabled when the current flow has fingerprintable rows - Consolidated parent rows show a expand/collapse control and +N badge; sub-items render inline when expanded; one open at a time - Capture ID field added to the message detail dialog; shows "multiple" when clicking a consolidated parent row - Devcontainer added (devbase Dockerfile target, postCreate/postStart hooks, contributor guide) - Vite dev proxy target configurable via HOMER_PROXY_TARGET in .env.local - Safari compat: -webkit-backdrop-filter added to all four backdrop-filter declarations in CallFlow.css | 27 天前 |
| feat(storage): add Azure Blob Storage as a DuckLake backend (#983) * feat(storage): add Azure Blob Storage as a DuckLake backend Adds "azure" as a third storage type alongside "local" and "s3", usable both as a single-volume backend (storage.ducklake.data_path + azure block) and as a cold tier in storage_policy.volumes, following the existing S3 integration pattern (DuckDB's azure extension + CREATE SECRET, mirroring the aws extension). Auth precedence: connection_string > account_name+account_key (assembled into a connection string internally — DuckDB's azure secret has no direct account-key parameter) > credential_chain (resolves Managed Identity on an Azure VM automatically, no static credentials needed). Native move_engine "native" gets a real azureCopier using azure-sdk-for-go, wired into the existing mover.Copier abstraction alongside the S3 copier. Live-tested end-to-end against Azurite (examples/docker/docker-compose_azuredirect.yaml, new fixture) — single-volume write/read, tiered hot/cold attach, and both the duckdb and native move engines all verified with real data landing in the emulated blob container. That testing caught and fixed two latent path-handling bugs exposed by Azure's stricter blob semantics (one also affecting S3, previously masked by more forgiving path handling there): - ducklake.go: NewMultiTableWriter's mkdir guard checked isS3Path instead of the general IsRemoteLakeDataPath, so a remote data_path fell through to os.MkdirAll and failed. - cli_cmd.go: glob path construction used naive string concatenation instead of the existing JoinLakeDataPath helper, producing a double slash that broke Azure's blob listing. Managed Identity verified separately on a real Azure VM (system-assigned identity, Storage Blob Data Contributor role, zero static credentials), using Homer's actual duckdb-go binding against a real storage account. That testing surfaced an upstream duckdb-azure bug: its bundled libcurl only checks the RedHat-family CA bundle path, which Debian/Ubuntu never creates, so every HTTPS request to *.blob.core.windows.net fails with "Problem with the SSL CA cert" — for every auth method, not just Managed Identity (Azurite testing never caught this since it runs over plain HTTP). Filed upstream: https://github.com/duckdb/duckdb-azure/issues/185. Worked around here with EnsureAzureCACertPath (called everywhere Homer loads the azure extension) plus the same symlink baked into the Dockerfile as defense-in-depth; both point at the upstream issue and should be removed once it's fixed and released. * fix(storage): refresh Azure credential_chain secret on the writer path Addresses PR review must-fix #1 (github.com/sipcapture/homer/pull/983): Azure secrets have no REFRESH auto, so a PROVIDER credential_chain secret's Managed Identity token (~1h IMDS lifetime) was fixed at CREATE SECRET time on the single-volume writer and node attachVolume paths and never refreshed — same class of bug as #980, but for Azure. The tiered storage path already handled this correctly. - Export BuildAzureSecretSQL / UsesAzureCredentialChain (were already shared package-private helpers in tiered_storage.go) so the writer and node paths can stop duplicating the CREATE SECRET SQL construction. - EnsureWriterAzureSecret (tuning.go) and node.go's attachVolume now both delegate to BuildAzureSecretSQL instead of re-implementing the connection-string synthesis inline. - New CompactionAzureClient + CompactionService.ensureAzureClientSettings, mirroring the existing S3 CompactionS3Client / ensureS3ClientSettings, called at the same three maintenance call sites. Only recreates the secret when it is actually a credential_chain one; static-key secrets are left untouched since they do not expire. - MultiTableWriter.flushLoop also recreates the secret on its own 20-minute cadence (ensureAzureSecretFresh), independent of compaction. Compaction-only refresh left a gap flagged in review: with compaction.enable: false, flush is the only periodic operation left on a single-volume Azure writer, so it needs its own refresh rather than relying on compaction's cycle. Scoped to the single-volume writer path only, per review discussion: the node's own read-path attach has the identical missing-refresh gap (and S3 has the same gap there too, pre-existing and not introduced by Azure support) — left alone for now as a separate follow-up rather than widened here, and commented as such at the call site. * fix(cli): stop loading Azure extension on S3-only data paths Addresses PR review must-fix #3 (github.com/sipcapture/homer/pull/983): openDuckLakeReadOnly gated both the S3 secret setup and the Azure LOAD+secret setup on the same IsRemoteLakeDataPath check, so any S3-only homer cli invocation also attempted LOAD azure and printed a spurious "failed to load azure extension" warning. The writer path already branched isS3Path vs isAzurePath correctly; the CLI did not. - Export IsS3Path / IsAzurePath (were already package-private in ducklake.go) so cli_cmd.go can branch the two setups independently. - openDuckLakeReadOnly now gates S3 secret creation on IsS3Path and Azure LOAD+secret creation on IsAzurePath, instead of both on the generic IsRemoteLakeDataPath check. - New regression tests exercising openDuckLakeReadOnly end-to-end for both an S3-only and an Azure-only data_path, asserting via duckdb_secrets() that only the matching secret gets created. The Azure-side test intentionally uses a fake CONNECTION_STRING rather than a bare account_name: a credential_chain secret in a sandboxed test environment (no IMDS route) makes the immediately-following parquet-view glob() attempt Managed Identity resolution and hang for ~10 minutes instead of failing fast — found by writing the test the other way first and watching it time out. * fix(storage): honor a custom Azure Blob endpoint for account_key and credential_chain Addresses PR review must-fix #2 (github.com/sipcapture/homer/pull/983): account_key and the native mover's credential_chain branch both hardcoded the public-cloud Blob endpoint, so neither could reach Azurite, Gov/China cloud, or any custom endpoint — only a raw connection_string escaped this. - New azure_endpoint config field, threaded through the writer/node/ tiered/compaction secret builders and mover.AzureConfig. - BuildAzureConnectionString (extracted from BuildAzureSecretSQL) injects BlobEndpoint when set, else the existing public-cloud default. credential_chain now also passes ENDPOINT when configured (verified directly against DuckDB v1.5.5 that both are accepted). - mover/azure.go's account_key branch now builds a connection string and uses NewClientFromConnectionString instead of a hardcoded URL, via a small self-contained copy of the same helper (mover can't import ducklake — ducklake already imports mover). - Tests cover three distinct endpoint URL shapes: Azurite (account name as a path segment on a fixed host:port), Gov/China cloud, and the standard public cloud's own DNS-subdomain shape passed explicitly. Docs updated. * fix(storage): treat CA cert symlink EEXIST as success, document manual fallback Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): EnsureAzureCACertPath warned whenever os.Symlink failed for any reason, including the symlink already existing — treat that specific case as success instead, per review ask. - Check os.IsExist on the symlink error and return silently. - Documented the manual fallback for deployments where Homer can't create /etc/pki/tls/certs itself (bare host package, docker run as non-root). - Fixed comments/docs claiming this only affects blob.core.windows.net — it's any Azure Blob HTTPS endpoint (Gov/China cloud, custom too). * chore(deps): go mod tidy — azblob/azidentity as direct dependencies Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): mover/azure.go imports azidentity and azblob directly, but go.mod listed both as // indirect. go mod tidy also dropped a few now-unused go.sum entries. * test(storage): cover az:// and azure:// in JoinLakeDataPath/joinLake Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): only the S3 scheme was tested for the Unix filepath.Join collapse bug ("s3://" -> "s3:/") that this PR's mkdir/glob fixes guard against. Adds the same coverage for both Azure scheme aliases, in both the ducklake and mover packages. * docs: add Azure examples to ENVIRONMENT_VARIABLES.md Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): the doc only showed S3 field examples for the dots-to-underscores env var naming mechanism. Viper maps HOMER_STORAGE_DUCKLAKE_AZURE_* correctly regardless, but an operator reading the doc had nothing to go on for the Azure field names. Adds the same style of example for azure_account_name/account_key/ connection_string/endpoint. * test(writer,cli): cover Azure config wiring when only account_name is set Addresses PR review should-fix (github.com/sipcapture/homer/pull/983): no test proved Azure config is copied through when only account_name is set (Managed Identity — no key, no connection string). A wrong gate could silently drop it, leaving the writer with no Azure secret. Extracted applyAzureDuckLakeConfig out of writer.go's New() so the gate is testable without real DuckLake I/O. cli_cmd.go's duckLakeConfigFromModular was already pure — just needed the test case. * fix(storage): verify size and set upload parity options in azureCopier.Copy Addresses PR review nit (github.com/sipcapture/homer/pull/983): azureCopier.Copy ignored the size parameter entirely, unlike LocalCopier which verifies the copied byte count matches the catalog's recorded size. Also had no BlockSize/Concurrency set, unlike the S3 uploader. Unlike s3Copier, which declares size via PutObjectInput.ContentLength, UploadFileOptions has no equivalent field. - Check the source file's on-disk size against the catalog's recorded size before uploading instead, catching a truncated/resized source up front. - Set BlockSize/Concurrency to the same values as s3Copier's manager.Uploader (8 MiB, concurrency 3). * fix(storage): fail fast when az:// data_path has no Azure config Addresses PR review nit (github.com/sipcapture/homer/pull/983): EnsureWriterAzureSecret silently no-opped when all Azure fields were empty, so a misconfigured az:// data_path failed later with an opaque ATTACH error instead of a clear message. Every caller already gates on Azure being meant to be configured, so this fixes all 4 call sites at once. * fix(config): copy AzureEndpoint in EnsureNodeDuckLakeVolumes The legacy flat-config synthesis path copied account name, key, and connection string into the default volume but dropped Endpoint, unlike the S3 branch one block above it. Azurite and Gov/China cloud setups relying on the legacy data_path/azure config (not storage_policy.volumes) silently lost their endpoint override. * fix(storage): lock catalogMu around Azure secret refresh flushLoop calls ensureAzureSecretFresh before flushAll, while flushQueueWorker can be mid-INSERT on the same DuckDB connection. DROP+CREATE SECRET now runs under catalogMu, matching how compaction already refreshes credential_chain secrets under the same lock. * fix(node): periodically refresh Azure credential_chain secrets A standalone node attaches volumes once and only revisits them via refreshCatalog, which only runs on a timer when FlightSQL is enabled. An Azure credential_chain secret (Managed Identity via IMDS) is only valid for its token's lifetime, so without FlightSQL it would go stale and queries would start failing after the node had been running for a while. Adds an independent 20-minute ticker, gated to standalone nodes with at least one credential_chain Azure volume, that recreates just the Azure secrets without a full catalog reconnect. | 5 天前 |
| Update license details Updated copyright holder in the LICENSE file. | 3 个月前 |
| chore: bump DuckDB to 1.5.5 and version to 11.0.303 Upgrade duckdb-go/v2 to v2.10505.0 and sync bundled extension downloads and CI packaging to v1.5.5. | 1 个月前 |
| feat(cli): add homer config show (#945) (#958) * feat(cli): add show-running-config for effective settings Operators had no way to see resolved compaction.engine after defaults (#945). Dump file+env+defaults as JSON (secrets redacted) and log the engine on each compaction cycle. * feat(cli): rename dump command to config show Use the same noun-plus-action shape as catalog. homer config defaults to show; this is effective file+env+defaults, not a live process dump. * feat(cli): require explicit config show action Do not treat bare homer config as show. The only supported form is homer config show. | 13 天前 |
| docs: publish Homer 11 guides on GitHub Pages via MkDocs Add MkDocs Material site, CI workflow on homer11, and relocate Swagger UI under docs/swagger/ so it does not conflict with the docs home page. | 3 个月前 |
| chore(pkg): install fetch-doom-wad.sh into scripts/ instead of bin/ | 2 个月前 |
| chore(pkg): install fetch-doom-wad.sh into scripts/ instead of bin/ | 2 个月前 |
| feat(cli): add catalog backup and restore (11.0.322) (#951) Expose the existing VACUUM INTO catalog snapshot as `homer catalog backup`, `restore`, and `list` so operators can rewind DuckLake metadata without rebuild-catalog. Restore takes the writer lock and keeps the previous catalog as a `.pre-restore-*` copy. | 15 天前 |
