nango/packages/pubsub · na/nango

文件	最后提交记录	最后更新时间
lib	feat(audit): batch the consumer's writes and stop dropping events on failure (#6954) - A failed write no longer discards the event. The consumer reads up to ten messages per poll, writes them in one ClickHouse insert carrying a deduplication token, and deletes them only if that insert succeeded. A failure leaves them for redelivery and, past the queue's `maxReceiveCount`, the DLQ — where they can be redriven once the cause is fixed. Previously every failure was acknowledged, so a ClickHouse problem silently dropped audit events. - Reads collapse duplicate events. At-least-once delivery can store an event twice, because a redelivered batch has a different composition and so a different deduplication token, and `ReplacingMergeTree` only merges the copy away eventually. The read path drops copies from the result set so the API and dashboard never show one. - Insert errors are sanitised before logging. ClickHouse quotes the offending row back in `VIOLATED_CONSTRAINT` and format-parse errors, and the audit blob holds the actor's email and IP. The code, constraint name and failing expression are kept; the quoted payload is not — it stays in the DLQ message, under the same access controls as the audit store. ### Why the read doesn't deduplicate server-side `FINAL` and `LIMIT 1 BY id` both defeat the short-circuit this query depends on: `ORDER BY` matches the primary key prefix, so ClickHouse stops as soon as the page is filled. Rows read for one unbounded page on a 5M-row account: \| \| rows read \| \|---\|---\| \| as written \| 99K \| \| `LIMIT 1 BY id` \| 1.1M \| \| `FINAL` \| 5M — the entire account \| Extrapolated to the 300M-row account in the [performance benchmark](https://linear.app/nango/document/audit-logs-clickhouse-performancecost-benchmark-c54e4fd0a36d), `FINAL` would read 300M rows per page and its interactive latency conclusions would no longer hold. The benchmark stands as measured, with one constraint now attached to it: this query must not deduplicate server-side. Copies share `(account_id, occurred_at, id)`, so they are adjacent in this ordering and a single pass catches them, and the keyset cursor's strict comparison excludes one straddling a page boundary. Because the query already fetches one spare row, a page containing a single copy is still full. ### Failure handling Every failure is treated the same way — nothing is deleted — so there is no error classification in the control flow. Classification is a metric dimension instead: ingest failures are tagged with ClickHouse's own error code, so alerting can separate a rejected event (a bug in what we emit) from an unavailable backend without this code enumerating codes or matching message text. Undecodable messages and messages published under another subject are counted as rejections and also left undeleted, so they reach the DLQ with their payload intact. Failure logs carry, per message, the SQS message id, the audit event id, its resource and action, and the delivery count — enough to identify which event in a batch was the problem and how close it is to the DLQ, without copying personal data into logs. The write timeout is derived from the queue's visibility timeout rather than configured separately, so the two cannot be put out of step by a deploy: an insert outliving a message's invisibility would be redelivered while the first attempt was still running, and both would write the row. ### Behaviour changes The consumer polls SQS directly instead of going through `@nangohq/pubsub`'s Subscriber, so it can batch and control acknowledgement. Consequences worth knowing: - It no longer starts on ActiveMQ deployments. It needs an SQS queue URL, so on `NANGO_PUBSUB_TRANSPORT=activemq` tiers it logs and stays off rather than consuming. Audit is already gated on `CLICKHOUSE_URL`, so no environment loses a working path, but it is a capability removal for non-SQS tiers. - The producer still publishes through `pubsub.publisher`, so the two halves of the pipeline now use different abstractions. Deliberate: batching and ack timing only exist at the consuming end. - `nango.audit.clickhouse.ingest.result` gains a `code` tag on failures and now increments by the batch size rather than by one. `@nangohq/pubsub` itself is behaviourally unchanged — only its wire-format helpers become exported, so the consumer decodes with the same code that produced the message instead of reimplementing a v8-serialised format outside the package that defines it. ### Notes for the reviewer - The consumer's poll-loop, abort and shutdown scaffolding intentionally mirrors `packages/jobs/lib/webhook/dispatch-queue/consumer.ts`. This is the second use of that shape; a third should extract a shared SQS batch consumer. - One line in `sns-sqs.ts` (`topicArns` spread) is an incidental `oxlint --fix` change applied by the pre-commit hook, not a deliberate edit. It cannot be reverted while the file is staged — the hook reapplies it. - Conflicts with #6934 in three files (`clickhouse.ts`, `index.ts`, `metering/lib/app.ts`), all keep-both. Either can merge first. ## Test plan - [x] `npm run ts-build` clean - [x] Unit: audit 24, metering 4, pubsub 35, utils 350 - [x] Integration against real ClickHouse: audit store 6, including a duplicate stored with merges stopped, proving the read collapses it rather than a merge having done so - [x] Read cost measured on a 5M-row account across 1-day, 14-day and unbounded windows - [x] Verified by deliberate regression that the tests fail when the dedup, the delete-on-success rule, the sanitiser and the typed error code are each removed - [x] prettier + oxlint clean - [ ] Re-measure read cost on Cloud — the numbers above are a local single node at 5M rows, not SharedMergeTree at 300M - [ ] DLQ alert and redrive runbook (NAN-6277), which this PR makes load-bearing and does not deliver - [ ] Watch batch size, rejection count and DLQ depth on dev after the cutover <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/6954?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	1 个月前
package.json	fix: upgrade dd-trace (#6905) <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> <!-- Testing instructions (skip if just adding/editing providers) --> <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/6905?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->	1 个月前
tsconfig.json	feat: ingest usage events to clickhouse (#5798) This commit introduce clickhouse and ingest usage events to raw table It doesn't ingest connections or records usage because they are not event-based and I haven't figure out how I am gonna model those metrics yet. CLICKHOUSE_URL isn't set in staging/prod. Until then code is noop Next steps will be creating materialized views for the metrics. <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> <!-- Testing instructions (skip if just adding/editing providers) --> <!-- Summary by @propel-code-bot --> --- This PR also establishes the foundational ingestion pipeline around ClickHouse by integrating migration and ingestion lifecycle handling into the metering flow and introducing buffered event processing so event-based analytics can be captured reliably without disrupting existing billing and usage-tracking paths. It formalizes the initial event-oriented ingestion scope as a staged rollout toward downstream metric modeling, while keeping behavior safely disabled in environments where ClickHouse is not configured. --- This summary was automatically generated by @propel-code-bot	4 个月前