nango/packages/persist · na/nango

文件	最后提交记录	最后更新时间
lib	chore(all): Bump zod versions (#7700) <!-- Describe the problem and your solution --> Bump zod to 4.3.6 across repo so 1) it is consistent, especially between `cli`, `runner`, and `lambda-runner` and 2) to remove a memory leak from early zod 4.0.x versions due to how `globalRegistry` is stored <!-- Issue ticket number and link (if applicable) --> [NAN-7294: chore(all): Bump zod to 4.3.6](https://linear.app/nango/issue/NAN-7294/choreall-bump-zod-to-436) <!-- Testing instructions (skip if just adding/editing providers) --> <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/7700?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->	7 天前
.gitignore	Add persist API (#1575) The goal is to remove the database dependency for the runner. Writing customers data (batchSave/.../activityLogs/lastSyncDate) will go through this API	2 年前
nodemon.json	feat: add metering app (#4424) Actions and MAR metrics are published Events are only logged for now <!-- Summary by @propel-code-bot --> --- Add Metering Microservice and Event-Driven Usage Metrics to Nango Monorepo This major PR introduces a new metering microservice (`packages/metering`) and refactors Nango's metric emission by decoupling usage metric tracking (actions, monthly active records - MAR) from the server and persist services. Usage events are now published asynchronously through a new pluggable pubsub pipeline (`packages/pubsub`), which supports ActiveMQ (with failover/multi-URL support) or falls back to a no-op transport if not configured. The metering service subscribes to 'usage' events and logs the data, laying the groundwork for future analytics, billing, and notification extensions. Supporting changes span event schema refactors, lifecycle management across services, build scripts, CI/CD deployment, Docker integration, and wide infrastructure updates. Key Changes: • Added packages/metering microservice that subscribes to usage events (actions and ``MAR``) and logs results (no business logic yet) • Introduced @nangohq/pubsub package with default, `ActiveMQ`, and no-op transport abstractions; pubsub lifecycle (connect/disconnect) managed in all services • Server and persist services now publish standardized usage events instead of inlined metric updates • Replaced legacy billing event shape with a strongly typed and extensible usage event schema • Expanded Dockerfile, tsconfig, and package.json to include metering service in all build/test/deploy flows • Enhanced ``CI``/``CD`` (deploy.yaml) with metering tasks • `ActiveMQ` transport now supports comma-separated `URLs` for multi-broker/failover • Tests updated/added for pubsub transports, event publishing, and relevant flows across packages Affected Areas: • packages/metering (new service: app, processor, utils, env, tracer, config) • packages/pubsub (event/transport abstraction, `ActiveMQ`/no-op/default, schema changes) • packages/server (controllers, pubsub integration, build/runtime hooks, lifecycle) • packages/persist (records, pubsub integration, lifecycle) • monorepo infra: Dockerfile, tsconfig.build.json, package.json, .github/workflows/deploy.yaml • Testing: pubsub transports, event payloads, usage flows Potential Impact: Functionality: No immediate user-visible behavior change; usage events are published and metering logs events for future use. Prepares the platform for downstream analytics, billing, or notification features. Performance: Publishing/logging is asynchronous; low impact unless broker is under high load or down. In broker-down cases, fallback transport ensures business logic is not blocked (but usage metrics are dropped). Security: No new external interfaces; only internal event pipeline. Strong typing enforces internal event payload discipline. Future consumers need robust validation. Scalability: Improves scalability significantly by decoupling metric production/consumption and introducing extensible, event-driven design. Horizontal scaling of analytics, billing, and other consumers now possible. Review Focus: • Correct handling of pubsub connection/disconnection in all affected microservices to avoid leaks or race conditions • Strict schema/type compatibility between event publishers (server/persist) and consumers (metering); watch for drift • Auditing event publishing error/failure paths (currently failures are mostly silent/dropped), especially for high-throughput or transient broker issues • Assess setup for future enhancements (real billing/analytics, retries, dead-letter queues, batch handling) • Docker and ``CI``/``CD`` changes - validate new metering service lifecycle and isolation <details> <summary><strong>Testing Needed</strong></summary> • Test metering, server, and persist service startup/shutdown with/without `ActiveMQ` available (ensure fallback and no hangs) • Manually trigger actions and record ``MAR`` workflows; verify events are published and arrive in metering service logs • Simulate broker outages; verify services gracefully degrade with events dropped (no core logic failure) • End-to-end regression for ``API`` and core Nango flows to check that addition of pubsub is non-breaking • Ensure Docker build, tsconfig, and ``CI``/``CD`` deploy new metering service correctly </details> <details> <summary><strong>Code Quality Assessment</strong></summary> packages/metering/lib/app.ts: Robust startup/shutdown with signal/Sentry/error handling; suggestion to move 'close' function before registering signal handlers to avoid referencing before declaration. packages/pubsub/lib/transport/default.ts: Clean abstraction between ActiveMQ and no-op transports; propagates connection/disconnection errors, but publishing failures are dropped silently. packages/server/lib/controllers/sync.controller.ts: Asynchronous usage event emission integrated, but no error logging on publish failure; recommendation to add error logs. packages/persist/lib/records.ts: Usage event + MAR emission added, but publish failures are silent; logging or retries should be considered for future reliability. </details> <details> <summary><strong>Best Practices</strong></summary> Robustness: • No-op fallback transport ensures safe degradation mode if no broker available • Connection/disconnection lifecycle with error reporting but no retry/backoff yet Maintainability: • Type-safe, shared event schema reduces future breakage • Producer/consumer logic decoupled for easier refactor/extensibility Architecture: • Event-driven separation of usage tracking enables horizontal scaling and future extension • Pluggable transports ease infra migration or local dev </details> <details> <summary><strong>Possible Issues</strong></summary> • Usage events are dropped silently if pubsub publishing fails, with no current retries or dead-letter queue. This is mitigated by author awareness and fallback, but risk increases as reliance on usage data grows. • Schema drift between event producers and consumers is possible as code evolves, which may break analytics/billing in future. • High event rates could overwhelm logging or pubsub broker due to lack of batching; manage log verbosity and broker configuration. • Race conditions in connect/disconnect sequences; confirm startup/shutdown hooks can't cause resource leaks or missed events. • Future external consumers (for analytics/billing) must validate payloads to prevent misuse or data leaks. </details> --- This summary was automatically generated by @propel-code-bot	1 年前
package.json	fix: vulns (#7725) ## Summary - Bump direct dependencies with published advisories: `undici` 6.29.0, `nodemailer` 10.0.13, `vitest` 4.1.11, `@vitest/browser-playwright` 4.1.11, `qs` 6.16.0, `multer` 2.4.0, `@elastic/elasticsearch` 8.19.2, and `testcontainers` 12.2.0. - Pin transitive `joi` to 17.13.8 and `valibot` to 1.5.0 with root overrides. `simple-oauth2` and Storybook keep their own declared ranges. - Refresh the lockfile for the other semver-compatible audit fixes, including the high `brace-expansion` finding. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/7725?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->	6 天前
tsconfig.json	feat: integrate feature flags across services (#6677) Added the `@nangohq/feature-flags` dependency and integrated its initialization and destruction in the `app.ts` files of the `metering`, `orchestrator`, and `persist` packages. Updated relevant `package.json` and `tsconfig.json` files to include the new dependency and its path references. <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> <!-- Testing instructions (skip if just adding/editing providers) --> <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/6677?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->	2 个月前