master

GGitHubfeat(cli): support mtls (#7325)

529cfc2a创建于 5 天前历史提交

文件	最后提交记录	最后更新时间
.nango	feat(cli): zero deploy (#4129) ## Changes Fixes https://linear.app/nango/issue/NAN-3242/new-compilation-and-deploy - Deploy zero yaml Since I needed to rewrite a lot, I decided to clean up and hopefully improve confirmation output. Let me know (check the video). It's missing json schema and single file deploy. And of course it's not working on the platform yet. - API: Add information about updated scripts ## 🧪 Tests - Init a new folder `node ../nango/packages/cli/dist/index.js init --zero test` - `cd test` - Set env key and hostport in `.env` - Deploy `node ../../nango/packages/cli/dist/index.js deploy dev` https://github.com/user-attachments/assets/8bc4a0ac-0d19-497f-8407-3110e2fbbfa4 <!-- Summary by @propel-code-bot --> --- **feat(cli): Initial Zero Deploy Implementation and Cross-Stack Deploy Diff Refactor** This PR introduces a major overhaul to the Nango CLI's deployment system by implementing 'zero deploy'-a structure allowing CLI-driven deployments without relying on `nango.yaml` configuration. This required substantial changes and refactoring across the CLI, server, API types, example templates, and dependency management. The PR adds a fully packaged/compiled deploy mechanism for yaml-less integration projects, enhances deploy flow feedback and confirmation output, and overhauls server/API-side diff logic to categorize new, updated, and deleted items (syncs, actions, scripts) for more accurate reconciliation and messaging. It also includes improvements for the onboarding experience, example project scaffolding, and dependency updates/cleanups. **Key Changes:** • New zero deploy (`yaml-less`) workflow for ``CLI``, with full packaging/compilation and deploy orchestration (packages/cli/lib/`zeroYaml`/deploy.ts, index.ts, init.ts) • Enhanced diff/confirmation output for deployments; server/``API`` now supports `updated` in addition to new/deleted syncs/actions/`onEvent` scripts (shared/lib/services/sync/sync.service.ts, deploy `postConfirmation`.ts, types) • ``API`` and types expanded to support granular diff responses, with broader backwards-compatibility (types/lib/deploy/api.ts) • ``CLI`` example integrations refactored for zero deploy, with improved initial scaffolding (.env handling, onboarding instructions) • Dependency and lockfile updates: esbuild major/minor bumps, new columnify for ``CLI`` output, dead deps removed • Assorted bugfixes and ergonomic tweaks in ``CLI`` arg parsing, error handling, and onboarding flow **Affected Areas:** • ``CLI`` main entrypoint and `zeroYaml` workflow • Deployment (packaging, compilation, post-confirmation, deploy) • ``API`` & types for deploy/diff responses • Sync/action/`onEvent` diffing & reconciliation logic (shared/services/sync) • Example/template integrations & onboarding docs • ``CLI``/example env setup • Package dependency and lockfile management **Potential Impact:** **Functionality**: Significantly changes the way CLI deployments function for yaml-less projects, introduces more granular confirmation outputs, and can affect legacy deploy flows (e.g., must ensure backward compatibility where yaml is still used). **Performance**: Negligible for most use cases; slightly increased compute/memory during diff resolution for large configurations. **Security**: No new external attack surfaces or privileged flows introduced. **Scalability**: Improved internal structure lays the groundwork for handling larger sets of integrations and more granular updates as projects scale. **Review Focus:** • Correctness and ``UX`` of packaging/compilation/deploy workflow in `zeroYaml`/deploy.ts • Consistency and accuracy of diff outputs (new/updated/deleted) across ``CLI`` and server • Backward compatibility in both ``CLI`` and server ``API`` paths • Completeness and safety of dependency updates and package-lock sync • Type safety and expansion impact in ``API`` responses • Clarity/ergonomics of onboarding and example template updates <details> <summary><strong>Testing Needed</strong></summary> • End-to-end zero deploy: initialize new repo, set env, deploy using new ``CLI`` workflow • Deploy/diff combinations with mixes of new, updated, and deleted syncs/actions/scripts to verify plan confirmation output • Regression test existing (nango.yaml) deploys for backward compatibility • Server ``API`` endpoints: diff and deploy with different payloads; integration and unit test all paths </details> <details> <summary><strong>Code Quality Assessment</strong></summary> **packages/cli/lib/zeroYaml/deploy.ts**: Structured and readable, though file is large and could be further modularized; robust error handling; strong TypeScript idioms. **packages/shared/lib/services/sync/sync.service.ts**: Logic improved for update handling; some complex state flow could benefit from future extraction. **packages/cli/lib/index.ts**: Clear argument handling and command separation; dual-path support for legacy and new flows. **packages/cli/example/github/***: Matches new flow and onboarding; temporary type alias present and commented. **types/lib/deploy/api.ts**: Expanded types maintain compatibility and add needed granularity; size of response should be monitored. </details> <details> <summary><strong>Best Practices</strong></summary> **Type-Safety**: • Comprehensive `TypeScript` annotations and correct interface extensions for new workflow **Error Handling**: • Consistent try/catch and error surface; clear ``CLI`` messaging **Dependency Management**: • Explicit version bumps; test/dev/main split obeyed; dead deps removed **Testing/Modularization**: • `E2E` flows well tested (per author), but large modules should be broken down further </details> <details> <summary><strong>Possible Issues</strong></summary> • Potential backward compatibility breakages with legacy yaml-based deploys if dual paths diverge. • Zero deploy flow lacks single-file deploy and JSON schema support as flagged by author (roadmap limitation, not a regression). • Monolithic logic in zeroYaml/deploy.ts could slow future refactor and bug resolution. • Dependency bumps (notably esbuild and new columnify) require full downstream smoke test. </details> --- *This summary was automatically generated by @propel-code-bot*	1 年前
github	feat: make track deletes work across execution (#5517) The track deletes features currently assume the entire dataset is being fetched/saved within a single execution. Which means it isn't compatible with checkpoints and fetching/saving across multiple executions. To solve this problem, this PR is introducing a way to explicitly define the start and the end of the track deletes interval, storing the starting point in a special checkpoint that survive across executions, instead of assuming the start is always the beginning of the current execution cc @bastienbeurier to agree on the naming `trackDeletesStart/trackDeletesEnd` ex: ``` exec: async (nango) => { await nango.trackDeletesStart('MyModel'); ... await nango.batchSave([...], 'MyModel'); ... const deleted = await nango.trackDeletesEnd('MyModel'); } ``` <!-- Summary by @propel-code-bot --> --- The runner stores a per-model delete-window checkpoint keyed off the sync checkpoint and, when the window closes, uses it to invoke deletion of outdated records before clearing the checkpoint. <details> <summary><strong>Key Changes</strong></summary> • Added `trackDeletesStart`/`trackDeletesEnd` to `NangoSyncBase` and `NangoSyncRunner`, with per-model checkpoint keys and `deleteOutdatedRecords` using stored `syncJobId` • Refactored `Checkpointing` in `packages/runner/lib/sdk/checkpointing.ts` to manage per-key state and accept `key` arguments for checkpoint operations • Expanded CLI parser/compiler validations to track `trackDeletes` calls per model and enforce ordering around `batchSave` • Updated docs and examples to use `trackDeletesStart`/`trackDeletesEnd` and mark `deleteRecordsFromPreviousExecutions` as deprecated • Reset behavior in `packages/shared/lib/clients/orchestrator.ts` now hard-deletes all checkpoints with a key prefix </details> <details> <summary><strong>Possible Issues</strong></summary> • A run that calls `trackDeletesStart` but exits before `trackDeletesEnd` will leave the delete-window checkpoint in place for future runs • The per-key `stateByKey` map in `Checkpointing` is not pruned, which could grow in long-lived processes with many dynamic keys • Full reset now returns an error if `hardDeleteCheckpoints` fails, which could block user-initiated resets </details> --- *This summary was automatically generated by @propel-code-bot*	6 个月前
.env.example	feat(cli): support mtls (#7325) ## Summary - Let the CLI present a client certificate when talking to a self-hosted Nango API behind mTLS, so customers no longer have to disable mTLS for `deploy` / `dryrun` / `pull`. - Configure it with `NANGO_CLI_TLS_CERT` (path to a PEM). A file that contains both cert and key is enough; `NANGO_CLI_TLS_KEY` / `NANGO_CLI_TLS_CA` cover split files and a private CA. - The same TLS agent is used on every CLI-to-API path (axios, fetch, and the Node SDK). Invalid cert/key pairs fail at load time. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/NangoHQ/nango/pull/7325?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	5 天前
.gitignore	feat(cli): add init --zero (#4112) ## Changes Fixes https://linear.app/nango/issue/NAN-3318/nango-init - Add `nango init --zero` `--zero` to be able to optin. NB: it's missing compilation which is coming in an other PR ## Tests  <!-- Summary by @propel-code-bot --> --- **Add 'nango init --zero' Command for Zero-Yaml Project Bootstrapping** This PR introduces a new CLI flag, `--zero`, to the `nango init` command that enables bootstrapping Nango integrations without a `nango.yaml` file (termed 'zero-yaml' mode). It provides a ready-to-use example integration project structure (with syncs, actions, on-events) under `packages/cli/example`, and initializes a new folder, manages `package.json`, installs dependencies, and outlines future compilation steps (compilation not in scope for this PR). Several supporting updates are also included for CLI options, type definitions, project structure, and dependency management. **Key Changes:** • Introduces `--zero` flag to `nango init` ``CLI`` command for zero-yaml project creation. • Adds full project example templates for `GitHub` integration (sync, action, on-event) under `packages/cli/example/`. • Implements ``initZero`` function for initializing and provisioning a zero-yaml workspace (creating folders, copying example, updating dependencies, running `npm install`). • Updates ``CLI`` logic to handle new options for `--zero` and refactors argument parsing, directory checks, and invocation flow. • Augments the global options and service type interfaces for better flag propagation. • Updates and extends `.gitignore` and `package.json` configuration, and adds `ora` for spinner/progress ``UI``. • Improves robustness of folder existence/config validation (safer handling of missing directories) in verification logic. **Affected Areas:** • ``CLI`` command handling (`packages/cli/lib/index.ts`) • Init/zero-yaml logic (`packages/cli/lib/`zeroYaml`/init.ts`) • Type definitions (`packages/cli/lib/types.ts`) • Verification service (`packages/cli/lib/services/verification.service.ts`) • Example integration templates (`packages/cli/example/`) • Dependency management and configuration (`package-lock.json`, `packages/cli/package.json`, `.gitignore`) **Potential Impact:** **Functionality**: Enables a new workflow for initializing integrations without using a YAML file, accelerating onboarding and improving flexibility. Existing flows are updated to correctly respect the presence of zero-yaml or classic yaml mode. **Performance**: Negligible impact; adds file copy and npm install routines to init process. **Security**: No direct security impact, as changes deal with scaffolding and CLI surface. **Scalability**: No material change, as the new code is invoked only during initialization, but template design makes future extension easier. **Review Focus:** • Robustness of directory/file detection and error handling in init workflow. • Correctness of ``CLI`` argument parsing and propagation, especially interactions of new --zero flag with other options. • Sanity and completeness of the example integration templates (do they represent minimal working/demo project?). • Backward compatibility and no disruption to yaml-based workflows. • Potential race conditions or failure modes in file system operations (`initZero`, verification, etc.). <details> <summary><strong>Testing Needed</strong></summary> • Test `nango init --zero` in a fresh directory and verify correct template files, package.json, and node modules are present. • Test init both with and without the `--zero` flag to ensure both flows are functional and fallbacks work (including error handling when initializing in a non-empty or already-initialized directory). • Check proper population of dependencies and project structure in created integration. • Check non-regression in existing flows (standard ``YAML`` project initialization, validation, deployment, etc.). </details> <details> <summary><strong>Code Quality Assessment</strong></summary> **package.json/package-lock.json**: Dependency updates are targeted and necessary for the new spinner UI enhancements. **packages/cli/lib/zeroYaml/init.ts**: Well-structured, async/await code for project setup, includes error handling and progress spinners; should consider more granular error messages for edge cases. **packages/cli/lib/index.ts**: Sound CLI refactor, but further modularization (separating options logic/handlers) would improve maintainability as CLI grows. **packages/cli/lib/services/verification.service.ts**: Better handling for missing folders, but could add more tests or preconditions to check file validity throughout. **packages/cli/example/**: Example templates are clear and provide typical Nango use cases for syncs, actions, and events. </details> <details> <summary><strong>Best Practices</strong></summary> **Code Organization**: • Separation of logic into `initZero` and ``CLI`` handler • Centralized type definitions and options propagation **Project Structure**: • Example integration maintained under version control • Dependency pinning and template clarity **Error Handling**: • File system and command errors caught and reported to the user **CLI UX**: • Provides opt-in flag for experimental features • Validates directory state before writing • Handles errors and gives user feedback using spinner/``UI`` </details> <details> <summary><strong>Possible Issues</strong></summary> • If compilation logic is not yet implemented (deferred to later PR), users may expect fully working integrations immediately after init. • OS-level file permissions or partial directory setups could cause init to fail; more robust rollback/cleanup could improve reliability. • Future changes to the integration template might require manual updates across copies if not handled as a dynamic or external template source. • If example templates get outdated with Nango core, could lead to onboarding issues for new users. </details> --- *This summary was automatically generated by @propel-code-bot*	1 年前
index.ts	feat(cli): migrate to zero (#4138) ## Changes Fixes https://linear.app/nango/issue/NAN-3245/migration-command - `nango migrate-to-zero-yaml` Add massive script to migrate full codebase to new syntax. On best case it migrates everything without a single error, in most cases there are a few types error at the end (e.g: import leftover, unstrict typings that becomes invalid). Tested on a few customers with great success but it's probably missing a few edge cases. - Change export to import in index.ts I realized there was some TS conflict because they all export the same things, it doesn't really matter since I'm not compiling this file but merely using it has declaration. - Try to use cli-test setup to fix windows test Not sure if it's that but worth the shot, otherwise I'll create another PR for the fix ## Tests > [!CAUTION] > This operation is destructive, git commit or save your folder before testing - `node packages/cli/dist/index.js migrate-to-zero-yaml` - ??? - Profit <!-- Summary by @propel-code-bot --> --- **Add Full Migration Command: Zero-YAML TypeScript-Only Refactor for CLI Integrations** This PR introduces the `nango migrate-to-zero-yaml` CLI command, which automates migration of integration codebases from a legacy nango.yaml+script pattern to a Zero-YAML, TypeScript-only structure with Zod models. The solution includes a large-scale transformation script using jscodeshift to refactor all actions, syncs, on-event handlers, and supporting files, generates new models/type definitions, updates index/project structure, and provides comprehensive handling of imports/exports, TS type safety, and edge cases. Substantial improvements are made to CLI internals as well as the runner SDK, and a full suite of snapshot/unit tests validate migration behavior and type expectations. This PR is highly disruptive and is intended to be a one-time destructive migration that must be performed with backups in place. **Key Changes:** • Introduces `nango migrate-to-zero-yaml` command to convert entire Nango integration folders to `TypeScript`/Zod-only Zero-``YAML`` structure • Implements extensive code transformation via jscodeshift: rewrites syncs, actions, on-event handlers, model usage, and imports/exports • Replaces legacy nango.yaml, autogenerates models.ts (Zod models), and builds a new index.ts with ``ESM``/``CJS`` import compatibility • Upgrades, reorganizes, and validates package.json (adds nango, zod, jscodeshift dependencies, triggers npm install); refines tsconfig and project ignores • Enhances ``CLI`` and runner ``SDK`` type safety: enforces `ZodModel` typings, batch method signatures, type parameterization, and more robust model conversion • Updates test infrastructure: adds full unit and snapshot test coverage for migration/transformations, helper files, and all edge cases handled • Improves symlink/edge-case skipping and error reporting throughout the migration flow **Affected Areas:** • ``CLI`` core (command registry, entrypoint) • Migration logic: packages/cli/lib/migrations/`toZeroYaml`.ts • Integration project structure (models.ts, index.ts, package.json, tsconfig) • Runner-``SDK`` typings and batch methods • All script files (syncs, actions, on-events, helpers) • Unit and snapshot testing suites • Dependencies (adds jscodeshift, updates zod handling) **Potential Impact:** **Functionality**: Completely replaces nango.yaml/config with a TypeScript/Zod-based structure, enforcing stricter typings and migration of all integration logic. This is a destructive operation that overwrites many files, deletes nango.yaml, and can cause breakage if backup is not taken. **Performance**: No runtime performance regression, but migration script may be CPU/IO intensive on large codebases, and npm install step will take several seconds. **Security**: No additional security surface, but the destructive migration requires user diligence to backup to avoid data loss; symlink skipping reduces some risks. **Scalability**: Designed to handle large/complex codebases and numerous integrations, with robust model/topological sorting and batch handling; possible edge cases on highly non-standard or legacy patterns. **Review Focus:** • Evaluate correctness/safety of ``AST``/code transformations: does migration preserve script semantics for syncs, actions, on-events, helpers? • Edge case handling: are non-standard imports, legacy type patterns, or symbolic links handled and skipped as intended? • Destructive workflow: Is the backup/``CAUTION`` warning enforced and is failure mode reasonable? • Test coverage: Are new transformation tests and snapshots sufficiently exhaustive? • Review enforcement of new runner-``SDK`` typings, especially for batch methods and Zod models. • Compatibility for ``CLI`` users: any potential for migration to break common or documented integration project structures? <details> <summary><strong>Testing Needed</strong></summary> • Run `nango migrate-to-zero-yaml` on real integration projects-verify output files for correct import/exports, model conversion, and type correctness. • Check migrated `TypeScript` files for unexpected type errors or unresolved imports (especially batch, sync, and model usage). • Validate generated models.ts and index.ts files; run all unit and snapshot tests in various platforms (Linux, Mac, Windows) for cross-compatibility. • Ensure npm install completes and all dependencies are updated as intended. </details> <details> <summary><strong>Code Quality Assessment</strong></summary> **packages/runner-sdk/lib/sync.ts**: Stricter typing and batch method refinement; clarified signatures and prevented accidental type mismatches. **index.ts**: Adds migration command and corrects index imports to ESM-friendly form. **package.json, tsconfig, dependencies**: Upgraded as needed, adding jscodeshift and cleaning dependencies. **packages/cli/lib/migrations/toZeroYaml.ts**: High-complexity, hand-modified imperative code with extensive AST transformation logic. Well-commented but dense; challenging to fully audit. **packages/runner-sdk/lib/scripts.ts**: Type and interface generalization to enforce ZodModel typing, batch method refactor to require id field, generally precise. </details> <details> <summary><strong>Best Practices</strong></summary> **Migration**: • Automated backup/caution prompts and warnings • Robust ``AST`` transformation using jscodeshift • Topological sort for model dependencies to ensure deterministic generation **Codegen/Testing**: • Exhaustive snapshot/unit testing of all transformations • Batch and edge case test data • Preserves/block comments for context in transformed scripts **Cli**: • Clear new command registration and help text • Graceful error reporting, spinner feedback **Type Safety**: • Strict enforcement of `ZodModel` types, runner ``SDK`` parameters • Prevents legacy improper batch method types </details> <details> <summary><strong>Possible Issues</strong></summary> • Partial migration for codebases using deeply non-standard, legacy, or very loosely typed import/export or model references-manual fixes likely after migration. • Symlink skipping is supported, but integrations depending on symbolic-linked source files may not fully migrate. • Destructive operation: any build or migration failure may leave partially migrated state. User must back up before migration. • package.json/tsconfig/npm install may introduce dependency mismatches in custom setups. • Some TypeScript errors will remain for edge API cases, legacy batch calls, or incomplete model refactoring; these are surfaced for manual fix. </details> --- *This summary was automatically generated by @propel-code-bot*	1 年前
package.json	chore(cli): Bump node version (#6253)	3 个月前
tsconfig.json	fix(cli): tsconfig correct options, dev mode parallel error display (#4211) ## Changes - Correct options for tsconfig (need to find a way to sync all those different representation) - Correctly display errors and no error message in dev mode `nango dev` <!-- Summary by @propel-code-bot --> --- This PR updates the TypeScript configuration used by the CLI and its generated artifacts to unify on 'node16' for both 'module' and 'moduleResolution', replacing various legacy values. Additionally, it significantly improves error tracking and messaging during development (`nango dev`), ensuring that errors from both TypeScript and bundling processes (which run in parallel) are accurately captured, displayed, and cleared as files change. Line numbers and clear error output are now shown, and 'No error' is displayed only when truly appropriate. **Key Changes:** • Updated tsconfig.json files and tsconfig constants/templates to use 'module': 'node16' and 'moduleResolution': 'node16'. • Refined parallel error tracking logic in dev mode (packages/cli/lib/zeroYaml/dev.ts), maintaining shared state across TypeScript and bundler errors. • Improved error output: accurate reporting with line numbers, stale error clearing, and correct display of the 'No error' message. • Adjusted CLI test fixture and package.json to align with ESM and the new tsconfig. • Minor type fixes and improved error formatting utilities. **Affected Areas:** • CLI dev tooling (dev.ts) • TypeScript configuration management (constants.ts, example/tsconfig.json) • Error/message utilities (utils.ts) • CLI unit test setup *This summary was automatically generated by @propel-code-bot*	1 年前