A unified fine-tuning framework for Vision-Language-Action (VLA) models on openEuler. It unifies mainstream VLA models (PI0, OpenVLA, ACT, etc.) and multiple data formats (LeRobot v2/v3, HDF5, ROSbag, etc.), supporting LoRA/Freeze/Full/Selective fine-tuning strategies with native Ascend NPU training path, producing weights deployable downstream via IB_Robot.
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: refactor transform pipeline, align architecture docs, add data module design doc - Refactor TransformStep base class and TransformRegistry to support from_config with TransformContext injection; move pipeline helpers into registry and separate images/resize transforms into own modules - Update loader.py to resolve transform inputs from model profile, build TransformContext from recipe + schema + norm_stats - Refactor ACT adapter to use lerobot ACTConfig directly with dynamically built input_features from DataSchema cameras - Remove "Training Artifacts Are Deployable" section from architecture docs; renumber §2.5 to "Dependencies Are Installed On Demand"; move SVG diagrams to docs/graph/, move architecture-text.md to docs/graph/ - Add docs/modules/data-module.md and data-module.cn.md covering data module design: layer responsibilities, Canonical IR, format reader protocol, LeRobot V3 reader, transform pipeline, sampling, batch contract, Observation container, deployment metadata reuse, extension guide, design constraints | 14 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
• feat: add VLA Factory core framework Add recipe-driven training, LeRobot data loading, ACT model integration, inference utilities, deployment adapters, examples, docs, and regression tests. Highlights: - Add CLI entry point and package metadata - Add recipe parsing, model defaults, and example configs - Add LeRobot v3 data pipeline with video decoding and transforms - Add ACT registry adapter and full-parameter training flow - Add inference engine, evaluation flow, and ZMQ deployment adapters - Add architecture docs, README, test fixtures, and regression coverage | 20 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 18 小时前 |
VLA Factory
English: README.md
VLA Factory 是一个 recipe 驱动 的机器人视觉-语言-动作（VLA）模型微调框架：用一份 YAML 描述模型、数据、微调策略与训练参数，框架自动完成「数据管线 → 模型构建 → 训练 → 部署」的完整闭环。
架构和主要特性
架构概述：VLA Factory 的核心目标是把 VLA 微调链路中的数据、模型、训练产物和部署入口统一到一套稳定契约下。用户用 recipe 描述实验意图，框架负责把外部数据格式转换成统一样本语义，通过模型适配层调用上游模型生态，并产出可复用、可验证的训练结果。
- 统一实验入口：用一份 recipe 描述模型、数据、动作空间、微调策略、训练参数和输出位置，减少散落脚本和临时配置约定。
- 统一数据语义：把不同来源的数据集转换成一致的 observation/action 表达，保留 schema、统计量和 key 顺序，便于训练、评估和后续复用。
- 模型生态适配：通过轻量 adapter 接入外部模型实现，框架关注模型协议和训练契约，不重新维护上游模型架构代码。
- 可复用训练产物：训练输出包含 recipe、数据 schema、归一化统计量和模型权重等必要信息，方便复现、评估和后续推理验证。
安装
# 在仓库根目录
pip install -e ".[act]" # ACT 训练（带 lerobot>=0.5）
# PI0 / PI0.5 训练（openpi 栈；openpi 有严格依赖 pin，需走 uv 安装脚本）
bash scripts/install.sh
# 或，在不需要 openpi patch 安装路径时安装全部 extras
pip install -e ".[all]" # 全部模型生态依赖
pip install -e ".[dev]" # 开发依赖：pytest / pytest-cov / tensorboard
在 AutoDL 等弱网络环境中，仍然使用同一个安装脚本，按需增加环境变量：
export VLA_PYPI_INDEX=https://mirrors.aliyun.com/pypi/simple
export VLA_UV_ATTEMPTS=8
export VLA_LOCAL_LEROBOT=1 # 避免 GitHub git 长连接 fetch，改用 tarball 源
bash scripts/install.sh
安装后会注册 vlafactory-cli 命令（如 vlafactory-cli train --config recipe.yaml）；未安装或从源码运行时，等价的 vlafactory-cli ... 同样可用。
快速开始
0. 准备数据
VLA Factory 使用 LeRobot v3 格式的数据集。将数据集放置在 recipe YAML 中 data.source.path 所指定的路径。期望的目录结构为：
<dataset_path>/meta/info.json # 数据集元数据
<dataset_path>/data/ # Parquet episode 文件
<dataset_path>/videos/ # MP4 视频文件
在 recipe YAML 中修改 data.source.path 即可指向自己的数据。
1. 列出已注册模型
vlafactory-cli list
# act backend=pytorch head=...
1. 预处理视频（可选，但推荐）
把数据集的视频帧解码到 .npy 磁盘缓存，避免训练时反复解码：
vlafactory-cli preprocess --config examples/act_lekiwi_banana.yaml
2. 训练
vlafactory-cli train --config examples/act_lekiwi_banana.yaml
常用覆盖参数（不改 YAML 也能临时调整）：
vlafactory-cli train --config <recipe.yaml> \
--steps 5 --batch-size 2 --output-dir outputs/smoke_test
3. 推理 / 评估
# 在数据集某条样本上做一次 smoke-test 推理
vlafactory-cli infer --checkpoint outputs/act_so101_banana \
--dataset-index 0 --split val
# 在整个数据集上评估每条 episode 的 L1 误差
vlafactory-cli evaluate --checkpoint outputs/act_so101_banana \
--dataset /path/to/dataset
4. 启动推理服务
# 仿真器平台
vlafactory-cli serve --checkpoint outputs/act_so101_banana \
--platform simulator --strategy receding_horizon
# lerobot 真机平台
vlafactory-cli serve --checkpoint outputs/act_so101_banana \
--platform lerobot --remote-ip <robot-ip> --strategy receding_horizon
Recipe 配置
最完整的带注释模板是 examples/reference.yaml，每个字段都标注了含义、可选值与典型用法。开箱即用的示例：
| 示例 | 说明 |
|---|---|
examples/act_lekiwi.yaml |
lekiwi 从零训练 |
examples/pi0.yaml |
openpi pi0 系列 smoke —— 默认 pi0 LoRA;注释里给出切 pi05 与全量微调的开关 |
examples/reference.yaml |
全字段注释模板 |
模型默认配置：每个模型在 vla_factory/config/model/<name>.yaml 下附带一份 baseline profile（如 vla_factory/config/model/act.yaml）。工厂以它为默认，再与 recipe 里逐 run 的 model.config 做深度合并——recipe 的值优先，profile 只是实验起点而非冻结契约。未知 key 会被上游配置对象报错（不会因拼写错误静默失效）。
支持路标
| 数据 | 模型 | 算法 | 部署 |
|---|---|---|---|
| ✅ LeRobot v2 / v3 | ✅ ACT | ✅ Full-parameter SFT | ✅ LeRobot |
| ⬜ RLDS | ✅ π₀ / π₀.₅ | ✅ LoRA SFT | |
| ⬜ ROS bags | ⬜ π-FAST | ⬜ Selective SFT | |
| ⬜ HDF5 | ⬜ GR00T / OpenVLA |
项目介绍
A unified fine-tuning framework for Vision-Language-Action (VLA) models on openEuler. It unifies mainstream VLA models (PI0, OpenVLA, ACT, etc.) and multiple data formats (LeRobot v2/v3, HDF5, ROSbag, etc.), supporting LoRA/Freeze/Full/Selective fine-tuning strategies with native Ascend NPU training path, producing weights deployable downstream via IB_Robot.
定制我的领域