| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
• feat: add VLA Factory core framework Add recipe-driven training, LeRobot data loading, ACT model integration, inference utilities, deployment adapters, examples, docs, and regression tests. Highlights: - Add CLI entry point and package metadata - Add recipe parsing, model defaults, and example configs - Add LeRobot v3 data pipeline with video decoding and transforms - Add ACT registry adapter and full-parameter training flow - Add inference engine, evaluation flow, and ZMQ deployment adapters - Add architecture docs, README, test fixtures, and regression coverage | 22 天前 | |
feat: refactor transform pipeline, align architecture docs, add data module design doc - Refactor TransformStep base class and TransformRegistry to support from_config with TransformContext injection; move pipeline helpers into registry and separate images/resize transforms into own modules - Update loader.py to resolve transform inputs from model profile, build TransformContext from recipe + schema + norm_stats - Refactor ACT adapter to use lerobot ACTConfig directly with dynamically built input_features from DataSchema cameras - Remove "Training Artifacts Are Deployable" section from architecture docs; renumber §2.5 to "Dependencies Are Installed On Demand"; move SVG diagrams to docs/graph/, move architecture-text.md to docs/graph/ - Add docs/modules/data-module.md and data-module.cn.md covering data module design: layer responsibilities, Canonical IR, format reader protocol, LeRobot V3 reader, transform pipeline, sampling, batch contract, Observation container, deployment metadata reuse, extension guide, design constraints | 15 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
feat: refactor transform pipeline, align architecture docs, add data module design doc - Refactor TransformStep base class and TransformRegistry to support from_config with TransformContext injection; move pipeline helpers into registry and separate images/resize transforms into own modules - Update loader.py to resolve transform inputs from model profile, build TransformContext from recipe + schema + norm_stats - Refactor ACT adapter to use lerobot ACTConfig directly with dynamically built input_features from DataSchema cameras - Remove "Training Artifacts Are Deployable" section from architecture docs; renumber §2.5 to "Dependencies Are Installed On Demand"; move SVG diagrams to docs/graph/, move architecture-text.md to docs/graph/ - Add docs/modules/data-module.md and data-module.cn.md covering data module design: layer responsibilities, Canonical IR, format reader protocol, LeRobot V3 reader, transform pipeline, sampling, batch contract, Observation container, deployment metadata reuse, extension guide, design constraints | 15 天前 | |
feat: refactor transform pipeline, align architecture docs, add data module design doc - Refactor TransformStep base class and TransformRegistry to support from_config with TransformContext injection; move pipeline helpers into registry and separate images/resize transforms into own modules - Update loader.py to resolve transform inputs from model profile, build TransformContext from recipe + schema + norm_stats - Refactor ACT adapter to use lerobot ACTConfig directly with dynamically built input_features from DataSchema cameras - Remove "Training Artifacts Are Deployable" section from architecture docs; renumber §2.5 to "Dependencies Are Installed On Demand"; move SVG diagrams to docs/graph/, move architecture-text.md to docs/graph/ - Add docs/modules/data-module.md and data-module.cn.md covering data module design: layer responsibilities, Canonical IR, format reader protocol, LeRobot V3 reader, transform pipeline, sampling, batch contract, Observation container, deployment metadata reuse, extension guide, design constraints | 15 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
feat: refactor transform pipeline, align architecture docs, add data module design doc - Refactor TransformStep base class and TransformRegistry to support from_config with TransformContext injection; move pipeline helpers into registry and separate images/resize transforms into own modules - Update loader.py to resolve transform inputs from model profile, build TransformContext from recipe + schema + norm_stats - Refactor ACT adapter to use lerobot ACTConfig directly with dynamically built input_features from DataSchema cameras - Remove "Training Artifacts Are Deployable" section from architecture docs; renumber §2.5 to "Dependencies Are Installed On Demand"; move SVG diagrams to docs/graph/, move architecture-text.md to docs/graph/ - Add docs/modules/data-module.md and data-module.cn.md covering data module design: layer responsibilities, Canonical IR, format reader protocol, LeRobot V3 reader, transform pipeline, sampling, batch contract, Observation container, deployment metadata reuse, extension guide, design constraints | 15 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
feat: adapt openpi pi0/pi05 — LoRA + full finetune, data pipeline, inference round-trip (Issue #3) Model layer - entries/pi0.py: thin composition adapter around openpi's PI0Pytorch (no inheritance, no architecture copy); translates vla Observation ↔ openpi Observation via recipe-declared camera_mapping (unmapped roles get -1 placeholder + zero mask); loads pytorch safetensors ports (lerobot/pi0_base: model. prefix strip, lm_head→embed_tokens tied-weight fix, dtype cast). - entries/pi05.py: pi05 is Pi0Config(pi05=True) on the same upstream class (discrete state tokens, adaRMSNorm timestep, max_token_len 200); shares the parameterized pi0 loader. Pi0Config kwargs are version-adaptive (pytorch_compile_mode / pi05 passed only when the installed openpi declares them — the pinned commit predates the compile knob). - base_contract.py: read a base checkpoint's config.json to learn its real input contract (camera roles/dims/resolution); powers camera_mapping validation and vlafactory-cli list --config. - registry entries defer upstream imports to factory-call time (lazy + cached), so registry scanning never imports lerobot/openpi. Data layer - task chain: lerobot v3 task_index → task text → Frame.language → task_tokenize transform → tokenized_prompt(+mask). Fallback chain mirrors openpi: sample task > recipe default_task > "" (empty prompt, never skips — PI0Pytorch embeds the prompt unconditionally). - pi05: quantile normalisation (q01/q99 → [-1,1], openpi use_quantile_norm) with exact inverse for action output; FeatureStats carries q01/q99 parsed from lerobot v3 stats and round-tripped through inference_metadata. Discrete state prompt: normalized state digitized into 256 bins and embedded as "Task: ..., State: ...;\nAction: " BEFORE pad_dimensions (openpi order). - resize_images keep-ratio letterbox (resize_with_pad 224); state+actions padded together to the model action dim. Training - strategies/lora.py: peft LoRA via target_components → ModelMetadata components subtree (pi0: adapters on the paligemma VLM subtree, base weights inside frozen by peft; action expert + projections stay fully trainable — matches openpi's get_freeze_filter semantics). lm_head excluded from default target_modules (unused by flow matching, tied to embed_tokens). - train.py: finetune-only validation (pi0/pi05 require model.path), LoRA merge at save (chunked in-place delta to fit 12 GB, then merge_and_unload), peft-prefix strip; save_safetensors=False for tied weights. - scripts/merge_lora_checkpoint.py: merge an unmerged intermediate HF Trainer checkpoint into the clean final/model.pt format for evaluation. - Param stats logs split LoRA adapters vs full-FT params, with human-readable units (utils/format.py human_count). Inference / CLI - evaluate/infer pass each frame's task text (not just default_task) so multi-task datasets are scored under the real episode task. - list --config describes any base checkpoint's camera needs and validates the recipe's camera_mapping against it. Install / packaging - scripts/install.sh: pure-uv env for pi0/pi05 (openpi strict pins break pip), CUDA wheel index auto-detected from GPU compute cap, openpi pinned to a known-good commit, transformers_replace patch applied; weak-network overrides (VLA_PYPI_INDEX / VLA_UV_ATTEMPTS / VLA_LOCAL_LEROBOT). - pyproject: [pi0]/[pi05] extras, uv sources, pytest testpaths=["test"]. Examples / docs / tests - examples: pi0_smoke (full FT), pi0_lora_smoke, pi05_lora_smoke — portable repo-relative dataset paths; reference.yaml documents model.config (camera_mapping, default_task resolution chain). - README/README.cn: support matrix (pi0/pi05/LoRA shipped), install section. - Development rules: imports at module top (factory-call-time deferral reserved for optional heavy ecosystem deps and documented cycles). - tests: 108 passed, 20 skipped — pi0/pi05 adapters, base contract, LoRA strategy, quantile round-trip, discrete state prompt, task fallback chain. Verified end-to-end on an RTX 4080 Laptop (12 GB): pi0 LoRA and pi05 LoRA train→save→infer round-trips (pi05: weights missing=0 unexpected=0, 3-step loss 0.058/0.227/0.053, inference actions in ground-truth range). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
feat: integrate RoboTwin 2.0 — native hdf5 reader, serve platform, docs Data layer - formats/robotwin.py: read RoboTwin native episode hdf5 directly (no conversion step). Concatenates /joint_action as the fixed qpos order [left_arm, left_gripper, right_arm, right_gripper] so it matches take_action(action_type='qpos') at deploy time; state[t]=qpos[t], action[t]=qpos[t+1] per RoboTwin/ACT convention. RoboTwin ships no precomputed stats, so get_norm_stats scans the joint trajectories and emits mean/std/min/max plus q01/q99 (the quantile pair pi05 normalisation needs). - codec/hdf5_jpeg.py: decode the per-frame JPEG byte streams embedded in the hdf5. VideoRef gains a stream selector so one container can hold several camera streams (MP4 datasets keep stream=None, RoboTwin carries the camera name and the codec reads /observation/{stream}/rgb). - dataset.py: skip non-tensor metadata when collating. ACT on RoboTwin keeps a raw task string with no tokenizer transform to consume it; it is not a model input and must not reach torch.stack. Deploy layer - deploy/robotwin_server.py: RoboTwin-compatible raw-TCP model server, so the SAPIEN simulator and the model keep separate environments. - deploy/robotwin_adapter.py: native RoboTwin observation → ObsDict. Cameras are selected by the checkpoint schema's camera_keys rather than a hand-edited list, and a missing camera or state-dim mismatch fails loudly instead of silently degrading. - deploy/robotwin_connector.py: the module RoboTwin imports as its policy. Deliberately import-free so it runs inside the SAPIEN environment with no VLA Factory dependencies installed; loaded via PYTHONPATH, so nothing is copied into the RoboTwin checkout. - deploy/configs/robotwin.yml: the minimal bootstrap config RoboTwin's eval_policy_client.py requires. It lives beside the connector rather than under examples/ — it is not a user-facing sample but a runtime artifact of the deploy layer, and it only names the connector: the model and checkpoint stay server-side. - cli.py: serve --platform robotwin plus --host/--port. RoboTwin executes a full chunk per get_action, so the platform forces the synchronous strategy. Training - train.py: pass save_safetensors only when the installed transformers still accepts it. The flag is gone in transformers >=5, and pi0 (4.x) and ACT (5.x via lerobot) live in different environments — both must construct TrainingArguments cleanly. Recipes, docs, packaging - examples/act_robotwin.yaml: ACT-on-RoboTwin training recipe, with a repo-relative data path (no machine-specific absolute path). - README.md / README.cn.md: a RoboTwin Platform section covering the two-environment setup, starting the model server, and driving RoboTwin's eval client. Documented in the READMEs rather than a per-example file so the platform flow is discoverable next to the other supported platforms. - docs/architecture/robotwin-integration-options.cn.md: the integration-option comparison behind the external-connector decision. - pyproject: [robotwin] extra (h5py only — the simulator is not a framework dependency). - .gitignore: virtualenvs, the vendored third-party skills checkout (a separate repo, not a submodule), and Claude Code per-user state. Tests - test_robotwin_reader.py, test_robotwin_server.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | 1 天前 | |
fix: guard non-string cmd in robotwin_server before getattr (PR #6 review) getattr(model, cmd, None) raises TypeError ("attribute name must be string") when cmd is None (missing/non-string "cmd" in request), bypassing the callable() check and returning a confusing error to the client. Explicitly validate isinstance(cmd, str) first so malformed requests get a clean "No model method named" error. Add a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> | 2 小时前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 22 天前 | ||
| 15 天前 | ||
| 1 天前 | ||
| 15 天前 | ||
| 15 天前 | ||
| 1 天前 | ||
| 15 天前 | ||
| 1 天前 | ||
| 1 天前 | ||
| 1 天前 | ||
| 1 天前 | ||
| 2 小时前 |