sig_lerobot:

分支210Tags9
文件最后提交记录最后更新时间
chore(deps): cap torch ceiling at <2.12, pin Linux wheels to cu128 (#3570) * chore(deps): ceiling + cuda * ci: bump cuda version docker image * ci: add cpu wheel to release workflow * chore(deps): update uv.lock * docs: update installation with cuda note25 天前
chore(deps): cap torch ceiling at <2.12, pin Linux wheels to cu128 (#3570) * chore(deps): ceiling + cuda * ci: bump cuda version docker image * ci: add cpu wheel to release workflow * chore(deps): update uv.lock * docs: update installation with cuda note25 天前
feat(policies): add VLA-JEPA (#3568) * first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale action_num_heads / action_attention_head_dim config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add num_target_vision_tokens and action_max_seq_len config fields required by the action head's future-token embedding and positional embedding tables. - Fix default qwen_model_name to 2B (matches all released checkpoints). - Rename ActionEncoder attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace nn.Sequential decoder/state-encoder with _MLP2 (layer1/layer2 naming). - Fix VLAJEPAActionHead to size ActionEncoder and StateEncoder at inner_dim (DiT input width) rather than action_hidden_size (DiT output width). - Rename DiT.blockstransformer_blocks and attnattn1 to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add DiT-test preset for unit tests. - Rewrite ActionConditionedVideoPredictor with explicit ViT-style blocks (_PredictorBlock with fused qkv) to match checkpoint structure; rename encoder/norm/projpredictor_blocks/predictor_norm/predictor_proj. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the action_is_pad tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use action_model_type="DiT-test" now that action_num_heads / action_attention_head_dim have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete test_native_to_lerobot_wm_only test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>1 天前
refactor: support custom progress parquet overlays (#3640)15 天前
feat(docs): modernize readme (#2660) 5 个月前
feat(sim): VLABench benchmark integration (#3396) feat(sim): add VLABench benchmark integration Add VLABench as a new simulation benchmark in LeRobot, following the existing LIBERO and MetaWorld patterns. This PR wires VLABench end-to-end across environment integration, Docker setup, CI smoke evaluation, and documentation. It also fixes a number of upstream packaging and runtime issues required to make VLABench usable and reproducible in CI. What’s included Benchmark integration Add VLABench as a new simulation benchmark. Expose supported VLABench tasks through the LeRobot env interface. Follow the established LIBERO / MetaWorld factory patterns. Preserve lazy async-env metadata so env.unwrapped.metadata["render_fps"] continues to work. CI smoke evaluation Add a VLABench smoke-eval job using lerobot/smolvla_vlabench. Use the correct rename_map for the 3-camera dataset layout. Expand smoke coverage from 1 to 10 primitive tasks. Extract task descriptions after eval so metrics artifacts include per-task labels. Skip Docker Hub login when secrets are unavailable (e.g. fork PRs). Docker / install fixes Install VLABench from GitHub rather than PyPI. Use uv pip, not pip, in the base image. Fail loudly on install errors instead of masking them. Clone VLABench into the non-root user’s home directory. Use shallow editable installs for VLABench and rrt-algorithms to work around missing __init__.py issues. Pin upstream clones to exact commit SHAs for reproducibility. Add undeclared runtime dependencies required by VLABench (open3d, colorlog, scikit-learn, openai). Unpin open3d so Python 3.12 wheels resolve. Assets Support downloading VLABench assets from a Hugging Face Hub mirror via VLABENCH_ASSETS_REPO. Keep Google Drive download support as fallback. Install huggingface_hub[hf_xet] so Xet-backed assets download correctly. Validate required mesh/XML asset subtrees at build time. Patch VLABench constants to tolerate missing asset directories at import time. Runtime / env correctness Import VLABench robots and tasks explicitly so decorator-based registry population happens. Resize and normalize camera observations so they always match the declared (H, W, 3) uint8 observation space. Reinstall LeRobot editably inside the image so the new env code is actually used. Coerce agent_pos / ee_state to the expected shape. Pad actions when needed to match data.ctrl. Replace zero-padding fallback with proper dm_control IK for 7D end-effector actions. Refetch dm_control physics on each step instead of caching weakrefs. Retry unstable resets with reseeding and handle PhysicsError gracefully at step time. Dataset / policy alignment Align VLABench observations and actions with Hugging Face dataset conventions used by lerobot/vlabench_unified: convert EE position between world frame and robot-base frame at the env boundary, expose / consume Euler XYZ instead of raw quaternion layout, align gripper semantics with dataset convention (1 = open, 0 = closed). This fixes policy/env mismatches that previously caused incorrect IK targets and unstable behavior at evaluation time. Docs Add a full docs/source/vlabench.mdx page aligned with the standard benchmark template. Document task selection forms (single task, comma list, suite shortcut). Document installation, evaluation, training, and result reproduction. Point examples at lerobot/smolvla_vlabench. Add a benchmark banner image. Remove outdated / misleading references to upstream evaluation tracks. Document manual install flow instead of a broken vlabench extra. Packaging cleanup Remove the unresolvable vlabench extra from pyproject.toml. Remove the no-op VLABench processor step. Remove the obsolete env unit test that only covered the dropped gripper remap helper. Apply formatting / logging / style cleanup from review feedback. Why this is needed VLABench is not currently consumable as a normal Python dependency and requires several upstream workarounds: no PyPI release, missing package declarations, undeclared runtime deps, SSH-only submodule references, asset downloads outside normal package install flow, registry population that depends on import side effects, env outputs that do not always match declared observation shapes, task resets that can diverge under some random layouts. This PR makes the benchmark usable in LeRobot despite those constraints, and ensures CI runs are reproducible and informative. If you want a much shorter squash commit message, I’d use this: feat(sim): integrate VLABench benchmark with CI, Docker, and docs Add VLABench as a new LeRobot simulation benchmark, following the existing LIBERO / MetaWorld patterns. This includes: LeRobot env integration and task exposure, CI smoke eval with lerobot/smolvla_vlabench, Docker install and asset-download fixes, runtime fixes for registry loading, assets, camera obs, action handling, dm_control IK, and PhysicsError recovery, alignment of obs/action semantics with HF VLABench datasets, docs and packaging cleanup. The PR also incorporates review feedback, improves reproducibility by pinning upstream commits, and makes VLABench usable in CI despite upstream packaging and asset-management issues.1 个月前
feat(policies): add VLA-JEPA (#3568) * first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale action_num_heads / action_attention_head_dim config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add num_target_vision_tokens and action_max_seq_len config fields required by the action head's future-token embedding and positional embedding tables. - Fix default qwen_model_name to 2B (matches all released checkpoints). - Rename ActionEncoder attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace nn.Sequential decoder/state-encoder with _MLP2 (layer1/layer2 naming). - Fix VLAJEPAActionHead to size ActionEncoder and StateEncoder at inner_dim (DiT input width) rather than action_hidden_size (DiT output width). - Rename DiT.blockstransformer_blocks and attnattn1 to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add DiT-test preset for unit tests. - Rewrite ActionConditionedVideoPredictor with explicit ViT-style blocks (_PredictorBlock with fused qkv) to match checkpoint structure; rename encoder/norm/projpredictor_blocks/predictor_norm/predictor_proj. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the action_is_pad tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use action_model_type="DiT-test" now that action_num_heads / action_attention_head_dim have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete test_native_to_lerobot_wm_only test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>1 天前
feat(policies): add VLA-JEPA (#3568) * first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale action_num_heads / action_attention_head_dim config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add num_target_vision_tokens and action_max_seq_len config fields required by the action head's future-token embedding and positional embedding tables. - Fix default qwen_model_name to 2B (matches all released checkpoints). - Rename ActionEncoder attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace nn.Sequential decoder/state-encoder with _MLP2 (layer1/layer2 naming). - Fix VLAJEPAActionHead to size ActionEncoder and StateEncoder at inner_dim (DiT input width) rather than action_hidden_size (DiT output width). - Rename DiT.blockstransformer_blocks and attnattn1 to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add DiT-test preset for unit tests. - Rewrite ActionConditionedVideoPredictor with explicit ViT-style blocks (_PredictorBlock with fused qkv) to match checkpoint structure; rename encoder/norm/projpredictor_blocks/predictor_norm/predictor_proj. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the action_is_pad tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use action_model_type="DiT-test" now that action_num_heads / action_attention_head_dim have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete test_native_to_lerobot_wm_only test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>1 天前
Organize test folders (#856) Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>1 年前
Hardware API redesign (#777) Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com> Co-authored-by: Steven Palma <imstevenpmwork@ieee.org> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Steven Palma <steven.palma@huggingface.co> Co-authored-by: Adil Zouitine <adilzouitinegm@gmail.com> Co-authored-by: Pepijn <pepijn@huggingface.co>11 个月前
feat(ci): add uv.lock (#3292) * feat(ci): add uv.lock * feat(ci): use uv.lock in CI PR testing * chore(ci): rename nightly to docker publish and test * feat(ci): automated update of uv.lock + remove unbound check + docker images now use uv.lock * fix(ci): add --force-with-lease + set -e for silent erros1 个月前
feat(dependencies): require Python 3.12+ as minimum version (#3023) * feat(dependecies): upgrade to python3.12 * fix(test): processor regex message * fix(test): processor regex message * fix(dependecies): resolve all tags in python 3.12 * fix(dependecies): add more hints to faster resolve * chore(dependecies): remove cli tag huggingface-hub dep * refactor(policy): update eagle for python3.12 * chore(docs): update policy creation for python 3.12 * chore(test): skip failing tests in macos2 个月前
docs(agents): add AGENT_GUIDE.md for user facing agent (#3430) * docs(agents): add AGENT_GUIDE.md with SO-101, data, policy, training, eval guidance Adds an agent-facing companion to AGENTS.md that helps AI agents (Cursor, Claude, ChatGPT, etc.) guide end-users through LeRobot without needing to re-read every doc: - Mandatory "ask the user first" block (goal, hardware, GPU, skill level) - SO-101 end-to-end cheat-sheet: install -> calibrate -> record -> train -> eval - Data-collection tips distilled from the folding project (practice before you record, quality > speed, start constrained then add diversity) - Policy decision table with indicative profiling numbers (update ms, peak GPU mem) and AdamW-vs-SGD caveats - Training duration guidance: 5-10 epoch rule, epoch<->step conversion, scheduler/checkpoint scaling with --steps, SmolVLA unfreeze tip - Real-robot eval via lerobot-record --policy.path and sim eval via lerobot-eval, including the pre-baked docker/Dockerfile.benchmark.* images AGENTS.md gets a short pointer to AGENT_GUIDE.md at the top. CLAUDE.md (symlink to AGENTS.md) inherits the pointer automatically. Made-with: Cursor * docs(agents): recommend 2 cameras (front + wrist) as default Made-with: Cursor * docs(agents): add Feetech wiring check and broaden visualizer note Made-with: Cursor * docs(agents): clarify Feetech LED behavior (steady-on, not flash) Made-with: Cursor * docs(agents): expand Feetech troubleshooting (blinking LED, 5V vs 12V variants) Made-with: Cursor * docs(agents): tighten Feetech LED wording Made-with: Cursor1 个月前
docs: add policy & compute guide (#3534) * docs(policy): contributing a policy guide * docs(training): HW compute guide * chore(docs): add to readme and index * Apply suggestions from code review Co-authored-by: Haoming Song <1847575517@qq.com> Signed-off-by: Steven Palma <imstevenpmwork@ieee.org> * chore(docs): slight improvements * refactor(docs): consolidate add policy docs * chore(style): fix pre-commit --------- Signed-off-by: Steven Palma <imstevenpmwork@ieee.org> Co-authored-by: Haoming Song <1847575517@qq.com>25 天前
chore: add AI policy (#3055) 3 个月前
feat(ci): add agent assitance workflow (#3332) Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>1 个月前
chore(ci): update PR template (#2665) * chore: update code of conduct to transformers one * chore: update PR template5 个月前
chore(docs): update contributing (#3387)1 个月前
Add simxarm license 2 年前
Expose sarm package API and ship reward model card template (#3477) * chore: List lerobot_rewardmodel_modelcard_template.md in MANIFEST.in * chore: export SARMConfig, SARMRewardModel, and make_sarm_pre_post_processors from rewards.sarm.1 个月前
feat(sim): add metaworld env (#2088) * add metaworld * smol update Signed-off-by: Jade Choghari <chogharijade@gmail.com> * update design * Update src/lerobot/envs/metaworld.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jade Choghari <chogharijade@gmail.com> * update * small changes * iterate on review * small fix * small fix * add docs * update doc * add better gif * smol doc fix * updage gymnasium * add note * depreciate gym-xarm * more changes * update doc * comply with mypy * more fixes * update readme * precommit * update pusht * add pusht instead * changes * style * add changes * update * revert * update v2 * chore(envs): move metaworld config to its own file + remove comments + simplify _format_raw_obs (#2200) * update final changes --------- Signed-off-by: Jade Choghari <chogharijade@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>7 个月前
docs: add policy & compute guide (#3534) * docs(policy): contributing a policy guide * docs(training): HW compute guide * chore(docs): add to readme and index * Apply suggestions from code review Co-authored-by: Haoming Song <1847575517@qq.com> Signed-off-by: Steven Palma <imstevenpmwork@ieee.org> * chore(docs): slight improvements * refactor(docs): consolidate add policy docs * chore(style): fix pre-commit --------- Signed-off-by: Steven Palma <imstevenpmwork@ieee.org> Co-authored-by: Haoming Song <1847575517@qq.com>25 天前
chore: add security policy (#2809) * chore: add security policy * pre-commit style4 个月前
feat(ci): release workflow publish to pypi test + lock files (#1643) * chore(ci): add some release stuff * chore(ci): add requirements-macos * chore(ci): added lockfiles for future reference * feat(ci): add draft & prerelease option to release workflow tag9 个月前
feat(policies): add VLA-JEPA (#3568) * first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale action_num_heads / action_attention_head_dim config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add num_target_vision_tokens and action_max_seq_len config fields required by the action head's future-token embedding and positional embedding tables. - Fix default qwen_model_name to 2B (matches all released checkpoints). - Rename ActionEncoder attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace nn.Sequential decoder/state-encoder with _MLP2 (layer1/layer2 naming). - Fix VLAJEPAActionHead to size ActionEncoder and StateEncoder at inner_dim (DiT input width) rather than action_hidden_size (DiT output width). - Rename DiT.blockstransformer_blocks and attnattn1 to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add DiT-test preset for unit tests. - Rewrite ActionConditionedVideoPredictor with explicit ViT-style blocks (_PredictorBlock with fused qkv) to match checkpoint structure; rename encoder/norm/projpredictor_blocks/predictor_norm/predictor_proj. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the action_is_pad tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use action_model_type="DiT-test" now that action_num_heads / action_attention_head_dim have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete test_native_to_lerobot_wm_only test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>1 天前
chore(deps): update requirements file (#3114)2 个月前
chore(deps): update requirements file (#3114)2 个月前
chore(deps): update requirements file (#3114)2 个月前
chore: adds dynamic README handling and setup script (#2724) 5 个月前
feat(policies): add VLA-JEPA (#3568) * first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale action_num_heads / action_attention_head_dim config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add num_target_vision_tokens and action_max_seq_len config fields required by the action head's future-token embedding and positional embedding tables. - Fix default qwen_model_name to 2B (matches all released checkpoints). - Rename ActionEncoder attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace nn.Sequential decoder/state-encoder with _MLP2 (layer1/layer2 naming). - Fix VLAJEPAActionHead to size ActionEncoder and StateEncoder at inner_dim (DiT input width) rather than action_hidden_size (DiT output width). - Rename DiT.blockstransformer_blocks and attnattn1 to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add DiT-test preset for unit tests. - Rewrite ActionConditionedVideoPredictor with explicit ViT-style blocks (_PredictorBlock with fused qkv) to match checkpoint structure; rename encoder/norm/projpredictor_blocks/predictor_norm/predictor_proj. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the action_is_pad tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use action_model_type="DiT-test" now that action_num_heads / action_attention_head_dim have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete test_native_to_lerobot_wm_only test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>1 天前

LeRobot, Hugging Face 机器人技术库

测试 测试 Python 版本 许可证 状态 版本 贡献者公约 Discord

LeRobot 旨在为 PyTorch 中的现实世界机器人技术提供模型、数据集和工具。其目标是降低使用门槛,让每个人都能为共享数据集和预训练模型做出贡献并从中受益。

🤗 一个硬件无关、原生 Python 的接口,可标准化各种平台的控制,从低成本机械臂(SO-100)到类人机器人。

🤗 一种标准化、可扩展的 LeRobotDataset 格式(Parquet + MP4 或图像),托管在 Hugging Face Hub 上,支持海量机器人数据集的高效存储、流式传输和可视化。

🤗 已被证明可迁移到现实世界的最先进策略,随时可供训练和部署。

🤗 对开源生态系统的全面支持,以推动物理人工智能的普及。

快速开始

LeRobot 可直接从 PyPI 安装。

pip install lerobot
lerobot-info

Important

有关详细的安装指南,请参见 安装文档

机器人与控制

Reachy 2 Demo

LeRobot 提供了统一的 Robot 类接口,将控制逻辑与硬件细节解耦。它支持多种机器人和遥操作设备。

from lerobot.robots.myrobot import MyRobot

# Connect to a robot
robot = MyRobot(config=...)
robot.connect()

# Read observation and send action
obs = robot.get_observation()
action = model.select_action(obs)
robot.send_action(action)

支持的硬件: SO100、LeKiwi、Koch、HopeJR、OMX、EarthRover、Reachy2、游戏手柄、键盘、手机、OpenARM、Unitree G1。

虽然这些设备已原生集成到 LeRobot 代码库中,但该库设计为可扩展的。您可以轻松实现 Robot 接口,以便将 LeRobot 的数据收集、训练和可视化工具用于您自己的定制机器人。

有关详细的硬件设置指南,请参阅 硬件文档

LeRobot 数据集

为解决机器人领域的数据碎片化问题,我们采用了 LeRobotDataset 格式。

  • 结构: 用于视觉的同步 MP4 视频(或图像)以及用于状态/动作数据的 Parquet 文件。
  • HF Hub 集成:Hugging Face Hub 上探索数千个机器人数据集。
  • 工具: 无缝删除片段、按索引/比例拆分、添加/移除特征以及合并多个数据集。
from lerobot.datasets.lerobot_dataset import LeRobotDataset

# Load a dataset from the Hub
dataset = LeRobotDataset("lerobot/aloha_mobile_cabinet")

# Access data (automatically handles video decoding)
episode_index=0
print(f"{dataset[episode_index]['action'].shape=}\n")

LeRobotDataset Documentation 中了解更多相关信息。

最先进模型

LeRobot 以纯 PyTorch 实现了最先进的策略,涵盖模仿学习、强化学习和视觉-语言-动作(VLA)模型,更多模型即将推出。它还为您提供了用于监控和检查训练过程的工具。

Gr00t Architecture

训练策略就像运行脚本配置一样简单:

lerobot-train \
  --policy=act \
  --dataset.repo_id=lerobot/aloha_mobile_cabinet
类别 模型
模仿学习 ACTDiffusionVQ-BeTMultitask DiT Policy
强化学习 HIL-SERLTDMPC 及 QC-FQL(即将推出)
VLAs 模型 Pi0FastPi0.5GR00T N1.5SmolVLAXVLA

与硬件部分类似,您可以轻松实现自己的策略,并利用 LeRobot 的数据收集、训练和可视化工具,还能将您的模型分享到 HF Hub。

有关详细的策略设置指南,请参阅 策略文档。有关各策略的 GPU/RAM 要求和预期训练时间,请参阅 计算硬件指南

推理与评估

使用统一的评估脚本在仿真环境或真实硬件上评估您的策略。LeRobot 支持 LIBEROMetaWorld 等标准基准测试,更多基准测试即将推出。

# Evaluate a policy on the LIBERO benchmark
lerobot-eval \
  --policy.path=lerobot/pi0_libero_finetuned \
  --env.type=libero \
  --env.task=libero_object \
  --eval.n_episodes=10

通过查阅 EnvHub 文档,了解如何实现您自己的仿真环境或基准测试,并从 HF Hub 进行分发。

资源

引用

如果您在项目中使用了 LeRobot,请引用 GitHub 仓库,以认可其持续的开发工作和贡献者:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}

如果您引用我们的研究或学术论文,请同时引用我们在ICLR上发表的文章:

ICLR 2026 论文
@inproceedings{cadenelerobot,
  title={LeRobot: An Open-Source Library for End-to-End Robot Learning},
  author={Cadene, Remi and Alibert, Simon and Capuano, Francesco and Aractingi, Michel and Zouitine, Adil and Kooijmans, Pepijn and Choghari, Jade and Russi, Martino and Pascal, Caroline and Palma, Steven and Shukor, Mustafa and Moss, Jess and Soare, Alexander and Aubakirova, Dana and Lhoest, Quentin and Gallou\'edec, Quentin and Wolf, Thomas},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://arxiv.org/abs/2602.22818}
}

贡献

我们欢迎社区中的每一个人参与贡献!开始之前,请阅读我们的 CONTRIBUTING.md 指南。无论您是添加新功能、改进文档,还是修复错误,您的帮助和反馈都非常宝贵。我们对开源机器人技术的未来感到无比兴奋,迫不及待地想与您携手共创未来——感谢您的支持!

SO101 Video

Hugging FaceLeRobot 团队用心打造

下载使用量

0

项目总下载次数(含Clone、Pull、 zip 包及 release 下载),每日凌晨更新

语言类型

Python99.93%
Makefile0.07%
MDX0%