| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Compatibility Mode – pre-ln split qkv hooks (#1319) * Initial resolution of issue with formerly pre-ln hook firing post-ln * Resolved pre-ln issues * setup arena to use new flag | 3 个月前 | |
Fix n_devices max_memory caps (#1501) * Fix n_devices max_memory caps * Ignore import output in activation patching demo | 1 个月前 | |
fix: scope meta device_map targets to load_weights=False per maintainer review (#1493) Per jlarson4's review on PR #1493: meta device_map values crash at boot (NotImplementedError in HF tie_weights, KeyError in Accelerate offload hooks) and are silently ignored when load_weights=False. Changes: - Remove 'meta' from _UNSUPPORTED_OFFLOAD_DEVICE_MAP_VALUES so the resolver passes it through unconditionally - Validate meta at boot() level where load_weights is in scope: reject with ValueError when load_weights=True, accept when False - Disk offload remains blocked in the resolver (Bridge wrappers bypass Accelerate hooks) - Updated tests: resolver-level meta pass-through, boot-level rejection (default) and acceptance (load_weights=False) Signed-off-by: hiro-nikaitou <vieteviete@proton.me> | 1 个月前 | |
Add GLM-4 MoE TransformerBridge adapter (#1428) * tidy: Remove reference to deleted additional comments (#1410) Simple resolution for #1409 * Add GLM-4 MoE TransformerBridge adapter * Ignore BERT import cell output in nbval --------- Co-authored-by: Dan Raviv <dan.raviv@gmail.com> Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 2 个月前 | |
Demos/bridge lm eval demo (#1286) * Updating tokenizer information on TransformerBridge table to be more detailed * Updating docs build watcher to properly hot-reload for model table changes * Update Interactive Model table to properly load information again * Improved docstring to wrap up #99 * Updated `prepend_bos` comments to resolve #100 * Added tokenization tests for the Bridge * Added new demo to show how to run lm-eval-harness with TransformerBridge * stripped stale output * Add optional evals dependencies * Fixing notebook checks for eval * Clean up issues with demo | 3 个月前 | |
Pre-release cleanup (#1320) * Remove plan scratch files * Documenting HookedRootModule file change, and adding deprecation warning for old import path * Qwen 43.5 transformers version cleanup * Adding transformers update note to the migration docs * Additional small adjustments * Format checks and mypy | 3 个月前 | |
Fix llama demos (#619) * revised demo testing to check all demos * separated demos * changed demo test order * rearranged test order * updated attribution patching to run differnt code in github * rearranged tests * updated header * updated grokking demo * updated bert for testing * updated bert demo * ran cells * removed github check * removed cells to skip * ignored output of loading cells * changed notebook tests to run on separate jobs * added all notebooks to CI * renamed file temporarily * fixed file case * reorganized setup * updated head detector demo * reran othello * updated installation section * updated no position install to install deps in github * updated output of beginning areas * updated starting block for llama * regenerated no position experiment * removed llama gpu * skipped llama * ran ineractive neuroscope * updated neuroscope diff areas * removed cell output * updated install steps * fixed import * made sure to only run llama 1 block if the model is available * added llama to tests again * fixed deprecation message * locked transformers version * fixed llama 2 demo * removed llama from ci * removed llama 2 from ci * turned off some demos in ci * removed activation patching * removed grokking demo | 2 年前 | |
Issue resolution for #341, #644, and #210 (#1300) * Resolution for #644 and #341 * Started activation cache improvement * Full resolution for 210 + a demo notebook | 3 个月前 | |
Direct path patch demo (#1398) * chore: Plot helper allows customizing graph before showing it * feat: Direct path patching in exploratory analysis demo, resolves #111 * doc: fix head index in prose | 2 个月前 | |
Fix error opening jupyter notebook (#1437) * tidy: Move jupyter packages to jupyter dependency group * fix: error opening jupyter notebook By updating jupyter notebook (v6.5.4->v6.5.7) Previously, `uv run jupyter notebook demos/xxx.ipynb` failed with: ModuleNotFoundError: No module named 'jupyter_server.contents' This was fixed in jupyer 6.5.6 (see https://github.com/jupyter/notebook/issues/7048) * chore: Add matplotlib dev dependency, used in some notebooks * fix: Invalid import in GPT_OSS_Demo notebook | 2 个月前 | |
Fixing issues with Multimodal text generation | 3 个月前 | |
deprecate remaining hooked entry points (#1592) * deprecate remaining hooked entry points * test: account for hooked transformer warning in notebook * fix: address deprecation warning review feedback * fix: correct encoder deprecation warning stacklevel | 30 天前 | |
Fix type of HookedTransformerConfig.device (#1230) * Fix type of HookedTransformerConfig.device This is typed as `Optional[str]` but sometimes returns `torch.device`. Updated the code to just return the `str` instead of wrapping with a device. I'm not confident that every function which takes a device will always be passed a string, so I didn't change functions like warn_if_mps. Found while working on #1219 * more cleanup * 3.0 CI Bugs (#1261) * Fixing `utils` imports * skip gated notebooks on PR from forks * Updating notebooks * Ensure LLaMA only runs when HF_TOKEN is available --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 4 个月前 | |
Fix type of HookedTransformerConfig.device (#1230) * Fix type of HookedTransformerConfig.device This is typed as `Optional[str]` but sometimes returns `torch.device`. Updated the code to just return the `str` instead of wrapping with a device. I'm not confident that every function which takes a device will always be passed a string, so I didn't change functions like warn_if_mps. Found while working on #1219 * more cleanup * 3.0 CI Bugs (#1261) * Fixing `utils` imports * skip gated notebooks on PR from forks * Updating notebooks * Ensure LLaMA only runs when HF_TOKEN is available --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 4 个月前 | |
Drop optional unused aliases on hybrid architectures (#1579) * Drop optional unused aliases * feat: assign fallbacks on pruned * Improve unittests | 29 天前 | |
feat: Add LIT integration for interactive model analysis (#121) (#1163) * feat: Add LIT integration for interactive model analysis (#121) * fix: resolve type errors and linting issues in LIT integration - Fix HookedTransformer type imports to use class from module path - Remove unused imports (TYPE_CHECKING, Optional, Sequence, etc.) - Fix return type annotations for functions returning Optional values - Add proper type: ignore comments for dynamic base classes - Add noqa comments for intentionally unused TYPE_CHECKING imports - Add explicit type annotation for output dict - Format code with ruff formatter * fix: add Python version restriction to lit-nlp dependency lit-nlp requires Python >=3.9, so add python constraint to ensure compatibility with the project's Python >=3.8 support * fix: replace forward reference string hints with Any to avoid beartype errors Beartype tries to resolve forward reference strings at runtime, which fails for TYPE_CHECKING-only imports. Replace string type hints like 'HookedTransformer' and 'ActivationCache' with Any to allow runtime type checking to pass while maintaining documentation in docstrings. Also clean up unused imports from test_lit.py: - Remove unused TYPE_CHECKING, Any, Dict, List imports - Remove unused sys and lit_nlp imports - Remove unused inputs variable in test * fix: formatting and add NBVAL_IGNORE_OUTPUT to BERT.ipynb import cell - Apply black formatting to test_lit.py and __init__.py - Add NBVAL_IGNORE_OUTPUT comment to BERT.ipynb import cell to ignore warnings during import * fix: add NBVAL_IGNORE_OUTPUT to Othello_GPT.ipynb and skip doctests in LIT module - Add NBVAL_IGNORE_OUTPUT to Othello_GPT.ipynb import cell (Cell 5) to ignore stderr - Add doctest: +SKIP to all doctest examples in LIT module since lit-nlp is optional - This prevents docstring tests from failing when LIT is not installed * fix: add NBVAL_IGNORE_OUTPUT to Othello_GPT.ipynb transformer_lens import cell * fix: add NBVAL_IGNORE_OUTPUT to Main_Demo.ipynb transformer_lens import cell * fix: add NBVAL_IGNORE_OUTPUT to Exploratory_Analysis_Demo and Patchscopes_Generation_Demo notebooks * fix: add NBVAL_IGNORE_OUTPUT to correct cell (Cell 1) in Exploratory_Analysis_Demo.ipynb * fix: revert the notebooks back and change __init__.py to fix the offending bug * fix: regenerate poetry.lock with lit-nlp dependency * fix: regenerate poetry.lock with Poetry 1.8.2 for upstream compatibility * fix: format LIT __init__ and stabilize decoder attention tolerance | 6 个月前 | |
Llama demos tidy (#1427) * chore: Rename Llama demo notebook to match #1233 changes * chore: fix title in Llama demo notebook to match #1233 changes | 2 个月前 | |
Llama demos tidy (#1427) * chore: Rename Llama demo notebook to match #1233 changes * chore: fix title in Llama demo notebook to match #1233 changes | 2 个月前 | |
updated loading in llava demo to use transformer bridge (#1020) * Migrate LLaVA demo to TransformerBridge - Replace HookedTransformer with TransformerBridge.boot_transformers() - Move weight processing kwargs to enable_compatibility_mode() - Remove unsupported hf_model= and vocab_size= kwargs - Clear stale outputs from unrun cells 0/8 cells pass locally. Requires CUDA + PIL + LlavaForConditionalGeneration. Also blocked on TransformerBridge not supporting hf_model= for loading pre-instantiated HuggingFace models. * merged dev-3.x-canary * Updating LLaVA demo * Updated LLaVA demo for KV Cache handling * Type cleanup --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 4 个月前 | |
Delete dup divergent info in Main Demo, resolves #1415 (#1416) | 2 个月前 | |
deprecate remaining hooked entry points (#1592) * deprecate remaining hooked entry points * test: account for hooked transformer warning in notebook * fix: address deprecation warning review feedback * fix: correct encoder deprecation warning stacklevel | 30 天前 | |
deprecate remaining hooked entry points (#1592) * deprecate remaining hooked entry points * test: account for hooked transformer warning in notebook * fix: address deprecation warning review feedback * fix: correct encoder deprecation warning stacklevel | 30 天前 | |
updated loading in patchscopes generation demo to use transformer bridge (#1021) * updated loading in patchscopes generation demo to use transformer bridge * Migrate Patchscopes Generation Demo to TransformerBridge - Replace HookedTransformer with TransformerBridge.boot_transformers() - Fix deprecated ipython.magic() to ipython.run_line_magic() - Clear stale outputs from unrun cells All 20 cells pass locally. * Fixes to ensure functionality with v3.x --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 5 个月前 | |
updating loading in qwen demo to use transformer bridge (#1025) * updating loading in qwen demo to use transformer bridge * add qwen demo to CI * Updating Qwen Notebook for TransformerLens 3.x * Changing model to fit in CI --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 5 个月前 | |
Created a system for running Native models (#1353) * Initial native source setup * Format and comment cleanup | 3 个月前 | |
Fix type of HookedTransformerConfig.device (#1230) * Fix type of HookedTransformerConfig.device This is typed as `Optional[str]` but sometimes returns `torch.device`. Updated the code to just return the `str` instead of wrapping with a device. I'm not confident that every function which takes a device will always be passed a string, so I didn't change functions like warn_if_mps. Found while working on #1219 * more cleanup * 3.0 CI Bugs (#1261) * Fixing `utils` imports * skip gated notebooks on PR from forks * Updating notebooks * Ensure LLaMA only runs when HF_TOKEN is available --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net> | 4 个月前 | |
Test coverage/cleaning up xfails (#1334) * Cleaning up NSP & weight processing tests * Clean up old unused skipped tests * Ignoring output in demo notebook import cells | 3 个月前 | |
Added n_ctx override to TransformerBridge (#1269) * Added n_ctx override to TransformerBridge * Prevent output of progress bars in T5 demo | 4 个月前 | |
Fix Tracr demo output_label: compiled labels are auto-numbered (#1525) The Tracr demo passes output_label="reverse", but Tracr builds labels as f"{name}_{unique_id}", so the compiled expression's residual labels are reverse_1:* and the demo raises ValueError at the state-dict cell. Pass reverse.label instead of hardcoding the name, and correct the same wrong label shape in the unit-test fixture (real Tracr never emits "reverse:1") and in the infer_tracr_output_label docstring example. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
updated loading in activation patching demo to use transformer bridge (#1011) * updated loading in Activation Patching in TL Demo to use transformer bridge * use undeprecated ipython code to avoid deprecation warnings * revert metadata changes * updated installation source * Fix notebook CI: skip widget MIME type comparison and clear stale cell output Add application/vnd.jupyter.widget-view+json to conftest.py skip_compare to avoid false failures from random widget model_id values. Clear outputs from unrun cell (execution_count=null) in Activation Patching demo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Neel Plotly import does not run in CI * Extend processing time for slow cell * Added NBVAL_SKIP for long running process that cant pass CI --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> Co-authored-by: jlarson4 <jonahalarson@comcast.net> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 5 个月前 | |
feat: add get_act_patch_direct_path for head-to-head circuit analysis (#1396) * feat: add get_act_patch_direct_path for head-to-head circuit analysis Closes #111. Implements direct path patching — a finer-grained variant of activation patching that isolates the direct information flow between two specific attention heads, rather than replacing the full residual stream. Why --- Standard activation patching tells you that *some* component at layer L matters, but it cannot distinguish whether head B at layer L+2 matters because it received information directly from head A, or because A's output propagated through many intermediate components first. Direct path patching isolates the A → B causal edge precisely. Implementation -------------- For a fixed source head A = (src_layer, src_head) and every downstream destination head B = (dst_layer, dst_head): delta_resid = clean_A_result - corrupted_A_result # [batch, pos, d_model] delta_B_q = (delta_resid / ln1_scale) @ W_Q[hb] # [batch, pos, d_head] patched_B_q = corrupted_B_q + delta_B_q The per-head residual contribution is computed from hook_z @ W_O (always available in the default cache) rather than hook_result, which requires the non-default cfg.use_hook_result=True flag. New files --------- - transformer_lens/direct_path_patching.py get_act_patch_direct_path() [n_layers, n_heads] sweep get_act_patch_direct_path_all_sources() [n_layers, n_heads, n_layers, n_heads] full sweep - tests/unit/test_direct_path_patching.py 12 tests covering output shape, causal structure, manual correctness verification, and edge cases. All pass on a tiny randomly-initialised 3-layer model (no downloads, runs in ~3s on CPU). - demos/direct_path_patching_ioi.py Validated on GPT-2 small / IOI task. S-inhibition heads (7.3, 7.9, 8.6, 8.10) show strongest direct paths into name-mover heads (9.9, 9.6, 10.0), confirming the Wang et al. 2022 IOI circuit. (8,6) → (9,9): +0.083 normalised logit diff (8,10) → (9,9): +0.066 (7,9) → (9,9): +0.036 API matches existing get_act_patch_* functions in patching.py for drop-in use alongside the existing circuit analysis toolkit. * style: apply black + isort formatting (line-length=100) * fix: remove unused imports, add type: ignore for mypy, clean up demo import * fix: add type: ignore[index] on W_O[h] indexing for mypy * fix: address reviewer feedback — TransformerBridge support, fold_ln guard, independent test, notebook demo * fix: check fold_ln for TransformerBridge via .weight attribute TransformerBridge wraps the original HuggingFace LayerNorm module, which stores the learned scale as .weight rather than the .w used by HookedTransformer. Fall back to .weight so the guard actually fires when a TransformerBridge model is passed without folded LN, rather than silently skipping the check. * refactor: move to tools/analysis/, logit-diff metric, TestCheckFoldLn, fix _check_fold_ln tensor bug - Move direct_path_patching.py to transformer_lens/tools/analysis/ alongside the Direct Logit Attribution tool; add tools/analysis/__init__.py exporting both public functions; update transformer_lens/__init__.py accordingly. - Fix _check_fold_ln: replace 'getattr(...) or getattr(...)' with explicit None checks to avoid RuntimeError on multi-element tensors. - test_correctness_against_actual_ln_forward: switch patching metric to logit diff (correct_tok - incorrect_tok), which cancels the centering offset introduced by process_weights_() and tightens tolerance 0.15 -> 1e-3. - Add TestCheckFoldLn (5 tests): folded model no-warning, unfolded model warns, pre-fold .w attribute present, no crash on missing attribute, no RuntimeError on multi-element tensor regression check. All 17 tests pass. * fix: loosen _check_fold_ln type hint to Any for beartype compatibility _check_fold_ln is a private defensive helper with a try/except that handles arbitrary model types. The Union[HookedTransformer, TransformerBridge] annotation was causing beartype to reject valid test fixtures (and any non-standard model) at the call boundary before the function's own exception handling could run. Any is the correct annotation for a function intentionally designed to tolerate unknown model shapes. * style: black formatting on test_direct_path_patching.py | 2 个月前 | |
deprecate remaining hooked entry points (#1592) * deprecate remaining hooked entry points * test: account for hooked transformer warning in notebook * fix: address deprecation warning review feedback * fix: correct encoder deprecation warning stacklevel | 30 天前 | |
Cleaning up CI issues with LLaMA notebooks (#1233) * Trimmed checks temporarily down to just the notebook tests so we can test them in isolation without constantly blowing our HF rate limit * HF login * Adding quantized handling to quantized notebook * Try testing LLaMA demos with llama 3.2 for faster loading – avoid CI timeout * Final testing for notebook changes * Restore quantized install * Clean up stderr | 4 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 2 年前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 30 天前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 29 天前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 4 个月前 | ||
| 2 个月前 | ||
| 30 天前 | ||
| 30 天前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 3 个月前 | ||
| 4 个月前 | ||
| 3 个月前 | ||
| 4 个月前 | ||
| 1 个月前 | ||
| 5 个月前 | ||
| 2 个月前 | ||
| 30 天前 | ||
| 4 个月前 |