| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Additional Bloom cleanup (#1642) | 27 天前 | |
Bridge unit test compatibility coverage (#1031) * added test coverage for ensuring compatibility * ran format * fixed unit tests * resolved type issue * added init files * added init file * removed broken test * reverted type change * removed attention mask test * ran format * fixed test * removed failing test * ran format | 1 年前 | |
created new base config class (#1042) * created new base config class * cleaned up imports * reorganized config * setup transformer bridge config * fixed docstring issue * fixed typing issues * ran format * fixed docstring * fixed import issues * ran format * fixed type checking again * fixed import again * fixed doc string * removed import * seperated devices * cleaned up utils a bit * ran format * fixed import * fixed imports * ran format * fixed typing * updated typing * updated name * changed to python 3.12 * cleaned up comments * removed extra functions | 11 个月前 | |
Weight processing generalization (#1131) * finished matchign 2 & 3 * cleaned up components * cleaned up weight processing * fixed embed key mapping * cleaned up key processing * removed extra functions * cleaned up a bit more * cleaned up bridge a bit * generalized weight processing a bit more * allowed attention to be passed through * restored gemma 3 * fixed gpt 2 again * ran format * cleaned up a couple thigns * refactored weight processing to not use split weights * fixed granular tests * added more granular tests * removed invalid import * cleaned up a couple things * removed extra function call * moved weight integration * cleaned up a bit more * cleaned up some more * added verbose mode * ran format * fixed doc string and mypy issues * fixed tets * fixed more tests * fixed issue * added function calls again * updated format * restored last ln step * moved around stop at layer * rested some tests | 9 个月前 | |
Test coverage (#1074) * fixed some ci issues * fixed type issues * ran format * fixed test * fixed type issues * fixed type issue * fixed type issue * fixed test * fixed test * fixed issues * ran format * fixed typing * fixed tests * fixed tests * simplified test * sped up tests * added check for kv cache * ran format * skipped some tests * marked a couple tests to skip * ran some more optimizations * ran poetry lock * regenerated lock * fixed commands * set random seed * updated parallelism prop * updated command * reverted some changes * updated notebook settings * updated verbosity * removed extra test * cleaned up tests some more * marked test as skipped * fixed more tests * sped up CI * reverted CI changes * reverted actions changes * improved cache * sped up some tests * optimzed more tests * sped up some more tests * made more speed improvements * fixed error * fixed typing | 10 个月前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Improve attention masking (#699) * Allow attention_mask to override the default mask in HookedTransformer.forward(). * Add attention_mask argument to loss_fn() and lm_cross_entropy_loss() and adjust the cross entropy calculation to ignore masked (padding) tokens. --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com> | 2 年前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Test coverage/cleaning up xfails (#1334) * Cleaning up NSP & weight processing tests * Clean up old unused skipped tests * Ignoring output in demo notebook import cells | 3 个月前 | |
Add HuggingFace ModelOutput support to TransformerLens generation API (#1144) * Address device/dtype mismatches that caused failures in various contexts. We also update .gitignore to exclude .env (commonly used local file exclution), e.g. to allow collaborators to add their on HF_TOKEN for test suite Core Fixes: ----------- transformer_lens/components/abstract_attention.py: - Replace pattern.to(self.cfg.dtype) with pattern.to(v.dtype) to handle cases where tensors are upcast to float32 for numerical stability while cfg.dtype remains float16/bfloat16 - Add explicit device/dtype synchronization for output projection: * Move weights (W_O) and bias (b_O) to match input device (z.device) * Ensure z matches weight dtype before final linear operation transformer_lens/model_bridge/bridge.py: - Replace direct original_model.to() call with move_to_and_update_config() utility to ensure: * All bridge components (not just original_model) are moved to target device * cfg.device and cfg.dtype stay synchronized with actual model state * Multi-GPU cache tensors remain on correct devices Test Fixes: ----------- tests/acceptance/test_hooked_encoder.py: - Fix test_cuda() to use correct fixture name 'tokens' instead of 'mlm_tokens' tests/acceptance/test_multi_gpu.py: - Update test_cache_device() to pass torch.device("cpu") instead of string "cpu" for proper device type validation tests/unit/components/test_attention.py: - Add test_attention_forward_half_precisions() to validate attention works correctly with bfloat16/float16 dtypes on CUDA devices tests/unit/factored_matrix/test_multiply_by_scalar.py: - Add test IDs to parametrize decorators to avoid pytest cache issues when random numbers appear in test names Tests Fixed by This Commit: --------------------------- - tests/acceptance/test_multi_gpu.py::test_cache_device - tests/acceptance/model_bridge/compatibility/test_legacy_hooked_transformer_coverage.py::TestLegacyHookedTransformerCoverage::test_memory_efficiency[gpt2] - tests/acceptance/model_bridge/compatibility/test_legacy_hooked_transformer_coverage.py::TestLegacyHookedTransformerCoverage::test_consistent_outputs[gpt2] - tests/acceptance/test_hooked_transformer.py::test_half_precision[dtype0] - tests/acceptance/test_hooked_transformer.py::test_half_precision[dtype1] - tests/unit/components/test_attention.py::test_attention_forward_half_precisions[dtype0] - tests/unit/components/test_attention.py::test_attention_forward_half_precisions[dtype1] - tests/unit/model_bridge/compatibility/test_utils.py::TestUtilsWithTransformerBridge::test_device_compatibility[gpt2] * Add HuggingFace ModelOutput support to TransformerLens generate() API Enhance both HookedTransformer and TransformerBridge generate() methods to support HuggingFace-style generation outputs for improved interoperability: - HookedTransformer: Add output_logits flag to return ModelOutput with sequences and logits - TransformerBridge: Forward HF dict flags (output_scores, output_logits, output_attentions, output_hidden_states) to underlying HF model - Maintain full backward compatibility with existing generate() usage patterns - Add 17 integration tests covering ModelOutput behavior and flag handling - Use Any type for ModelOutput returns due to beartype forward reference limitations (#546) This enables downstream libraries to leverage HF's standard generation output format for advanced mechanistic interpretability workflows. * fix(BlockBridge): Return tuple for HF generation compatibility Fix batch dimension bug where BlockBridge.forward() returned bare tensors instead of tuples, causing HuggingFace generation to incorrectly index into batch dimensions. Changes: - BlockBridge: Always return (first,) tuple for single-element outputs - Add regression test for batch dimension preservation during generation - Rename test_generation_modeloutput.py -> test_generation_compatibility.py The bug manifested when HF's GPT2Model did `hidden_states = outputs[0]` expecting a tuple but got a tensor, causing it to index the batch dimension instead of extracting the first tuple element. Fixes batch generation for TransformerBridge with multiple blocks. * Align TransformerBridge.to() with PyTorch nn.Module semantics Enhance to() method to properly handle both device and dtype arguments in all supported PyTorch formats (positional, keyword, combined). Separately invoke move_to_and_update_config for device/dtype to update cfg while delegating the actual tensor movement to original_model.to() with original args/kwargs. This ensures TransformerBridge respects standard PyTorch behavior for model.to() calls. * Add HuggingFace ModelOutput support to TransformerLens generation API Enhance HF compatibility of HookedTransformer and TransformerBridge Core changes: - HookedTransformer: Add output_logits flag returning ModelOutput with sequences and logits - TransformerBridge.generate(): Add output_logits flag for consistency with HookedTransformer - TransformerBridge.hf_generate(): New method for full HF API passthrough (output_scores, output_logits, output_attentions, output_hidden_states) - Maintain unified API: Both classes share same generate() signature per upstream design - hf_generate() for users needing full HF features, evaluate possibility of making it the default generate option in the future. Architecture: - Respects API consistency vision (unified generate() across both HookedTransformer/TransformerBridge classes) - Adds escape hatch for advanced HF use cases without compromising clean API - Clear separation: .generate() = TL-style, .hf_generate() = full HF Testing: - Comprehensive test suite (20 tests) covering ModelOutput behavior and flag handling - Full backward compatibility maintained with existing generate() usage This enables downstream libraries to leverage HF's standard generation output format for advanced workflows while maintaining TransformerLens's clean, consistent API. * minor formatting and type fix * rerun isort fix * type fixes and docstring test fix * minor sync enhancement * apply minor fixes triggered by copilot review * minor cleanup of generate() methods for HookedTransformer/TransformerBridge --------- Co-authored-by: jlarson <jlarson@equity-creative.com> | 6 个月前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Cleanup (#1129) * removed legacy private functions * cleaned up adapter * expanded benchmarks * got gemma 2 closer * added processing step to architecture adapter * setup rotary to sub component properly * cleaned up block * added recursive test of models * got gemma 2 closer * removed extras * ran format * restored files * removed some dead code * fixed test * fixed typing * fixed test * fixed more tests * removed more dead code * added xet to cache * fixed potential recursion issue * reverted bad change * bumped cache * added verbose output * Add --tb=short to coverage-report-test for better error visibility * added more verbose output * configuired output to display errors right away * skipped problematic tests * removed extra flags * skipped hooked encoder decoder * skipped failing test | 9 个月前 | |
Resolution to Issues #477 and #264 (#1288) * Add testing and documentation for GatedMLP * Added BERT tests and additional demo blocks | 3 个月前 | |
Pulling in v2.18.0 changes to v3.x (#1214) * Merging changes from main * CI prep * Fixing utils * Rename GPT OSS MOE test for 2.x to resolve test name conflict | 5 个月前 | |
Add Jacobian lens (J-lens): published-artifact loading, readout, native fitting, and interventions (#1507) * Add JacobianLens analysis tool: artifact loading, readout, fitting, interventions Implements the Jacobian lens (Gurnee et al., Transformer Circuits 2026) as a tools/analysis component working with both TransformerBridge (raw weights) and HookedTransformer (from_pretrained_no_processing): - load/save in the official anthropics/jacobian-lens artifact format (fp16 storage, fp32 in memory), including artifacts published on the HF Hub - per-layer readout through the model\x27s own final norm, unembed, and logit soft cap, with the logit lens as the J=I baseline in the same code path - native fitting reproducing the reference estimator (one-hot cotangents at all valid target positions, sum over targets / mean over sources / mean over prompts), plus n_prompts-weighted merge() for parallel fitting - interventions from the paper: steering, ablation, pseudoinverse coordinate swap in lens coordinates - loud guards refusing fold_ln/compatibility-mode models (the published lenses are fitted on raw HF activations) Unit tests pin the estimator to a closed-form linear toy model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add JacobianLens integration tests on gpt2 with published Hub lens Covers Hub artifact loading/validation, the final-row identity, the rank-collapse late-layer invariant, logit-lens divergence mid-stack, HookedTransformer/TransformerBridge raw-basis agreement, a directional steering smoke test, processed-model refusal, the exact fit==merge identity, and small-fit convergence toward the published lens (cosine 0.91 at L10 from 2 prompts). Fixes _unembed to respect the [batch, pos, d_model] component contract under runtime type checking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add Jacobian Lens demo notebook gemma-2-2b via raw TransformerBridge + the published Hub lens: J-lens vs logit-lens readout on a two-hop prompt with an unspoken intermediate (boot -> Italy -> euro), rank-trajectory plot for pinned tokens, the France->China pseudoinverse coordinate swap (top-1 flip on the language template at alpha=1; Beijing rank 968 -> 8 on the capital template), and J-lens steering. Outputs are baked; verified with nbval locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Index fit gradients on their own device (device_map robustness) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review findings: guards, bounds checks, steering scale, provenance wording - Detect center_unembed processing via the exactly-mean-centered W_U signature: from_pretrained(fold_ln=False) keeps normalization_type LN while still centering weights, which the fold_ln marker alone missed (lens vectors on such a model are silently wrong; readout ranks survive only because LN absorbs the centered component). - Bounds-check readout layers on the logit-lens path and positions after normalization (out-of-range values previously crashed with a raw KeyError or wrapped silently). - Steering: scale by the median per-position residual norm instead of the mean (attention-sink positions run orders of magnitude hot and made the effective strength prompt-length dependent), and attribute the norm-matched parameterization to the reference implementation experiments, not the paper (whose minimal form is h += alpha*v_t). - Move readout residuals to the unembedding device for sharded models; document merge() metadata handling; reword docstrings/error strings that tracked the reference implementation too closely. - Demo notebook: kernelspec metadata, honest alpha=2 overshoot phrasing (the paper reports alpha=2 recovering failures; gemma-2-2b overshoots), measured-GPU wording; re-executed, nbval clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Jacobian Lens review feedback --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Add kurtosis-profile validation test for JacobianLens (#1539 Tier-1) (#1616) * Add kurtosis-profile validation test for JacobianLens (#1539 Tier-1) Asserts the workspace-band signature as relative structure rather than absolute levels, per the cross-family measurement in #1539: band rise vs the model's own early-third baseline, lens-specificity vs the logit-lens control through the identical code path, a gpt2-small negative control, and final-layer identity-transport agreement between arms. | 1 个月前 | |
test(integration): add oracle parity test for JacobianLens (#1539 Tier-1) (#1543) * test(integration): add oracle parity test for JacobianLens (#1539 Tier-1) Compares TransformerBridge JacobianLens.readout() against the reference anthropics/jacobian-lens oracle (pinned to 581d398) on google/gemma-2-2b-it across 75 layer x prompt cells (5 prompts x 15 sampled layers). Pass criteria per the #1539 spec (matching #1505 spike numbers): - Worst-case top-8 token overlap >= 7/8 in every cell - Spearman rank-correlation >= 0.95 on the top-64 logit union per cell The oracle is installed at test time via pip from the pinned commit so the threshold is reproducible independent of upstream drift. Reuses the bridge's original_model + tokenizer to avoid a second model copy in RAM. * style: apply black formatting (line-length=100) * fix: remove unused Dict, Tuple typing imports (pycln) * style: fix black formatting for py310 target (double blank lines) * test: use pytest.importorskip for oracle dep; add oracle-parity CI workflow Replace subprocess pip-install fixture with pytest.importorskip so the test skips gracefully in standard uv venvs (no pip present) and does not mutate the developer environment with no cleanup. Add .github/workflows/oracle-parity.yml — a dedicated workflow that reads ORACLE_COMMIT from the test file (single source of truth) and installs the oracle out-of-band before running the @pytest.mark.slow suite. Triggers on workflow_dispatch and on pushes that touch the test or workflow file, keeping oracle runs opt-in for PR checks. | 1 个月前 | |
Feature kv cache (#1045) * created seperate cache functions * moved components and decoupled config * refactored references * updated transformer lens cache naming * replaced references * ran format * added full cache compatibility * fixed type issues * removed extra param setting | 11 个月前 | |
Feature kv cache (#1045) * created seperate cache functions * moved components and decoupled config * refactored references * updated transformer lens cache naming * replaced references * ran format * added full cache compatibility * fixed type issues * removed extra param setting | 11 个月前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Pulling in v2.18.0 changes to v3.x (#1214) * Merging changes from main * CI prep * Fixing utils * Rename GPT OSS MOE test for 2.x to resolve test name conflict | 5 个月前 | |
Clean up Tests and increase Full Coverage Test Time limit (#1371) * Clean up tautologies and duplicate test coverage * Fix hanging issue | 2 个月前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Test coverage (#1074) * fixed some ci issues * fixed type issues * ran format * fixed test * fixed type issues * fixed type issue * fixed type issue * fixed test * fixed test * fixed issues * ran format * fixed typing * fixed tests * fixed tests * simplified test * sped up tests * added check for kv cache * ran format * skipped some tests * marked a couple tests to skip * ran some more optimizations * ran poetry lock * regenerated lock * fixed commands * set random seed * updated parallelism prop * updated command * reverted some changes * updated notebook settings * updated verbosity * removed extra test * cleaned up tests some more * marked test as skipped * fixed more tests * sped up CI * reverted CI changes * reverted actions changes * improved cache * sped up some tests * optimzed more tests * sped up some more tests * made more speed improvements * fixed error * fixed typing | 10 个月前 | |
Refactor integration tests (#576) * moved some tests into new suite * rearranged tests * ran formatting * changed test name | 2 年前 | |
Merge branch 'main' into dev | 2 年前 | |
Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b-*-hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batch*seq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3*block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the *_global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <|im_end|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge *args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_* aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None | 'config' | 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via **kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_*/ hook_channel_mix_*; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 27 天前 | ||
| 1 年前 | ||
| 11 个月前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 1 个月前 | ||
| 2 年前 | ||
| 2 年前 | ||
| 2 年前 | ||
| 3 个月前 | ||
| 6 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 9 个月前 | ||
| 3 个月前 | ||
| 5 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 11 个月前 | ||
| 11 个月前 | ||
| 2 年前 | ||
| 1 个月前 | ||
| 5 个月前 | ||
| 2 个月前 | ||
| 2 年前 | ||
| 1 个月前 | ||
| 2 年前 | ||
| 10 个月前 | ||
| 2 年前 | ||
| 2 年前 | ||
| 1 个月前 |