TransformerLens/tests/unit/model_bridge · tra/TransformerLens

文件	最后提交记录	最后更新时间
compatibility	CI Warnings Cleanup (#1732) * Clean up warnings in CI * format fixes * regenerated lock file	24 天前
generalized_components	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
sources	fix(native): honor initializer_range and init_weights in native boot — mirror of dev-4.x b27b7c65 + 82104b8f hunks (#1577, #1614) boot_native ignored the config's initialization contract: the -1.0 initializer_range sentinel fell through to std=0.02 instead of 0.8/sqrt(d_model) in gpt2 mode, xavier/kaiming ignored initializer_range as gain, and init_weights=False still ran custom init. The sentinel now resolves at point of use (cfg keeps -1.0, matching HookedTransformerConfig consumers), gain threads through _NON_RESIDUAL_MODES, init_weights gates initialization, and bridge.init_weights() reinitializes natives in place. Residual 1/sqrt(2*n_layers) output scaling is kept and documented as an intentional delta from HookedTransformer. (mirrored from commits b27b7c65 and 82104b8f's native hunks; boot changes applied to TransformerBridge.boot_native — dev's home for the native boot path. LNPre/RMSPre tests from the 4.x context stay behind with their unmirrored feature. test_native_init_semantics.py taken at dev-4.x HEAD) Co-authored-by: Guan Tong <tguan@gmail.com> Co-authored-by: Jonah Larson <jonahalarson@comcast.net>	1 个月前
supported_architectures	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
__init__.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_alias_collection_traversal.py	Chore/comment cleanup (#1674) * Hook extensions for specialized bridges * Additional bug fixes * Comment cleanup	1 个月前
test_architecture_adapter.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_architecture_adapter_factory.py	Feat/external architecture registration (#1307) * feat: External architecture adapter registration and entry-point discovery * docs: External adapter registration guide with examples * test: ArchitectureAdapterFactory registration, selection, and entry-point tests * fix: Clarify matching requirement and add warnings for failed discovery * fix: Make register_adapter doctest self-contained with inline adapter class * fix: Address Copilot review - alias, loop isolation, test state leak * fix: Apply black formatting to factory file * fix: use select_architecture_adapter in register_adapter doctest * fix: guard native adapter override in discover_entry_points * test: drop TestSupportedArchitectures * test: add entry-point discovery tests * docs: document native adapter override prevention * fix: apply black formatting to test file * fix: handle None ep.dist in entry-point guard warning * fix: apply black formatting to factory file	4 个月前
test_batched_moe_fold_ln.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_batched_string_padding.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_benchmark_gated_hooks_fire.py	Split Q/K/V inputs with per-head attention results (#1257) * Fixing issue with exposed hooks * verification * Fix typing and formating * Improved QKV split attention/block system * Resolving failures * fixed hook error	5 个月前
test_boot_native.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_bridge_generate_bos.py	feat: respect prepend_bos and add return_input_tokens flag (#1439) * tidy: Remove reference to deleted additional comments (#1410) Simple resolution for #1409 * feat: respect prepend_bos and add return_input_tokens flag * added tests and generate_stream --------- Co-authored-by: Dan Raviv <dan.raviv@gmail.com>	2 个月前
test_bridge_generate_kv_cache.py	KV Cache implementation for AttentionBridge (#1232) * Setup KV Cache for Joint QKV Attention path * Adapted for usage on all attention implementations * Added change to layer_idx to make sure it propagates to all chidlren of AttentionBridge * Updated to resolve additional issues and reduce duplication in the AttentionBridge system * Cleaning up duplicate cache instances that had the potential to conflict with one another * encoder decoder error fix	5 个月前
test_bridge_generate_no_tokenizer.py	August 19th Verification Sweep Fixes (#1705) * kv cache layer_past issue, MoE norm gains * Bloom fixes, qwen fixes, gemma norm values * Improved backwards gradient testing	1 个月前
test_bridge_str.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_bridge_train_mode_propagation.py	Add TransformerBridge adapter for lightweight decoder-only pretraining models (#1519) * Add TransformerLens pretrain bridge adapter * Add TransformerLens pretrain bridge adapter	2 个月前
test_bridge_vs_hooked_transformer_mlp_in_patching.py	Raise clear error when adding hooks to gated-off hook points (#1696) * Raise clear error when adding hooks to gated-off hook points * Warn instead of erroring for filter/callable matches on gated hooks; only warn in run_with_cache when names_filter is explicit * Fix duplicated apply_hooks function and misplaced warning message causing mypy failures * Fix run_with_cache gated hook handling * Update gated MLP hook compatibility test * Format gated MLP hook test --------- Co-authored-by: Jonah Larson <jonahalarson@comcast.net>	1 个月前
test_bridge_vs_hooked_transformer_patching.py	Compatibility Mode – pre-ln split qkv hooks (#1319) * Initial resolution of issue with formerly pre-ln hook firing post-ln * Resolved pre-ln issues * setup arena to use new flag	4 个月前
test_bridge_weight_properties.py	Reducing v3.x beta code redundancies (#1228) * First pass comment cleanup * Second pass comment cleanup * Reducing redundancies between generalized components, architecture adapters, and benchmarks. Testing new utility functions * Test updates, CI Prep * Patching adjustments * Add gpt2-xl to cache * Increase timeout * Different timeout extension * Add a patching test outside of the notebook to make sure we get full coverage * formatting	5 个月前
test_build_bridge_from_module.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_checkpoint_revision.py	Clean up Tests and increase Full Coverage Test Time limit (#1371) * Clean up tautologies and duplicate test coverage * Fix hanging issue	3 个月前
test_component_hooks_fire.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_component_inspection.py	Test Suite Review (#1235) * Test Suite Review * mypy cleanup * Test failure cleanup	5 个月前
test_component_setup.py	Add forward-only LLaDA architecture adapter (#1512) Co-authored-by: jlarson4 <jonahalarson@comcast.net>	2 个月前
test_config_flag_assignment.py	Fix direct TransformerBridge config flag assignment (#1709) * Fix direct Bridge config flag assignment * Address Bridge config assignment review feedback * Warn on reused live Bridge configs	1 个月前
test_config_mapping.py	Fixing edge cases on heterogeneous configs (#1679)	1 个月前
test_diffusion_generate.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_end_to_end_bridge.py	Add support for layer norm and bias folding (#1044) * Split weights instead of logits for models with joint QKV activation * Adjust tests accordingly * Set split_qkv_matrix function inside init * Remove debugging print statements * Add support for folding layer norm and folding value biases * Enable layer norm folding by default in compatibility mode * Remove old parameters * Remove hardcoded filepath * Make sure conversion rules are not none * Make sure conversion rules are not none * ran format * optimized tests a bit * remvoed extra files * resolved test * fixed test * removed extra block * removed extra variable * restored hooks * cleaned up imports * Remove conversions out of layer norm folding * Add configuration dictionary during initialization * Fix typing error * Do not use weights and biases if weights are folded * Add uses_rms_norm configuration parameter --------- Co-authored-by: Bryce Meyer <bryce13950@gmail.com>	1 年前
test_expand_kv_heads.py	fix(bridge): expand grouped K/V heads in QK/OV and composition circuits — mirror of dev-4.x 67734250 (#1593) QK/OV factored-matrix accessors and all_composition_scores crashed on GQA models: grouped W_K/W_V (n_kv_heads) never expanded to n_heads before the einsums. _expand_kv_heads repeat_interleaves K/V weights and biases at the accessor boundary; MHA models are a documented no-op (bit-identical outputs). (path-retargeted from commit 677342504975da414d59b7292342966fa4f28d43: transformer_bridge.py hunks applied to bridge.py; test module imports retargeted to dev's bridge module)	1 个月前
test_fused_qkv_fold_ln.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_gated_hooks.py	Raise clear error when adding hooks to gated-off hook points (#1696) * Raise clear error when adding hooks to gated-off hook points * Warn instead of erroring for filter/callable matches on gated hooks; only warn in run_with_cache when names_filter is explicit * Fix duplicated apply_hooks function and misplaced warning message causing mypy failures * Fix run_with_cache gated hook handling * Update gated MLP hook compatibility test * Format gated MLP hook test --------- Co-authored-by: Jonah Larson <jonahalarson@comcast.net>	1 个月前
test_generate_stream_batch_decode.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_generation_benchmark_mechanics.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_generation_capability_flags.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_get_params_util.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_get_params_util_helpers.py	fix(bridge): rewrite get_bridge_params on TL-layout accessors with GQA expansion — mirror of dev-4.x 29854203 + 82104b8f hunks (#1603, #1614) get_bridge_params read raw module weights with shape-guessing reshapes: processed/converted layouts were misread, GQA models' grouped K/V never expanded (SVDInterpreter saw n_kv_heads-shaped W_K/W_V), and absent weights zero-filled silently with no signal. The rewrite reads the components' TL-layout accessors (W_Q/W_in/... , post-conversion and post-processing), expands grouped K/V and biases to n_heads via repeat_interleave, emits ln params so consumers can detect fold state, warns on zero-fills, and raises on cfg/blocks inconsistency. (extracted from mixed commits 29854203 and 82104b8f; the file is taken at dev-4.x HEAD, which already merges dev's interleaved-MoE dense handling from #1666 — upstream resolved that merge and this reuses its resolution. tests/unit/model_bridge/test_get_params_util.py taken at the same point. Dev's test_get_params_multi_query_attention_reshaping is removed as upstream did in reanchoring: it monkeypatched raw weights against the old shape-guesser and is superseded by test_grouped_kv_expanded_to_n_heads / test_grouped_biases_expanded under the accessor contract)	1 个月前
test_gpt_oss_moe.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_hf_generate_batch_padding.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_hook_alias_resolution.py	siglip vision tower hooking (#1734) * siglip vision tower issues * Removed expected failures	24 天前
test_lfm2_fold_ln.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_loss_attention_mask.py	fix(bridge): mask-aware causal loss — mirror of dev-4.x 7ebeab96 (#1608) Padding-aware TransformerBridge loss: -inf x 0 = nan no longer leaks into the reduced loss (lm_utils masked_fill), fully-masked padding rows no longer softmax to NaN in native attention, and forward() threads attention_mask into loss_fn for return_type='loss'/'both' with 2D/4D mask reduction. (cherry picked from commit 7ebeab969b2f5f35fd6dee4a98db3dbaf703e4cd, adapted for dev: bridge_core/transformer_bridge hunks applied to bridge.py's inlined dispatch; RemoteBridge portions dropped — class absent on dev) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_mistral_attn_in_fork.py	Mistral: use fork-capable attention bridge (enable hook_attn_in / set_use_attn_in) (#1497) * Mistral: use fork-capable PositionEmbeddingsAttentionBridge Mistral has separate q/k/v/o projections with RoPE and GQA, structurally identical to Qwen2, but its adapter used the plain AttentionBridge, which delegates q/k/v to HF and exposes no per-receiver fork point. As a result set_use_attn_in() raised and blocks.{i}.hook_attn_in did not exist, so per-head input interventions (activation/path patching on attention inputs) were impossible on Mistral. Switch the Mistral adapter to PositionEmbeddingsAttentionBridge (the same fork-capable bridge Qwen2/Qwen3 use) and pass config to its BlockBridge so hook_mlp_in is wired too. Add a test booting the tiny random Mistral that checks the attention bridge is fork-capable, hook_attn_in fires at [batch, pos, n_heads, d_model], and zeroing a single head's forked input actually changes the logits. * Skip non-fp params when picking attention compute dtype PositionEmbeddingsAttentionBridge used the first param's dtype, which is integer storage on quantized models (uint8 bnb, int32 GPTQ), so fp hidden states got cast to it. Ported the guard from AttentionBridge. Also updates the two Mistral tests that still assumed a plain AttentionBridge. The quantized one reads the dtype at q_proj now, since the reimplemented bridge never calls the HF attention forward.	2 个月前
test_n_params_total.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_native_features.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_nonpersistent_buffer_restoration.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_optimizer_compatibility.py	Architecture Gaps Fable Sweep (#1542) * feat: add MarianMTModel architecture adapter Adds TransformerBridge support for the Helsinki-NLP opus-mt translation family (154 models). Marian shares Bart's post-LN encoder-decoder block layout; deltas handled here: no layernorm_embedding, deterministic sinusoidal position embeddings, sqrt(d_model) embedding scale surfaced as cfg.scale_embedding (new TransformerBridgeConfig field), and the trained final_logits_bias covered by an integration parity test. PosEmbedBridge now skips hook_in for non-tensor first args — Marian's sinusoidal embedding receives a torch.Size, which previously crashed under runtime type checking. Verified: Helsinki-NLP/opus-mt-en-de P1=100 P2=100 P4=89 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3.5-MoE architecture adapters (text + vision-language) Adds Qwen3_5MoeForCausalLM and Qwen3_5MoeForConditionalGeneration support by composing existing infrastructure: the Qwen3.5 hybrid GatedDeltaNet + full-attention backbone with the Qwen3-Next-style sparse MoE MLP (256 experts, top-8, shared expert), and the Qwen3.5 vision tower for the VL variant. Registers the VL class in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: trl-internal-testing/tiny-Qwen3_5MoeForConditionalGeneration-3.6 P1=100 P2=100 P3=100 P4=80.3 (fp32, cpu); exact logit parity (0.0). Canonical 35B checkpoints (Qwen/Qwen3.5-35B-A3B, Qwen/Qwen3.6-35B-A3B) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MiniMaxM2ForCausalLM architecture adapter Pre-norm RMS decoder with RoPE (partial rotary 0.5), GQA, and sparse MoE. Quirks handled: full-projection-width Q/K RMSNorm (pre-head-reshape, unlike Qwen3's per-head norms), DeepSeek-V3-style sigmoid router with trained e_score_correction_bias (custom module, so the MoE block fully delegates), explicit head_dim larger than hidden/heads, and no BOS prepending (verified against the MiniMaxAI/MiniMax-M2 tokenizer). Verified: tiny-random/minimax-m2 P1=100 P2=100 P3=100 P4=70.2 (fp32, cpu); exact logit parity (0.0). Canonical M2/M2.1/M2.5/M2.7 (100B+) registered at status 0 — need big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register T5WithLMHeadModel as a T5 adapter alias Old google-t5 checkpoints (t5-3b, t5-11b) carry the legacy class name T5WithLMHeadModel in config.architectures, which previously failed factory lookup. Registers the existing T5ArchitectureAdapter under the alias and adds it to SEQ2SEQ_ARCHITECTURES so boot selects AutoModelForSeq2SeqLM. Verified: google-t5/t5-3b P1=100 P2=100 P4=64.4 (fp32, cpu), consistent with sibling T5ForConditionalGeneration checkpoints. t5-11b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4MoeLiteForCausalLM (GLM-4.7-Flash) architecture adapter Composes DeepSeek-V2-style MLA (LoRA-compressed Q/KV, nope/rope split) with GLM's sparse MoE: sigmoid router + e_score_correction_bias, batched routed experts, shared expert, and a per-layer dense/sparse mix from config.mlp_layer_types (router and shared expert mapped optional). Fixes MLAAttentionBridge to honor config.rope_interleave: the flag was read but never applied, so the hooked attention reconstruction used the non-interleaved rotation and diverged (P1 component max_diff ~5e-3) on interleaved checkpoints. DeepSeek-V3 also defaults rope_interleave=True, so its component-testing path is corrected by the same change; DeepSeek V2/V3 integration suites pass unchanged. Verified: tiny-random/glm-4.7-flash P1=100 P2=100 P3=85 P4=69.3 (fp32, cpu); exact full-forward parity (0.0). zai-org/GLM-4.7-Flash (30B+) registered status 0 — needs big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add ExaoneForCausalLM (EXAONE-3.x) architecture adapter LG AI Research's EXAONE-3.0/3.5/Deep family ships trust_remote_code modeling that follows current HF conventions (Cache API, position_embeddings), so standard bridges delegate cleanly. Quirks: GPT-2-flavored paths (transformer.wte / h / ln_f, rotary at transformer.rotary), double-nested attention (attn.attention with out_proj), gated MLP as c_fc_0 (gate) / c_fc_1 (up) / c_proj (down), no BOS prepending. Adds LGAI-EXAONE/ to verify_models' remote-code prefixes. Integration test is CI-gated (2.4B download; the only tiny mirror ships stale remote code) — QUARANTINES.md row added. Verified: LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct P1=100 P2=100 P3=100 P4=98.8 (fp32, cpu); exact logit parity (0.0). EXAONE-4.0 (native transformers, separate architecture) remains in the queue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add M2M100ForConditionalGeneration architecture adapter Covers Meta's M2M100 and NLLB-200 multilingual translation families. Same projection layout as Bart/Marian but PRE-norm, with an extra final LayerNorm after each stack (encoder_ln_final / decoder_ln_final, T5-style keys), and the sqrt(d_model) embedding scale baked into M2M100ScaledWordEmbedding — so embed hooks observe scaled output (opposite of Marian; integration-tested). PosEmbedBridge.W_pos now falls back to a "weights" buffer: M2M100's sinusoidal table is a plain nn.Module buffer, not an nn.Embedding weight, and the old assert crashed hook-registry scanning at boot. Verified: facebook/m2m100_418M P1=100 P2=100 (fp32, cpu); exact logit parity. P4=38.6 investigated, not an adapter bug: generation is token-identical to raw HF, and with forced_bos_token_id the model translates correctly — the P4 harness generates without a target-language token, which M2M100 requires by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FalconMambaForCausalLM architecture adapter TII's FalconMamba is Mamba-1 with a parameter-free RMS applied to the B, C, and dt projections inside the mixer. Module paths are identical to Mamba, so the adapter inherits the Mamba mapping wholesale. SSMMixerBridge now honors that RMS (keyed on the wrapped mixer's rms_eps attribute) in both the opt-in eager S6 scan and the S6-terms reconstruction (B/C only there — dt_proj.hook_out is already post-RMS). Integration tests include a counterfactual proving the RMS branch is load-bearing: hiding rms_eps degrades eager-scan parity from ~2e-7 to >1e-3. Verified: tiiuae/falcon-mamba-tiny-dev P1=100 P2=100 P3=100 (fp32, cpu); exact logit parity. P4=24.5 reflects the untrained tiny-dev checkpoint; falcon-mamba-7b registered status 0 for big-download verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BambaForCausalLM architecture adapter; fix hybrid stateful generation IBM's Bamba: Jamba-lineage hybrid where every layer carries a dense gated MLP (feed_forward: gate/up/down) and two RMSNorms, with the token mixer switching between Mamba-2 (.mamba, wired under the canonical .mixer slot) and llama-style GQA attention per config.layers_block_type. Final norm is model.final_layernorm. Fixes the stateful generation loop for modern hybrids: it hard-coded the Mamba-1 cache_params kwarg and conv-window cache positions, which cascaded a duplicate cache_params into hybrid layers (crash) — Bamba, NemotronH, and FalconH1 all take past_key_values at the top level. The loop now picks the kwarg from the model's forward signature and uses full-prompt cache positions for the past_key_values path. NemotronH's create_stateful_cache now passes config to DynamicCache (required for per-layer type detection; matches HF's own init) — its greedy generation now matches HF bit-for-bit. Bamba's final-token divergence vs hf.generate was measured, not assumed: the bridge is bit-identical to the wrapped model; HF Bamba itself shows 1.9e-2 cached-vs-uncached spread at its attention layer on the random-weight tiny checkpoint, exceeding the 2.7e-3 top-2 margin at the flipped position. Verified: hmellor/tiny-random-BambaForCausalLM P1=100 P2=100 P3=100 P4=61.8 (fp32, cpu); exact forward parity. Bamba-9B-v1/v2 registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add MBartForConditionalGeneration architecture adapter Multilingual BART: Bart's learned positional embeddings and per-stack layernorm_embedding combined with M2M100's pre-norm layer layout and per-stack final LayerNorms. Embedding scale is baked into MBartScaledWordEmbedding (surfaced as cfg.scale_embedding). Fixes decoder_start_token_id handling for MBart-family checkpoints that leave it unset (e.g. ai4bharat/IndicBART): both the bridge's enc-dec generation loop and the forward_pass benchmark used getattr defaults that never fire when the attribute exists as None, crashing with "Could not infer dtype of NoneType". Both now use HF's bos->eos fallback chain. Verified: ai4bharat/IndicBART P1=100 P2=100 (fp32, cpu); exact logit parity. P4=32.5 is intrinsic (multilingual model generating without a language tag — continuation is token-identical to raw HF generate). mbart-large-50 family registered status 0. sshleifer/tiny-mbart excluded: asymmetric and mislabeled as Bart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OpenAIGPTLMHeadModel (GPT-1) architecture adapter The original GPT: GPT-2's combined-QKV Conv1D internals under transformer.tokens_embed / positions_embed, POST-norm (ln_1 after attention, ln_2 after MLP), and no final LayerNorm. An adapter-local attention bridge returns lists — GPT-1's Block concatenates [h] + attn_outputs[1:] and crashes on tuples. Component benchmark hardening for pre-kwarg-era modules: shared-input calls retry positionally when hidden_states is rejected, and the output comparator accepts list-typed returns. Verified: openai-community/openai-gpt P1=100 P2=100 P3=100 P4=85.9 (fp32, cpu); exact logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add PegasusForConditionalGeneration architecture adapter Google's PEGASUS summarization family: pre-norm encoder-decoder with per-stack final LayerNorms (M2M100 layout), Marian-style deterministic sinusoidal positions, and the sqrt(d_model) embedding scale applied in the stack forward (embed hooks observe unscaled output — integration-tested). Verified: google/pegasus-xsum P1=100 P2=100 P4=100 (fp32, cpu); exact logit parity (0.0). Integration test CI-gated (568M download; distilled variants are asymmetric) — QUARANTINES.md row added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add SeedOssForCausalLM adapter; fix dropout firing at inference in bridge wrappers ByteDance's Seed-OSS is a Llama-layout decoder (thin subclass) with config-gated attention/MLP biases and no BOS prepending. Its tiny test checkpoint ships dropout=0.1, which exposed two latent cross-cutting bugs: 1. Bridge wrappers were permanently stuck in training mode: GeneralizedComponent.__setattr__ redirected `self.training = mode` to the wrapped HF module, so train()/eval() never changed the wrapper's own flag and attention-reconstruction dropout fired at inference (nondeterministic forward for any config with nonzero dropout). `training` is now always set locally; recursion still reaches the original module via _modules. TransformerBridge.__init__ also re-syncs the whole tree to the wrapped model's mode, since wrappers are inserted after from_pretrained's eval(). 2. get_tokenizer_with_bos crashed on special tokens containing ':' (Seed-OSS's <seed:bos>): tokenizers' TemplateProcessing uses ':' as the piece separator. Falls back to the original tokenizer. Verified: tiny-random/seed-oss P1=100 P2=100 P3=100 P4=74 (fp32, cpu); deterministic-forward regression test added. Seed-OSS-36B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NemotronForCausalLM architecture adapter NVIDIA's dense Nemotron-3/4 and Minitron families: llama-shaped GQA decoder with LayerNorm1P normalization (zero-centered gamma applied as weight + 1), a non-gated squared-ReLU MLP, and partial rotary embeddings. Norm bridges delegate to the native modules (use_native_layernorm_autograd) — the generic LN reconstruction would drop the +1 offset — and LN folding is disabled for the same reason. Also removes a pre-existing duplicate NemotronHArchitectureAdapter entry in the package __all__. Verified: nvidia/Minitron-4B-Base P1=100 P2=100 P3=100 P4=97 (fp32, cpu); exact logit parity. badaoui's tiny-random checkpoint is excluded from the registry — its tokenizer emits ids beyond the truncated 32k vocab. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: register BloomModel and BertLMHeadModel adapter aliases Two headless/variant class names sharing existing adapters' module trees: - BloomModel (bigscience-small-testing, TurkuNLP gpt3-finnish, norbloom): loads as BloomForCausalLM with tied embeddings -> BloomArchitectureAdapter. - BertLMHeadModel (decoder-style BERT with causal LM head: chemlm, TILDE, BEREL): identical module tree to BertForMaskedLM -> BertArchitectureAdapter. Both smoke-tested with exact logit parity (0.0) on bigscience/bigscience-small-testing and sagawa/molscaletransfer-chemlm-0.06m. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Idefics3ForConditionalGeneration architecture adapter HuggingFace's Idefics3 / SmolVLM lineage (granite-docling, SmolVLM, Idefics3-8B): SigLIP-style vision transformer at model.vision_model, a pixel-shuffle connector at model.connector, and a llama-shaped text model at model.text_model with a top-level lm_head. Registered in MULTIMODAL_ARCHITECTURES so boot selects AutoModelForImageTextToText. Verified: ibm-granite/granite-docling-258M P1=100 P2=100 P3=95 P4=92.1 (fp32, cpu); exact text-path logit parity (0.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2AudioForConditionalGeneration architecture adapter Alibaba's Qwen2-Audio: Whisper-style audio encoder at audio_tower, linear projector, and a full Qwen2ForCausalLM at language_model (text paths under language_model.model.* with language_model.lm_head). The model loads via AutoModelForSeq2SeqLM (new AUDIO_TEXT_ARCHITECTURES group) but classifies as causal_lm so the bridge does not apply encoder-decoder semantics. Component benchmarking skips the audio tower alongside vision towers — isolated text-shaped inputs cannot satisfy a 3000-frame mel encoder. Verified: trl-internal-testing/tiny-Qwen2AudioForConditionalGeneration P1=100 P2=100 P3=100 P4=68.8 (fp32, cpu); exact text-path parity (0.0). Qwen2-Audio-7B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5ForCausalLM adapter; support adjacent-pair RoPE in attention reconstruction Baidu's dense ERNIE 4.5 is a Llama-layout decoder with config-gated biases, no BOS prepending, and GLM-style interleaved RoPE — it rotates adjacent element pairs in fp32, not llama's half-split convention. PositionEmbeddingsAttentionBridge's reconstruction hard-coded llama's apply_rotary_pos_emb; on ERNIE this diverged from HF by up to 13.6 logits (measured, deterministic). The reconstruction now honors the existing cfg.rotary_adjacent_pairs flag with a faithful adjacent-pair implementation. Verified: baidu/ERNIE-4.5-0.3B-PT P1=100 P2=100 P3=100 P4=98.9 (fp32, cpu). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmForCausalLM architecture adapter Z.ai's dense GLM-4 family (glm-4-9b--hf, glm-edge): llama-shaped GQA decoder with adjacent-pair interleaved RoPE at a partial rotary factor (reuses the cfg.rotary_adjacent_pairs reconstruction path landed with ERNIE), attention biases, no BOS token, and a Phi3-style combined gate_up_proj MLP (reuses Phi3's splitter; LN folding disabled for the joint projection). Verified: zai-org/glm-edge-1.5b-chat P1=100 P2=100 P3=100 P4=96.9 (fp32, cpu); exact logit parity (0.0). glm-4-9b-chat-hf registered status 0. Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Starcoder2ForCausalLM architecture adapter BigCode's StarCoder2 code models: pre-norm decoder using plain LayerNorm (not RMS) despite the llama-like shape, separate biased q/k/v/o projections, GQA, RoPE, and a non-gated c_fc/c_proj MLP. Verified: optimum-intel tiny-random-Starcoder2 P1=100 P2=100 P3=100 (fp32, cpu; P4=35.3 reflects random weights); exact logit parity (0.0). starcoder2-3b/7b/15b registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BitNetForCausalLM architecture adapter Microsoft's BitNet b1.58 (bf16 master weights): llama layout plus sub-layer normalization — an RMSNorm on the attention output before o_proj and on the MLP activation before down_proj. Adds an overridable _pre_output_projection seam to PositionEmbeddingsAttentionBridge (the o projection was inlined, so adapter subclasses had no hook point); BitNet's bridge applies attn_sub_norm there. Without it the reconstruction diverged by hundreds of logits. Sub-layer norms are incompatible with HT-style processed-weight attention, so LN folding, W_O centering, and Phase-3 compatibility equivalence are scoped out (applicable_phases = [1, 2, 4]). Verified: microsoft/bitnet-b1.58-2B-4T-bf16 P1=100 P2=100 P4=94 (fp32, cpu). Integration test CI-gated (QUARANTINES row). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Exaone4ForCausalLM architecture adapter LG AI Research's EXAONE 4.0 (native transformers, distinct from the remote-code 3.x family): llama-shaped GQA decoder with per-head Q/K RMSNorms, hybrid sliding/global attention via layer_types, and post-norms applied inside the residual branch (no pre-norms) — LN folding and W_O centering scoped out accordingly. Verified: LGAI-EXAONE/EXAONE-4.0-1.2B P1=100 P2=92.3 P3=100 P4=67.4 (fp32, cpu). EXAONE-4.0-32B registered status 0. Integration test CI-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add BlenderbotForConditionalGeneration architecture adapter Meta's Blenderbot dialogue family: Pegasus-style pre-norm encoder-decoder with per-stack final LayerNorms and stack-forward embedding scale, but learned positions. First asymmetric-stack adapter — all public checkpoints pair a small encoder with a large decoder (2/12 on 400M-distill), so only heads and FFN width are required to match and cfg.n_layers follows the decoder. The component benchmark now derives per-stack lengths from the bound block lists instead of cfg.n_layers. Verified: facebook/blenderbot-400M-distill P1=100 P2=100 P4=78 (fp32, cpu); parity 2.1e-5 vs fresh HF; coherent greedy dialogue. blenderbot-3B registered status 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Ernie4_5_MoeForCausalLM architecture adapter Baidu ERNIE 4.5 MoE: reuses dense-ERNIE conventions (GLM-style interleaved RoPE via rotary_adjacent_pairs, no BOS prepend) with a MoEBridge MLP — sigmoid-corrected top-k router fully delegated, optional shared_experts GatedMLPBridge (absent in the dense-MLP prefix before moe_layer_start_index). Verified: yujiepan/ernie-4.5-moe-tiny-random P1=100 P2=100 P3=90.5 P4=69.4 (fp32, cpu); baidu/ERNIE-4.5-21B-A3B-PT registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForCausalLM architecture adapter Meta Llama 4 text decoder: attention stays fully delegated (maintain_native_attention) since Llama4 uses complex-tensor interleaved RoPE, NoPE layers with temperature tuning, post-RoPE weightless L2 QK-norm, and chunked attention masks. Feed-forward is a MoEBridge over feed_forward with an optional shared expert (dense gated MLP on non-MoE layers); the tuple-returning router stays unwrapped and hook_router_scores captures its logits. Two adapter-local shims: _Llama4MoEBridge fires hook_out in [batch, seq, d_model] (HF flattens to [batchseq, d]), and _Llama4SharedExpertBridge clones its output under grad because Llama4TextMoe accumulates routed output in place, which autograd forbids on backward-hook views. Verified: trl-internal-testing/tiny-Llama4ForCausalLM P1=100 P2=100 P3=100 P4=75.7 (fp32, cpu) on a local snapshot with the four feed_forward.experts tensors re-initialized (seed 42) — the upstream checkpoint ships uninitialized expert weights (NaN/1e38), so its raw forward is NaN on HF itself. No full-size public text-only checkpoint exists; Meta ships the multimodal Llama4ForConditionalGeneration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add AfmoeForCausalLM architecture adapter Arcee Trinity (AFMoE): sandwich-normalized blocks (pre+post norms around attention and MLP, Gemma-2 layout), attention delegated natively (per-head QK RMS-norm, sigmoid output gating via a separate gate_proj, RoPE only on sliding-window layers with NoPE full-attention layers). MoEBridge over mlp with optional router.gate Linear and shared-experts GatedMLP — layers below num_dense_layers hold a plain gated MLP under the same name; the tuple-returning token-choice router itself stays unwrapped. Verified: onnx-internal-testing/tiny-random-AfmoeForCausalLM P1=100 P2=100 P3=100 P4=47.4 (fp32, cpu); arcee-ai/Trinity-Mini registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LongT5ForConditionalGeneration architecture adapter Google LongT5: subclasses the T5 adapter, swapping the encoder stack for the local / transient-global attention module selected by encoder_attention_type (new _HF_PASSTHROUGH_ATTRS entry, both copies). Encoder attention stays delegated to HF — its block-wise position bias is [1, 1, heads, block, 3block], which the generic reconstruction cannot supply. Decoder mapping is inherited unchanged. Registered as a seq2seq architecture for loader auto-class selection. Verified: google/long-t5-tglobal-base P1=100 P2=100 P4=49.2 (fp32, cpu; P3 skipped, T5 family disables fold_ln). google/long-t5-local-base passes 233/234 components including the local-attention encoder; its only failure is the unembed — the checkpoint ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (pretraining artifact; noted in registry, not an adapter bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Florence2ForConditionalGeneration architecture adapter Microsoft Florence-2 (florence-community native ports): subclasses the BART adapter, re-prefixing the text stack under model.language_model; the DaViT vision tower and multi-modal projector are opaque delegated components. First multimodal encoder-decoder in the bridge — three cross-cutting fixes it surfaced: - GeneralizedComponent.forward now hooks last_hidden_state on ModelOutput-returning components (vision/audio towers previously hit a beartype violation when actually run with images). - Multimodal benchmark input prep probes for processors that auto-insert image placeholders (manual placeholder double-counted) and uses <CAPTION> for task-prompt captioners exposing post_process_generation. - multimodal_generation success check now understands enc-dec generate() returning decoder tokens only. Verified: florence-community/Florence-2-base-ft P1=100 P2=100 P7=100 (fp32, cpu; P3 skipped, post-LN BART disables fold_ln). P4 text_quality=0 is a domain artifact — Florence-2 emits EOS on free-text prompts; task-prompt captioning through the bridge produces correct captions (integration-tested: red square described). Florence-2-large-ft registered for follow-up verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4ForCausalLM architecture adapter Z.ai GLM-4-0414 / GLM-Z1: subclasses the dense GLM adapter (adjacent- pair partial RoPE, joint gate_up_proj MLP with the Phi3 splitter), rebuilding the block with GLM-4-0414's sandwich norms — GLM keeps its naming, so post_attention_layernorm is the pre-MLP norm (ln2) while post_self_attn_layernorm (ln1_post) and post_mlp_layernorm (ln2_post) normalize the sublayer outputs before their residual adds. Verified: tiny-random/glm-4 P1=100 P2=100 P3=100 P4=66.7 (fp32, cpu); zai-org/GLM-4-32B-0414 registered for big-hardware verification. snake7gun/tiny-random-glm4 was rejected as a verification target: it ships no lm_head.weight with tie_word_embeddings=false, so HF randomly re-initializes the head each load (bridge matches HF exactly through ln_final on it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LEDForConditionalGeneration architecture adapter AllenAI Longformer Encoder-Decoder: subclasses the BART adapter, re-prefixing the stack under `led.` and rebuilding the encoder around Longformer sliding-window + global attention — q/k/v live inside longformer_self_attn with an `output` projection, window chunking and the _global projections stay HF-native (maintain_native_attention). The encoder query projection is cloned under grad because Longformer scales it with an in-place `/=`, which autograd forbids on backward-hook views. Encoder hooks carry the window-padded length (1024), faithful to HF's internal flow. Verified: allenai/led-base-16384 P1=100 P2=100 P4=50.5 (fp32, cpu; P3 skipped, post-LN BART family disables fold_ln); allenai/led-large-16384-arxiv registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add Mistral3ForConditionalGeneration architecture adapter Mistral AI Mistral-Small-3.x VLM: subclasses the Llava adapter (same vision_tower / multi_modal_projector / language_model layout with a Mistral text decoder), swapping both vision components to opaque delegated GeneralizedComponents — Pixtral's 2D-RoPE block-diagonal attention has no CLIP/SigLIP-shaped bridge, and the patch-merging projector takes (image_features, image_sizes), which VisionProjectionBridge's single-input forward cannot accept. Verified: tiny-random/mistral-3 P1=100 P2=100 P3=100 P4=97 P7=100 (fp32, cpu); text and multimodal forwards match fresh HF exactly (integration-tested). mistralai/Mistral-Small-3.1-24B-Instruct-2503 registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GlmAsrForConditionalGeneration architecture adapter Z.ai GLM-ASR-Nano: thin subclass of the Qwen2-Audio adapter — identical audio_tower / multi_modal_projector / language_model layout, and the Llama text stack shares Qwen2's module names, so the mapping applies unchanged. Registered as an audio-text architecture (loads via AutoModelForSeq2SeqLM). Verified: zai-org/GLM-ASR-Nano-2512 P1=100 P2=100 P3=100 P4=94.8 (fp32, cpu, real 2.5B checkpoint; generation coherent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Gemma4ForCausalLM architecture adapter Google Gemma 4 bare text decoder: subclasses the multimodal Gemma4 adapter, dropping the vision components and re-prefixing the text stack from model.language_model.* to model.* (the CausalLM class holds Gemma4TextModel directly). Inherits applicable_phases=[1,2,4] — the PLE / layer_scalar / MoE topology is not fold-safe. Verified: veyra-ai/Kairo-5M-Gemma4-Base P1=100 P2=100 P4=53.3 (fp32, cpu; 5M community checkpoint — no official text-only release exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Llama4ForConditionalGeneration architecture adapter Meta Llama 4 Scout/Maverick composite: subclasses the Llama4 text adapter, re-prefixing the text stack under language_model.* (the composite holds a full Llama4ForCausalLM) and delegating vision_model and multi_modal_projector as opaque components. Verified: yujiepan/llama-4-tiny-random P1=100 P2=100 P3=95 P4=73.2 (fp32, cpu) on a local snapshot with text_config.attn_temperature_tuning coerced to bool — the upstream config declares int 4, which transformers 5.x strict validation rejects (integration fixture applies the same patch). P7 skipped: the tiny ships no processor files. meta-llama/Llama-4-Scout-17B-16E-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add OlmoHybridForCausalLM architecture adapter AllenAI OLMo Hybrid: alternating layer types with mixed norm topologies — OLMo2-style post-norm full-attention layers (no input norm; post_feedforward_layernorm; full-width QK-norm; NoPE mode) and pre-norm GatedDeltaNet linear-attention layers. All per-layer-type submodules (attn, linear_attn, ln1, ln2_post) are optional bindings. Attention stays HF-native; the OlmoHybrid GatedDeltaNet variant keeps separate q/k/v conv states unlike Qwen3Next's, so it is delegated opaquely rather than through GatedDeltaNetBridge's reimplementation. Generation uses the model's own OlmoHybridDynamicCache via create_stateful_cache. Validated on a locally-built seeded tiny-random OlmoHybrid (no tiny exists on the hub): P1=100 P2=100 P3=100 (fp32, cpu); the integration fixture rebuilds the same tiny. allenai/Olmo-Hybrid-7B registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route native-casing MptForCausalLM checkpoints to the MPT adapter Native-transformers MPT checkpoints declare architectures ['MptForCausalLM'] with model_type 'mpt'; the factory only had the legacy remote-code casing 'MPTForCausalLM' and model_type_mappings had no 'mpt' entry, so every native checkpoint raised 'Could not determine supported architecture' despite the adapter existing (11 MPT models are listed as verified in the registry). The existing integration test builds the model programmatically and bypassed hub routing, which is why this went unseen — added a config-driven routing regression test covering both casings. Verified: hf-internal-testing/tiny-random-MptForCausalLM P1=100 P2=92.3 P3=95 (fp32, cpu) — scores identical to the legacy-cased MPT fleet; the gated_hooks_fire miss is the long-standing fused-Wqkv limitation documented on all prior MPT entries, not a regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen2_5_VLForConditionalGeneration architecture adapter Alibaba Qwen2.5-VL (audit Tier-1 #2): windowed ViT tower at model.visual reusing the Qwen vision bridge with Qwen2.5 overrides (rotary_pos_emb instead of learned pos_embed; gated vision MLP), patch merger as projector, Qwen2-layout text decoder under model.language_model. Text attention stays HF-native: mRoPE splits temporal/height/width position streams across rotary channels, so the generic RoPE reconstruction would be text-only-correct but wrong for image runs. default_prepend_bos=False — Qwen tokenizers have no BOS and the prepend fallback injects <\|im_end\|>, which reads as an ended turn and silenced generation on the 3B until fixed. Verified: Qwen/Qwen2.5-VL-3B-Instruct P1=100 P2=100 P3=100 P4=96.3 P7=100 (fp32, cpu, real model with image forward/generation/cache); optimum-intel tiny-random P1-P3=100 P7=100. Multimodal parity vs fresh HF exact on both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLForConditionalGeneration architecture adapter Alibaba Qwen3-VL (audit Tier-1 #3): the vision tower matches the Qwen3.5 vision bridge defaults exactly (learned pos_embed + 2D rotary, qkv/proj attention, fc1/fc2 MLP); Qwen3-VL adds DeepStack — per-level patch mergers on early vision blocks whose features the text model injects into the residual stream at visual token positions. The mergers are wrapped per level (is_list_item template) so DeepStack features are hookable at the source; the injection add itself is a tensor op inside HF's text loop. Text attention stays HF-native (interleaved mRoPE + per-head QK-norm). default_prepend_bos=False per the Qwen convention. Verified: Qwen/Qwen3-VL-2B-Instruct P1=100 P2=100 P3=100 P4=93.7 P7=100 (fp32, cpu, real model); tiny-random/qwen3-vl P1-P3=100 P7=100. Text and multimodal parity vs fresh HF exact; DeepStack merger hooks fire per level (integration-tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Qwen3VLMoeForConditionalGeneration architecture adapter Alibaba Qwen3-VL-MoE (audit Tier-1 #9, shipped with Qwen3-VL): the Qwen3-VL adapter gains a _build_mlp_bridge seam; the MoE variant swaps the dense gated MLP for a MoEBridge — batched experts plus a parameter-only top-k router returning a (logits, scores, indices) tuple, so the router stays unwrapped and the sparse block is delegated whole. Dense mlp_only_layers share the name and delegate identically. Only Qwen3-VL family with DeepStack + expert routing (Qwen3.5 deletes DeepStack), making it the sole MoE+DeepStack interp target. Verified: tiny-random/qwen3-vl-moe P1=100 P2=100 P3=100 P4=72.1 P7=100 (fp32, cpu); text and multimodal parity vs fresh HF exact. Qwen/Qwen3-VL-30B-A3B-Instruct registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add Glm4vForConditionalGeneration architecture adapter Z.ai GLM-4V / GLM-4.1V (audit Tier-1 #4): GLM vision tower delegated opaquely at model.visual with the patch merger as projector; text decoder is the GLM-4-0414 sandwich-norm layout (post_self_attn / post_mlp norms, joint gate_up_proj MLP with the Phi3 splitter) nested under model.language_model. Text attention stays HF-native (mRoPE). Adds explicit QKV bias conversions keyed by kv-head count — the default bias reshape used n_heads and broke weight processing on GQA checkpoints (n_kv=1 tiny exposed it). Verified on a local snapshot of tiny-random/glm-4v with lm_head deterministically initialized (seed 42): every public GLM-4V tiny ships no lm_head.weight with tie_word_embeddings=false (HF re-inits the head randomly per load; bridge matches HF exactly through ln_final on the raw checkpoint). P1=100 P2=100 P3=89.5 P4=65.6 P7=100 (fp32, cpu); text + multimodal parity exact. zai-org/GLM-4.1V-9B-Thinking registered for big-hardware verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(registry): refresh model registry (HF scrape) + verify additional models per architecture Two registry-data updates (supported_models.json shares both): 1. Full HF scrape (2026-07-07): supported models 13,252 -> 15,585; architecture gaps 514 -> 484. The 49 architectures gained an adapter this sweep (or were already supported) drop out of the gaps; 19 new community/gap architectures appear. All prior verification data preserved; validator passes. 2. Additional-model verification (complete): for each completed architecture, the top models by download fitting the ~10B fp32 CPU ceiling (>=1 canonical where available), fp32 serial. 67 attempted: 46 newly verified (registry total 1,066 -> 1,109) including Minitron-8B, EXAONE-3.5-7.8B, falcon-mamba-7b, Bamba-9B v1/v2/1.8T, glm-4-9b family, GLM-4-9B-0414, Qwen2-Audio-7B, starcoder2-3b/7b, pegasus/mbart/nllb/blenderbot/led/long-t5 families, t5-11b. 5 CPU timeouts (3x OLMo-Hybrid-7B pure-torch scan, 2x 8B VLMs). ~10 multimodal models pass P1-P4 but await a P7 backfill for verified status. Failures triaged with accurate notes: BitNet -4T packed quantized (skip; -bf16 is the verified target), long-t5-tglobal-large ships no lm_head (checkpoint defect), Trinity-Nano exposed an AFMoE fold bug (fixed separately), codet5-large legacy layout, plus degenerate community/test checkpoints. Giant-only families (MiniMax-M2, SeedOss-36B, EXAONE-4-32B, 21-35B MoEs, Llama-4, GLM-4.7-Flash) keep only their verified tiny — nothing else fits the CPU ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable fold_ln for AFMoE — sandwich norms are not fold-safe Verifying the real arcee-ai/Trinity-Nano-Preview (6B) exposed what the tiny-random checkpoint masked: AFMoE's post-attention/post-MLP norms scale sublayer outputs before the residual add, so folding ln1/ln2 into the projections changes the function — compat-mode loss diverged to 10.87 vs 2.34 (max logit diff 30.1) while P1/P2/P4 all passed. Same disposition as OLMo2/OlmoHybrid for the same topology: supports_fold_ln=False, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(registry): carry phase4/7/8 scores through ModelEntry round-trips verify_models writes phase1-4 + phase7 + phase8, but ModelEntry only declared phase1-3 — any from_dict/to_dict pass silently reset P4/P7/P8 to None. Adds the three fields to the dataclass, docstring, to_dict, and from_dict, plus a regression test round-tripping a real verified multimodal record (Qwen2.5-VL-3B, P4=96.3/P7=100). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add optional kwarg to normalization bridges (DRY review C1) NormalizationBridge/RMSNormalizationBridge now accept optional=... like AttentionBridge and GeneralizedComponent already do, passing it through to the base. Removes olmo_hybrid's post-construction attribute workaround — per-layer-type norms declare optionality at construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add include_biases to _qkvo_weight_conversions (DRY review D8) Q/K/V bias reshapes are now an option on the shared helper, with K/V keyed by kv-head count — the exact GQA mistake a hand-rolled n_heads reshape makes (hit on the glm4v n_kv=1 tiny). Migrates glm4v and phimoe off their hand-rolled copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share clone-under-grad via CloneOutputUnderGradMixin (DRY review D9) The identical 5-line bodies protecting hook outputs from HF in-place ops (Llama4TextMoe's out.add_, Longformer's query /=) move to one mixin in generalized_components.base; the adapter-local classes keep their names and the why-comment. The pattern recurs whenever HF mutates a wrapped submodule's output in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: add _reprefix_components helper (DRY review D10) The four adapters that reuse a parent mapping under different module nesting (florence2, gemma4_text, led, llama4_multimodal) shared the same name-rewrite loop; it now lives on ArchitectureAdapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: single-source _HF_PASSTHROUGH_ATTRS (DRY review C2) The passthrough list existed in sources/_bridge_builder.py and as a function-local copy in sources/transformers.py, kept in sync only by discipline (every sweep adapter needing a new attr had to edit both). transformers.py now imports the canonical module-level list; _bridge_builder only imports transformers lazily, so no cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Idefics3-family P7: LayerNorm layout contract + processor probe guard The decomposed LayerNorm path preserved the input's strides while F.layer_norm materializes a contiguous output; Siglip embeddings feed a transposed view through the tower, so the bridge handed Idefics3's pixel_shuffle a non-contiguous tensor and its .view() crashed. Restore contiguity on the LayerNorm path (HF RMSNorms are pointwise, so the RMS path keeps preserving layout). Also guard the multimodal auto-insert processor probe: Idefics3-style processors raise on images without image tokens in the text; a failed probe means 'not auto-inserting', not 'no processor'. Registry: granite-docling-258M and SmolVLM-Instruct re-verified (P1=100, P7=100); root-cause notes for Josiefied-Qwen3-VL (broken shard upload, legacy tensor naming + missing layers) and LeChatonFat (no processor files shipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _set_rms_rotary_defaults() helper; delete no-op kv copies and dead default_config dicts D3+D5 from the sweep code review. The six Llama-family config flags (RMS/rotary/final_rms/gated_mlp/attn_only/uses_rms_norm) collapse to one base-adapter helper across 39 adapters; only the exact-match combos were migrated (LN/none/ungated variants untouched). Measured before deleting: adapter cfg is the same object as the ctor arg, so 'self.cfg.n_key_value_heads = cfg.n_key_value_heads' was a no-op (35 sites); instance default_config dicts are read by nothing (the base merges class-level default_cfg) — 11 dicts deleted. The olmo GQA mirror test asserted that dead dict; the behavioral coverage lives in test_kv_conversions_use_n_key_value_heads. Spot re-verified post-refactor: EXAONE-3.5-2.4B full phases (P1-P4 100/ 100/100/98.8, loss match 1.971269) and Qwen3-VL-2B phases 1/4/7 all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _gated_mlp() factory on the base adapter (41 sites) D4+D11 from the sweep code review. The GatedMLPBridge + gate/in/out LinearBridge construction collapses to self._gated_mlp(...) across 38 adapters, including the optional shared_experts blocks in the MoE adapters and the non-default projection names (t5 wi_0/wi_1/wo, qwen w1/w2/c_proj). Only exact-shape sites were rewritten; MoEBridge expert templates and native.py's module-level builder stay explicit. Mistral's site previously omitted config; resolve_activation_fn(None) and its hidden_act both resolve to silu, so passing config is behavior-identical. Spot re-verified post-refactor: EXAONE-3.5-2.4B and Trinity-Nano-Preview P1 green (154 and 499 components equivalent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: base eager-forcing prepare hooks + _extract_vision_dims() helper D6+D7 from the sweep code review. D6: prepare_loading/prepare_model on the base adapter now force eager attention (loading config + loaded model config) when cfg.attn_implementation == 'eager' — composite configs don't reliably inherit the from_pretrained kwarg. The eight identical per-adapter hook pairs (afmoe, glm4v, llama4, olmo_hybrid, qwen2_5_vl, qwen3_vl, lfm2_moe, recurrent_gemma) are deleted; lfm2_moe and recurrent_gemma now declare the cfg flag their hooks were standing in for. PhiMoE keeps its extras (trust_remote_code off, inner-module attr) and delegates the shared part to super(). D7: _extract_vision_dims() handles both HF-standard (num_hidden_layers/num_attention_heads) and Qwen (depth/num_heads) vision-config naming; replaces the copy in 8 multimodal adapters. florence2/llama4_multimodal keep their single-field sites. Spot re-verified: Qwen3-VL-2B and Qwen2.5-VL-3B phases 1/4/7 all green (both families had their hooks deleted; both exercise the vision dims). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: components property replaces per-adapter component_mapping asserts C3 from the sweep code review. Subclass adapters that extend a parent-built mapping needed 'assert self.component_mapping is not None' purely for type narrowing; the base adapter now exposes a 'components' property with the assert inside, and the ten call sites use it. The Optional attribute itself stays — None remains the 'adapter built no mapping' signal that bridge construction errors on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: BartFamilyArchitectureAdapter — six adapters collapse to five knobs D1 from the sweep code review (948 -> 282 lines). The family's block structure is byte-identical across bart/marian/mbart/pegasus/blenderbot/ m2m100; all divergence is declarative: require_symmetric_layers + n_layers_from (Blenderbot's asymmetric stacks), force_scale_embedding (BART checkpoints don't scale), has_layernorm_embedding (BART, MBart), has_final_stack_norm (the pre-LN members). Pre/post-LN ordering itself lives in HF's native forward and never appeared in the mappings. _encoder_block/_decoder_block are overridable so LED's Longformer encoder replacement and Florence2's reprefix+vision additions keep working unchanged (both subclass BartArchitectureAdapter, now a knob declaration on the family base). Spot re-verified post-refactor, scores bit-identical to pre-refactor: m2m100_418M (P1=100 P2=100 P4=38.6), pegasus-xsum (100/100/100), led-base-16384 (100/100/50.5), bart-large-cnn (P1=100 P2=100; P4 is the stochastic generation metric, 67.0 -> 78.7). Family unit tests (85) pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * C4+C5: VisionProjectionBridge args passthrough; shared test helpers C4: VisionProjectionBridge.forward now passes extra positionals through to the wrapped projector, so Mistral3's (image_features, image_sizes) projector can use the semantically-correct bridge (with its hook_vision_proj_ aliases) instead of a bare GeneralizedComponent — mistral3 now inherits Llava's projector entry. Passthrough covered by a new unit test. C5: shared make_bridge_cfg() config factory for per-adapter unit tests and assert_bridge_matches_hf() for integration parity asserts, per the review's deliberate scope (config factory + parity assert only, no test generation). test_qwen3_adapter migrated as the demonstration; new adapter tests should start from these. Added the missing tests/unit package __init__.py files so the helpers are importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DRY: _wire_rotary_for_testing() — 34 setup_component_testing bodies collapse D2 from the sweep code review. The canonical body (force eager, walk blocks, set the shared rotary on attention bridges + the template) varies on exactly three axes, so the helper takes lm_attr, hybrid, and eager (None \| 'config' \| 'layers'). Each file's current eager behavior is preserved exactly: the nine files that deliberately never touch _attn_implementation pass eager=None, the six config-only files pass eager='config' — a single-default helper would have silently changed their HF reference numerics. gemma3 (q/k-norm native-autograd) and falcon (ALiBi skip, transformer prefix) keep small extras around the helper call; openelm's pure no-op override is deleted (base is already a no-op). Twelve genuinely bespoke files stay untouched: per-layer rotary sources (baichuan, internlm2, exaone), eager-only gemma3n/gemma4/gemma4_text, nested multimodal lms (llava, idefics3, qwen2_audio, gemma3_multimodal, qwen3_5_multimodal), t5gemma's dual encoder/decoder rotary, and neox (which omits the template set the helper would add). The helper's missing-rotary early return converts a would-be AttributeError into a no-op — intentional (matches granite/falcon_h1), noted here since it makes a missing rotary quieter for future models. Spot re-verified P1 across the variant rows: gemma-2-2b-RMU (defaults), Qwen2.5-1.5B (eager=None), OLMo-2-1B (defaults), granite-4.0-tiny (hybrid) — all components equivalent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add DreamModel architecture adapter (diffusion LM) HKU-NLP Dream 7B (audit Tier-1 #5; also covers Apple DiffuCoder): discrete-diffusion text model initialized from Qwen2.5, so the module tree is exactly Qwen2 — but attention is fully bidirectional and generation is iterative denoising (diffusion_generate), so attention delegates to HF (the bridge reimplementation assumes causal masking), applicable_phases=[1,2,3], supports_generation=False (bd3lm precedent). Two transformers-v5 shims in prepare_loading, both measured against the live remote code: v5 dropped ROPE_INIT_FUNCTIONS['default'] (re-registered with exact v4 semantics) and v5 passes user_set_attributes to GenerationConfig.validate(), which the remote no-op override doesn't accept (replaced with a kwargs-tolerant no-op). Dream registers under plain AutoModel — new BASE_AUTOMODEL_ARCHITECTURES set in the loader. Remote-code prefixes: Dream-org/, apple/DiffuCoder. Verified: Dream-org/Dream-v0-Instruct-7B P1=100 P2=100 P3=100 (fp32 CPU, full applicable set). Registry rows added for Dream Base/Instruct and the three DiffuCoder variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add NanoChatForCausalLM architecture adapter Karpathy's nanochat (audit Tier-1 #6; native in transformers v5): a Llama-style decoder with three deliberate simplifications — weightless RMSNorms everywhere (no scale parameter, so supports_fold_ln=False: there is nothing to fold and the native-autograd norm path needs no weight), an ungated fc1 -> relu^2 -> fc2 MLP, and tanh logit soft-capping (reuses the gemma2 output_logits_soft_cap plumbing). Attention is MHA with full-width q/k norms applied AFTER rope — the reverse of the bridge reimplementation's order — so attention delegates to HF with q/k/v/o and both norms wrapped for hooks. Verified: nanochat-students/nanochat-d20 P1=100 P2=100 P3=89.5 P4=93.3 (fp32 CPU, full phases). Registry rows for the students' d20 and the dnakov mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add FlexOlmoForCausalLM architecture adapter AllenAI FlexOlmo (audit Tier-1 #7; NeurIPS 2025 federated expert merging with inference-time data opt-out): structurally the exact union of two supported OLMo variants — OLMo-2's post-norm blocks and full-width q/k norms with OLMoE's batched-parameter sparse MoE. The adapter is a seam override: olmo2 gains _build_mlp_bridge() (the qwen3 → qwen3-vl-moe pattern) and FlexOlmo swaps in a delegated MoEBridge whose router — a raw-parameter module, not nn.Linear — wraps as a plain GeneralizedComponent for hooks. Every published checkpoint is 2x7B+ (over the local fp32 verification ceiling), so parity is proven the way the audit planned: a seeded tiny FlexOlmoForCausalLM from config in integration tests (forward parity vs fresh HF < 1e-4, MoE/gate hooks captured). The seven allenai checkpoints are registered unverified for big hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add AudioFlamingo3 + MusicFlamingo architecture adapters NVIDIA Audio Flamingo 3 (audit Tier-1 #8) and its Music Flamingo / Audio Flamingo Next successor: a fine-tuned Whisper encoder at audio_tower, a projector, and a full Qwen2ForCausalLM at language_model — byte-identical module paths to Qwen2-Audio, so both adapters are pure subclasses (MusicFlamingo's temporal rotary conditioning rides along in the delegated tower). Added to AUDIO_TEXT_ARCHITECTURES (both load via AutoModelForSeq2SeqLM, matching their auto-map registration). Verified: nvidia/audio-flamingo-3-hf P1=100 P2=100 P3=100 P4=91.9 (fp32 CPU, 8.3B). Six nvidia checkpoints registered across the two archs, including music-flamingo-2601-hf (190k monthly downloads). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add HyenaDNAForCausalLM architecture adapter (attention-free) HazyResearch HyenaDNA (audit Tier-2 #12): genomic LMs on the Hyena long-convolution operator — the bridge's first attention-free non-SSM architecture. The mixer delegates wholesale (implicit modulated filters have no attention-shaped reconstruction) with in_proj/out_proj hookable; blocks are otherwise pre-LN transformer-shaped under hyena.backbone. _HyenaBlockBridge disables the tuple-normalizing standalone-call heuristic (the backbone's minimal layer(hidden_states) call is indistinguishable from a standalone call and fed block 1 a tuple) and replaces the attention-flavored hook aliases with hook_mixer_in/out. Three general fixes surfaced by the first attention-free model: - run_with_cache injected output_attentions unconditionally; it now signature-checks the wrapped forward (HF natives accept it directly or via *kwargs; Hyena's remote code rejects it). - Generation benchmarks now honor adapter.supports_generation — the flag existed but nothing read it, so bd3lm/dream/hyenadna-style models failed P2 generation benchmarks they can never run. - Weight processing no longer requires a positional embedding when positional_embedding_type == 'none' (only rotary/alibi were exempt). Verified: hyenadna-tiny-1k and hyenadna-large-1m both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Six LongSafari -hf checkpoints registered; LongSafari/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> feat: add LLaDA2MoeModelLM architecture adapter (diffusion MoE) Ant Group LLaDA 2.x (audit Tier-2 #11; 144k monthly downloads on mini): masked block-diffusion LMs on a DeepSeek-V3-style MoE decoder. Fused query_key_value attention with full-width q/k layernorms delegates to HF (bidirectional, is_causal=False); a fused-attention bridge subclass replaces the dead per-projection hook aliases with hook_qkv. MoE follows the deepseek_v3 pattern: per-expert routed MLPs delegated behind an optional router + optional shared experts (dense on the first first_k_dense_replace layers). Reuses Dream's v4 rope-init shim; the Dream/bd3lm diffusion treatment applies (phases 1-3, no autoregressive generation, fold_ln off for the fused layout). The remote forward validates masks strictly — 4D block-diffusion form (batch, 1, seq, seq) only — noted in the adapter docstring and exercised in tests. MoEBridge's router-scores hook now tolerates packed payloads (LLaDA2 returns (router_logits, topk_idx)) by hooking the first tensor. Every published checkpoint is 16B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config (bridge-vs-HF exactly 0.0; router + shared-expert hooks captured). Five inclusionAI checkpoints registered for big hardware; inclusionAI/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add LagunaForCausalLM architecture adapter poolside Laguna coding MoE (audit Tier-2 #18): two first-of-kind mechanisms — heterogeneous per-layer attention head counts (num_attention_heads_per_layer) and per-head softplus output gating (g_proj) — over a FlexOlmo-style batched-expert MoE with always-on shared experts and per-layer dense/sparse selection (mlp_layer_types). Attention delegates to HF (the bridge reimplementation assumes one uniform head count; the softplus gate has no reconstruction) with q/k/v/o and the gate projection hookable. Per-layer head counts also rule out uniform Q/K/V reshape conversions, so none ship and LN folding is disabled. Published checkpoints are 33B+ (over the local fp32 ceiling): parity proven on a seeded tiny from the remote config with heterogeneous heads [4,8,4,8] and mixed dense/sparse layers exercised — bridge-vs-HF exactly 0.0; softplus-gate and router hooks captured. Three poolside checkpoints registered for big hardware; poolside/ added to remote-code prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add six Tier-2 architecture adapters (Emu3, ModernBertDecoder, Youtu, Jais2, Ministral3, VaultGemma) All six verified on real checkpoints (fp32 CPU); one commit because they share the registration files. Emu3ForConditionalGeneration — BAAI's unified next-token text+image model: Llama-shaped decoder at model.text_model fully reimplemented; the VQ tokenizer is deliberately unmapped (no forward — encode/decode only). Emu3-Chat-hf VERIFIED P1-P3=100, P4=94.3, P7=100 (image expands to ~4k VQ tokens through the bridge). ModernBertDecoderForCausalLM — JHU Ettin decoders: delegated sliding/global attention mix, fused-GLU Wi/Wo, embedding norm, BERT-style prediction head before the untied decoder unembed. Layer-0's Identity attn-norm disables fold_ln AND center_writing_weights (residual readers are not mean-invariant; measured on ettin-17m). ettin-17m and ettin-1b both VERIFIED P1-P3=100. YoutuForCausalLM — Tencent's laptop-scale dense-MLA model: pure deepseek_v2 subclass via a new _build_mlp_bridge seam so the all-dense MLPs map as real gated MLPs (P3 76.2 -> 90 once weight processing could see them). Youtu-LLM-2B-Base VERIFIED. Jais2ForCausalLM — G42/Inception Arabic-English family: pure Nemotron subclass. JAIS2-IT-0.3 VERIFIED; gated flagship rows registered. Ministral3ForCausalLM — pure Mistral subclass. The -2512 main repos ship FP8 that transformers' auto-conversion cannot load (registered status 2 with pointer); Ministral-3-3B-Instruct-2512-BF16 VERIFIED. VaultGemmaForCausalLM — the only DP-SGD-pretrained open LLM: Gemma 2 minus the post-norms (block rebuild on the gemma2 base). Compat mode's stored-weights forward diverges for this offset-RMS variant even with every weight step disabled (9.6 logit shift, zero state-dict changes) — P3 excluded with the bisection evidence in the adapter comment and the machinery follow-up recorded in the sweep report. vaultgemma-1b VERIFIED P1=100 P2=100 P4=94.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add GiddForDiffusionLM architecture adapter (structural; fp32 parity open) GIDD uniform-noise diffusion LM (audit Tier-2 #10): bidirectional softcap attention delegates wholesale (Flex/Vanilla variants, optional per-head QK norms via the optional kwarg), ScaledLinear projections wrap as hookable Linears, rotary lives as a model-level buffer (no module entry), ungated up/down MLP. Diffusion treatment: phases 1-3, no autoregressive generation, no folding into scaled projections. Two v5 load shims measured against the live remote code: the bd3lm-style all_tied_weights_keys attribute, and a guarded _init_weights (v5 walks it over containers; the remote assumes module.weight exists and would re-randomize real tensors — internlm2 pattern skips non-meta modules). Verification is honest-but-partial: P1 components 100%, P2=100, P3=100, but full-forward parity failed because the remote code is numerically unstable under fp32 CPU eager for input-dependent token combinations — fresh HF reproduces identical NaNs (weights clean, softcap bounded, bridge==fresh bit-for-bit; bf16 and other draws finite). Registry note records the measurements; FlexAttention GPU is the author-supported path. Five dvruette checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add RwkvForCausalLM architecture adapter (WKV linear-attention RNN) BlinkDL RWKV-4 (audit Tier-2 #13), the canonical linear-attention RNN with a Pythia-parallel Pile suite. Time-mix delegates wholesale (the WKV recurrence has no attention-shaped reconstruction) with key/value/ receptance/output hookable; channel-mix maps as an MLPBridge (in=key, out=value, plus the receptance gate) so the component harness sizes inputs by the true d_model->4d->d_model dims. A custom block bridge replaces the attention-flavored aliases with hook_time_mix_/ hook_channel_mix_; layer 0's extra pre_ln uses the optional kwarg. Two load-time decisions, both measured: the eval-time 2^(layer//k) weight rescaling stays ON for both bridge and reference (the forward divides hidden states to compensate — one-sided neutralization made layers 6+ differ by exactly 2x), and use_cache is forced OFF (per-layer in-place recurrent state writes version-bump tensors autograd needs under backward hooks; version counts matched layer counts exactly). Generation phases are excluded: only the bespoke state-kwarg recurrent path consumes the cache the bridge's loop doesn't speak. Verified: rwkv-4-169m-pile and rwkv-4-430m-pile both P1=100 P2=100 P3=100 (fp32 CPU; bridge-vs-HF forward parity exactly 0.0). Five Pile checkpoints registered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add JetMoE + SwitchTransformers architecture adapters (Tier-2 complete) JetMoeForCausalLM (audit Tier-2 #14) — the only open at-scale Mixture-of-Attention-heads model: per-expert Q/O live inside the delegated MoA with the attention router and shared fused KV hookable (a fused-KV alias set replaces the per-projection aliases); the MLP MoE router is hookable too. The 8B checkpoints predate the native port and declare JetMoEForCausalLM (capital E) — aliased in the factory and registry sync. New general mechanism: adapters can declare component_test_skip_suffixes for subcomponents whose isolated forward cannot run on synthesized probes (TopKGating's sort/scatter crashes on them; routers stay hookable at runtime) — extends the previously hardcoded modality-tower skip list. Verified: jetmoe-8b P1=100 P2=100 P3=94.4 P4=95.4. SwitchTransformersForConditionalGeneration (audit Tier-2 #15) — the foundational top-1 capacity-routed MoE and the registry's only encoder-decoder MoE. v5 modernized Switch blocks to a tensor-in/ tensor-out protocol, so T5BlockBridge's tuple-chain patched forward poisons the stack (second block's norm received a tuple) — blocks delegate wholesale via a plain BlockBridge with the tuple-normalizing heuristic disabled, T5-named sublayers hookable, decoder aliases mapped to self_attn, and the FF as a delegated MoEBridge with an optional router (dense even layers, sparse odd). t5 gains a _build_ff_bridge seam; the bin-only google repos skip v5's Hub-side safetensors auto-conversion. Verified: switch-base-8 P1=100 P2=100 (P4=70.2 is the stochastic text-quality metric on a 2022 0.6B span-corruption pretrainer, the m2m100/led precedent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: complete compat-mode QKV-bias conversions (glm/glm4/starcoder2) + relu2 Addresses the senior-review MEDIUM cluster, characterized empirically (actual enable_compatibility_mode calls, not code-reading): - glm.py (+ glm4.py by inheritance): compat mode CRASHED, not silently mis-shaped as the review estimated. GLM's flat GQA b_V hit the split-format branch in fold_value_biases that reshapes as [n_heads, d_head]. Register _qkvo_weight_conversions(include_biases=True) so K/V bias reshapes use the kv-head count (mirrors glm4v). Now exact: 1.91e-06 log-prob diff on the real tiny-random/glm-4 checkpoint; fold_value_biases isolates to 0.0. - starcoder2.py: silent mis-shape confirmed (0.0322 log-prob, above the 3e-2 tolerance); include_biases=True brings it to 4.77e-07 (exact). - openai_gpt was flagged too but needs no fix: empirically 0.0 diff, since JointQKVAttentionBridge handles the combined c_attn correctly. verify_models never caught these: logits_equivalence/loss_equivalence, the only benchmarks that call enable_compatibility_mode, are gated behind ht_available, and the sweep runs with --no-ht-reference. Review LOW item: resolve_activation_fn now maps relu2/relu_2/relu_squared to squared-ReLU instead of silently falling to silu (a footgun for future gated ReLU^2 models; NanoChat/BitNet MLPs delegate, so not on a live path today). Regression tests added for all three. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * small bug fixes * GLM4 MoE Lite bug fix based on existing standard adapter * Fixes to the mamba layer names for transformers v5.13 * Architecture verificaion sweep * Rope-on-NoPE added for exaone4 and mixtral block rename fixe * improving small issues in MoE architectures * key name adjustments * Fixed huggingface canonical layer names. fixed gemma4_text test, reduced code duplication * Dry improvements to the system * Additional updates that reduce code duplication * First set of verifications with new patches * Added generation options for diffusion * completed diffuse generation * improvements to model generation * Added Provisional status for when --no-hf-reference is used for verification * Fixed bug where marian logit bias was not properly applied in component testing * Fixed flaws in llama4 and marian * Final set of verifications * Initial testing and comment cleanup * Adapter comment cleanup * ci fixes * swap * Additional CI fixes and improvements * fix formatting * Bug fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_original_model_not_registered.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_output_logits_contract.py	Add Jacobian lens (J-lens): published-artifact loading, readout, native fitting, and interventions (#1507) * Add JacobianLens analysis tool: artifact loading, readout, fitting, interventions Implements the Jacobian lens (Gurnee et al., Transformer Circuits 2026) as a tools/analysis component working with both TransformerBridge (raw weights) and HookedTransformer (from_pretrained_no_processing): - load/save in the official anthropics/jacobian-lens artifact format (fp16 storage, fp32 in memory), including artifacts published on the HF Hub - per-layer readout through the model\x27s own final norm, unembed, and logit soft cap, with the logit lens as the J=I baseline in the same code path - native fitting reproducing the reference estimator (one-hot cotangents at all valid target positions, sum over targets / mean over sources / mean over prompts), plus n_prompts-weighted merge() for parallel fitting - interventions from the paper: steering, ablation, pseudoinverse coordinate swap in lens coordinates - loud guards refusing fold_ln/compatibility-mode models (the published lenses are fitted on raw HF activations) Unit tests pin the estimator to a closed-form linear toy model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add JacobianLens integration tests on gpt2 with published Hub lens Covers Hub artifact loading/validation, the final-row identity, the rank-collapse late-layer invariant, logit-lens divergence mid-stack, HookedTransformer/TransformerBridge raw-basis agreement, a directional steering smoke test, processed-model refusal, the exact fit==merge identity, and small-fit convergence toward the published lens (cosine 0.91 at L10 from 2 prompts). Fixes _unembed to respect the [batch, pos, d_model] component contract under runtime type checking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add Jacobian Lens demo notebook gemma-2-2b via raw TransformerBridge + the published Hub lens: J-lens vs logit-lens readout on a two-hop prompt with an unspoken intermediate (boot -> Italy -> euro), rank-trajectory plot for pinned tokens, the France->China pseudoinverse coordinate swap (top-1 flip on the language template at alpha=1; Beijing rank 968 -> 8 on the capital template), and J-lens steering. Outputs are baked; verified with nbval locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Index fit gradients on their own device (device_map robustness) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review findings: guards, bounds checks, steering scale, provenance wording - Detect center_unembed processing via the exactly-mean-centered W_U signature: from_pretrained(fold_ln=False) keeps normalization_type LN while still centering weights, which the fold_ln marker alone missed (lens vectors on such a model are silently wrong; readout ranks survive only because LN absorbs the centered component). - Bounds-check readout layers on the logit-lens path and positions after normalization (out-of-range values previously crashed with a raw KeyError or wrapped silently). - Steering: scale by the median per-position residual norm instead of the mean (attention-sink positions run orders of magnitude hot and made the effective strength prompt-length dependent), and attribute the norm-matched parameterization to the reference implementation experiments, not the paper (whose minimal form is h += alphav_t). - Move readout residuals to the unembedding device for sharded models; document merge() metadata handling; reword docstrings/error strings that tracked the reference implementation too closely. - Demo notebook: kernelspec metadata, honest alpha=2 overshoot phrasing (the paper reports alpha=2 recovering failures; gemma-2-2b overshoots), measured-GPU wording; re-executed, nbval clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Address Jacobian Lens review feedback --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2 个月前
test_per_head_hooks_component.py	Split Q/K/V inputs with per-head attention results (#1257) * Fixing issue with exposed hooks * verification * Fix typing and formating * Improved QKV split attention/block system * Resolving failures * fixed hook error	5 个月前
test_position_ids_injection_gate.py	fix(bridge): derive position_ids from attention_mask for left-padded input — mirror of dev-4.x 21993bf0 (#1610) Masked-out tokens silently shifted the absolute position of every real token after them (no error, wrong logits/loss; gpt2 loss 4.50 -> 11.15 with three left pads). Positions are now derived from the mask via utils.get_offset_position_ids, per row, only when a masked token precedes a real one; explicit position_ids still wins; cached decode slices the derived positions back to the passed tokens. _accepts_derived_position_ids gates injection off fixed-signature forwards, mRoPE models that own their position derivation, and mask-consuming positional embeddings (OPT). (reimplemented from commit 21993bf06feeae4f8fc8c3573f3cfdfd2799a629 for dev: the gate introspects self.original_model — dev has no _driver abstraction; the gate unit test's stand-ins adapted to the same shape) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	1 个月前
test_process_weights_reentry.py	Followup review (#1770) * Bug fix follow ups * MoE fix * CI fixes	14 天前
test_qkvo_weight_conversions.py	Reducing v3.x beta code redundancies (#1228) * First pass comment cleanup * Second pass comment cleanup * Reducing redundancies between generalized components, architecture adapters, and benchmarks. Testing new utility functions * Test updates, CI Prep * Patching adjustments * Add gpt2-xl to cache * Increase timeout * Different timeout extension * Add a patching test outside of the notebook to make sure we get full coverage * formatting	5 个月前
test_reset_hooks_registry.py	fix(bridge): reset_hooks clears registry hook points with HookedRootModule semantics — mirror of dev-4.x de7531ba + d92683d0 hunks (#1335, #1538) reset_hooks only walked GeneralizedComponent children, so hook points that exist solely in the hook registry (alias-registered and scanned points outside the component tree) silently kept their hooks across resets — the session-fixture leak class. The registry is now cleared directly, with HookedRootModule-parity parameters (direction/including_permanent/level) and clear_contexts/ remove_all_hook_fns/hook_points helpers. (extracted from mixed commits de7531ba and d92683d0; dev divergence from 4.x HEAD: the component walk is kept alongside the registry pass on full resets — 4.x asserts its registry is canonical, dev's is not verified to be, so both passes run. Guard tests are new: neither origin commit shipped one)	1 个月前
test_stack_block_params_hybrid.py	Fixing edge cases on heterogeneous configs (#1679)	1 个月前
test_state_dict_composition.py	fix(bridge): recursive state dict composition for nested bridges — mirror of dev-4.x 80d9f36b (#1661) Composing a bridge inside a parent nn.Module broke state_dict round trips three ways: attn._ln1_module lazily registered ln1's raw module as a child at first forward (parent state_dict keys changed after forward), and the joint QKV / gate-up components filtered their combined weights out of state_dict() but rejected their own output under strict load. ln1's execution reference now lives outside the ownership tree and is wired at construction; both joint components register load pre-hooks that restore the filtered keys for strict matching. (mirrored from commit 80d9f36bb63f92b4e2f0d6acdb7963069c30a49a: bridge.py hunks path-retargeted from transformer_bridge.py; block.py applied into dev's _maybe_wire_capture_hooks, which merged 4.x's _maybe_wire_pre_ln_capture. Note for the audit record: the component_setup.py and joint_gate_up_mlp.py hunks — marked not-needed-on-dev in the mirror audit — ARE needed; upstream's own tests fail on dev without them)	1 个月前
test_state_dict_round_trip.py	fix(bridge): make state_dict()/load_state_dict() true inverses — mirror of dev-4.x a6e00330 (#1598) load_state_dict silently dropped TL-format keys (the very format state_dict() emits): unknown keys passed through to the wrapped model, strict was quietly downgraded, and aliased parameters (e.g. GPT-2's q/k/v views into c_attn) were only partially written. _tl_key_to_actual_keys inverts the state_dict() renaming with every alias; load accepts TL, raw, and stripped key formats; missing_keys treats an alias group as satisfied when any alias was written; strict raises with proper missing/unexpected lists. (path-retargeted from commit a6e003309760a34a20b16ab6b8ff8e04adcf6b6f: transformer_bridge.py hunks applied to bridge.py; no semantic changes)	1 个月前
test_t5_block_parity.py	Additional TransformerLens Bugs (#1681) * Fixing edge cases on heterogeneous configs * Additional transformer bridge bug fixes * T5 demo update * scope adjustment	1 个月前
test_tokenizer_reassignment.py	fix(bridge): make tokenizer assignment re-run wiring logic — mirror of dev-4.x cd89db7f (#1569) Reassigning bridge.tokenizer was a plain attribute write: d_vocab, BOS/EOS detection and padding setup all kept the old tokenizer's values, silently desyncing cfg from the tokenizer actually in use. tokenizer is now a property whose setter re-runs the wiring on reassignment and infers d_vocab on first assignment exactly as __init__ did. (reimplemented from commit cd89db7f0b9f5eb4784edd318102626f7f56b656 for dev: 4.x's configure_tokenizer does not exist here — the setter inlines dev's own setup_tokenizer + detect_tokenizer_bos_eos, the same pair dev's boot path runs; the test's BridgeCore/driver scaffolding became a bare TransformerBridge following test_generation_benchmark_mechanics's construction pattern)	1 个月前