master

ZzhengyangyangSpark2_5 tokenizer :auto hash

ee643aa6创建于 24 天前历史提交

文件	最后提交记录	最后更新时间
__init__.py	refactor: rename Spark3 model to Spark2_5	30 天前
afmoe.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
arctic.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
baichuan.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
bailingmoe.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
bailingmoe3.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
base.py	Spark2_5 tokenizer :auto hash	24 天前
bert.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
bitnet.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
bloom.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
chameleon.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
chatglm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
codeshell.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
cogvlm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
command_r.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
dbrx.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
deci.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
deepseek.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
dots1.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
dotsocr.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
dream.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
ernie.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
exaone.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
falcon.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
falcon_h1.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
gemma.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
glm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
gpt2.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
gpt_oss.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
gptneox.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
granite.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
grok.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
grovemoe.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
hunyuan.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
internlm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
internvl.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
jais.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
jamba.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
januspro.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
kimi_k3.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
kimi_linear.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
kimivl.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
laguna.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
lfm2.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
lighton_ocr.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
llada.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
llama.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
llama4.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
llava.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
maincoder.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
mamba.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
mellum.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
mimo.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
minicpm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
minimax.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
mistral.py	convert : fix conversion for Mistral-Medium-3.5-128B (#24268) Mistral explicitly sets moe and llama_4_scaling to null in params.json, breaking key in dict checks during conversion. Replace with dict.get(key) is not None where this matters. Fixes convert-hf-to-gguf.py --mistral-format Mistral-Medium-3.5-128B	3 个月前
mistral3.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
mpt.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
muse_glimmer.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
nanbeige.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
nemotron.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
olmo.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
openelm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
orion.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
pangu.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
phi.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
pixtral.py	Refactor: convert_hf_to_gguf.py (#17114) * move conversion code to a dedicated conversion directory and split the files akin to the src/models architecture --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	4 个月前
plamo.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
plm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
pockettts.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
qwen.py	model: support speculators-format checkpoints for DSpark (#26275) * dspark: support speculators-format checkpoints (SpecForge exports) Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the speculators draft class into DSparkModel One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5 * dspark: address review comments - rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the sample_from_anchor read into the block_size block * dspark: fix flake8 continuation indent * clean up * dspark: key the sample_from_anchor default off the export format Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: drop t2d in filter_tensors Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: map model.lm_head instead of bypassing the dflash prefix Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable --------- Co-authored-by: desovo7 <942845546@qq.com> Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>	1 个月前
qwen3tts.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
qwen3vl.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
qwenvl.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
refact.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
rwkv.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
sarashina2.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
smallthinker.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
smolvlm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
spark2_5.py	refactor: rename Spark3 model to Spark2_5	30 天前
stablelm.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
starcoder.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
step3.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
t5.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
talkie.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
ultravox.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
wavtokenizer.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
xverse.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前
youtuvl.py	convert: add @ModelBase.example (#27208) * convert: add @ModelBase.example * add docs * add more variants * BailingMoeV3ForCausalLM * rm pocket-tts	1 个月前