| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chat: refactor handling supports_string_content / supports_typed_content (#27130) * better supports_string_content cap detect * test: add "skip" * messages_inp_normalizer | 22 天前 | |
build : fix xcframework + cmake clean-up (#27304) * xcframework : fix build * mtmd : remove unused include path * vendor : use vendor::hash alias target in cmake CMake reserves "::" in target names for imported/alias targets, so the real target keeps the name vendor-hash and a vendor::hash ALIAS target is added. Consumers (mtmd, llama-gguf-hash) now link against the namespaced alias. Assisted-by: pi:llama.cpp/Qwen3.8-27B * vendor : add cmake targets for all vendored libs with vendor:: aliases Add INTERFACE targets for the header-only vendor libs (miniaudio, nlohmann, sheredom, stb) and ALIAS targets named vendor::<lib> for all of them, including cpp-httplib and hash. Each exposes the vendor/ root so includes are namespaced, e.g. <nlohmann/json.hpp>. Consolidate the per-lib add_subdirectory calls into a single add_subdirectory(vendor), keeping the cpp-httplib gate on LLAMA_BUILD_COMMON. Consumers (llama-common, mtmd) now link the aliases instead of relying on raw vendor/ include paths. hash: consumers now include via "hash/hash.h"; the vendor/hash dir is kept as a PRIVATE include so the synced upstream sources compile unmodified. Assisted-by: pi:llama.cpp/Qwen3.8-27B * readme : use foo/bar names in acknowledgements Assisted-by: pi:llama.cpp/Qwen3.8-27B * ocd : fix valign | 21 天前 | |
ui: move get_datetime tool to frontend (#27255) * ui: move get_datetime tool to frontend * clarify docs * server: drop the now unused ctime include strftime() and gmtime_r() were the only users, both went away with the get_datetime tool. Also make the renderer's catch inert: the browser executor always emits JSON, so a non-JSON result is no longer a date to display. --------- Co-authored-by: Pascal <admin@serveurperso.com> | 21 天前 | |
mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254) * convert text model * main model load ok * convert encoder ok * speaker encoder loading ok * speaker enc graph * adapt vocab for backbone (with some tricks) * add suppress_tokens * poc new mtmd gen api * convert code_predictor to gguf * load gen_code model ok * add clip_encode * wire up * code gen cgraph init version Co-authored-by: Pascal <admin@serveurperso.com> * code2wav convert to gguf * code2wav graph ok * wire up in/out * (wip) subgraph * wire up * wip, correct code2wav * demo (to be removed) * code2wav preserve kv between calls * demo voice clone * llama: add llama_model_get_tok_embd * mtmd_helper_gen_audio API * fix clamp cold prefix Co-authored-by: Pascal <admin@serveurperso.com> * fuse snake op Co-authored-by: Pascal <admin@serveurperso.com> * demo: use proper sampling * update dev docs * polymorphism helper * revamp llama-tts binary * update docs * fix compile * fix lint * nits * add guide + docs * more timings info * clean up code comments * security fixes * update docs * use ggml_build_forward_select, clean up comments * fix ci * use ISO 639-1 language code * rename CODE2WAV --> GEN_WAV, update docs * clean up * clean up tts.cpp * add seq_id * add step_prompt() * mtmd_helper_model_can_chat * clean up comments --------- Co-authored-by: Pascal <admin@serveurperso.com> | 1 个月前 | |
llava : expose as a shared library for downstream projects (#3613) * wip llava python bindings compatibility * add external llava API * add base64 in-prompt image support * wip refactor image loading * refactor image load out of llava init * cleanup * further cleanup; move llava-cli into its own file and rename * move base64.hpp into common/ * collapse clip and llava libraries * move llava into its own subdir * wip * fix bug where base64 string was not removed from the prompt * get libllava to output in the right place * expose llava methods in libllama.dylib * cleanup memory usage around clip_image_* * cleanup and refactor *again* * update headerdoc * build with cmake, not tested (WIP) * Editorconfig * Editorconfig * Build with make * Build with make * Fix cyclical depts on Windows * attempt to fix build on Windows * attempt to fix build on Windows * Upd TODOs * attempt to fix build on Windows+CUDA * Revert changes in cmake * Fix according to review comments * Support building as a shared library * address review comments --------- Co-authored-by: M. Yusuf Sarıgöz <yusufsarigoz@gmail.com> Co-authored-by: Jared Van Bortel <jared@nomic.ai> | 2 年前 | |
cmake : introduce semantic versioning (#26839) * cmake : introduce semantic versioning (wip) This commit introduces semantic versioning to llama.cpp. * squash! cmake : introduce semantic versioning (wip) * cmake : update test-cmake README notes [no ci] * include libmtmd in output so show its semversioned * ci : add make-release workflow * ci : fix build number check in build-cmake-pkg.yml * examples : remove trailing whitespace * ci : abort if upstream ggml version does not exist * ci : extract step contents into scripts * ci : add GGML_NATIVE=OFF to ubuntu job * examples : remove CI build information from test-cmake [no ci] This commit removes the nightly/release information that I added previously to keep this focused only on using building and installing llama.cpp with cmake and being able to quickly verify changes or troubleshoot issues. * ci : merge scripts into single script * remove -dev-build_number support This commit removes the incremental build number (versioning) support that I added. This was incorrect and we should only use the semver for the version. Releases will be tag a nightly build and package maintainers/managers that build from source can use the tag and it is therefor important that the correct version is reported. So a nightly-build will report the semver without the build number. The build number and commit as availble via cmake and test-cmake has been updated to include an example of using them: console $ ./build.sh [test-cmake] version: 0.1.0, build: 10360 (08c69e381) ... Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969 * docs: add initial release.md documentation * cmake : clean-up and add LLAMA_BUILD_IS_DEV option * ci : remove version input from make-release job * ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145 * docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci] * ci : add TODO to winget workflow [no ci] --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 26 天前 | |
cmake : introduce semantic versioning (#26839) * cmake : introduce semantic versioning (wip) This commit introduces semantic versioning to llama.cpp. * squash! cmake : introduce semantic versioning (wip) * cmake : update test-cmake README notes [no ci] * include libmtmd in output so show its semversioned * ci : add make-release workflow * ci : fix build number check in build-cmake-pkg.yml * examples : remove trailing whitespace * ci : abort if upstream ggml version does not exist * ci : extract step contents into scripts * ci : add GGML_NATIVE=OFF to ubuntu job * examples : remove CI build information from test-cmake [no ci] This commit removes the nightly/release information that I added previously to keep this focused only on using building and installing llama.cpp with cmake and being able to quickly verify changes or troubleshoot issues. * ci : merge scripts into single script * remove -dev-build_number support This commit removes the incremental build number (versioning) support that I added. This was incorrect and we should only use the semver for the version. Releases will be tag a nightly build and package maintainers/managers that build from source can use the tag and it is therefor important that the correct version is reported. So a nightly-build will report the semver without the build number. The build number and commit as availble via cmake and test-cmake has been updated to include an example of using them: console $ ./build.sh [test-cmake] version: 0.1.0, build: 10360 (08c69e381) ... Refs: https://github.com/ggml-org/llama.cpp/pull/26839#discussion_r3755836969 * docs: add initial release.md documentation * cmake : clean-up and add LLAMA_BUILD_IS_DEV option * ci : remove version input from make-release job * ci : add LLAMA_BUILD_IS_DEV=OFF to build-cmake-pkg.yml Refs: https://github.com/danbev/llama.cpp/actions/runs/31576801921/job/94050639145 * docs : update release notes with LLAMA_BUILD_IS_DEV info [no ci] * ci : add TODO to winget workflow [no ci] --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 26 天前 | |
Add support for Laguna XS.2 & M.1 (#25165) | 1 个月前 | |
server: fix checkpoints creation (#22929) * common : add common_chat_split_by_role * cont : fix spans to reach end of message * server: fix checkpoints creation - extract message_spans from chat templates - find the prompt token position before the latest user message - split prompt batching at that position - create a context checkpoint before the latest user input - avoid periodic mid-prompt checkpoints when that position is known - handle multimodal prompts when mapping text/template positions to server prompt tokens - add --checkpoint-min-step to control minimum spacing between checkpoints * cont : clean-up * Support autoparser detection for message barriers * server: fix message span delimiter and update docs --------- Co-authored-by: Alde Rojas <hello@alde.dev> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> | 3 个月前 | |
chat : avoid including json in chat.h (#21306) | 5 个月前 | |
Add support for Laguna XS.2 & M.1 (#25165) | 1 个月前 | |
model : BailingMoE3 Support (#26608) * Adding support for bailingmoe3 * Adds speculative decoding support * Make BailingMoE3 safe gate metadata optional * bailingmoe3: apply trained SwiGLU clamps * common: fix Bailing V3 tool argument parsing * llama-model-saver, instantiate float vector metadata writer * bailingmoe3: support Q-LoRA (Ling-3.0-tiny) Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256 and routes Q through a LoRA bottleneck instead: q_a_proj -> q_a_layernorm -> q_b_proj Conversion therefore failed with: ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight' Add the missing path, mirroring the existing deepseek2 MLA implementation: * constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3 * tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and q_a_layernorm * conversion - emit attention.q_lora_rank when the config has it * bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q through the bottleneck when q_lora_rank > 0 Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no q_lora_rank, the converter only emits the key when present, hparams.n_lora_q defaults to 0, and get_key(..., required=false) leaves the target untouched when the key is absent - so flash keeps taking the existing direct-Q branch. The LoRA path produces the same shape as the direct projection, so the nope/rope split, RoPE application and wk_b absorption downstream are unchanged. * small mtp change * bailingmoe3: support separate MTP GGUF and Q-LoRA MTP * gguf: remove duplicate add_kda_gate_lower_bound definition --------- Co-authored-by: bloomer <bloomer@booper.brushtail.me> Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com> | 22 天前 | |
chat : fix LFM2 tool call arg name prefix ambiguity (#26960) Assisted-by: Claude Opus 5 | 25 天前 | |
chat : add qwen3 specialized parser (#26252) * Add tagged thinking tool parser * chat : refactor and add permute helper * cont : add support for <tool_call> omission * cont : update tool delimiters * cont : add comment for qwen3-coder * cont : fix trigger pattern for <function --------- Co-authored-by: Bart de Boer <bart.deboer@gmail.com> | 1 个月前 | |
chat: refactor handling supports_string_content / supports_typed_content (#27130) * better supports_string_content cap detect * test: add "skip" * messages_inp_normalizer | 22 天前 | |
common/chat: add specialized minimax m3 parser (#26210) | 1 个月前 | |
common: share thread pools when n_threads differ (#27138) | 20 天前 | |
common: apply CPU parameters across tools (#27026) | 25 天前 | |
cli: fix stripping of \n in multiline input (#21485) * llama-cli: fix stripping of \n in multiline input * Change & string to string_view * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Fix EditorConfig linter error --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> | 4 个月前 | |
cli : add command and file auto-completion (#19985) | 6 个月前 | |
common: fix missing exports in llama-common (#22340) * common: refactor common/debug to move abort_on_nan into base_callback_data Passing bool abort_on_nan as template parameter for common_debug_cb_eval is unnecessary and creates an issue with LTO. It should just be a member of the base_callback_data instead. * cont : cleanup * common : use pimpl in debug.h to reduce header dependencies Move common_debug_cb_user_data's data members (std::regex, std::vector<uint8_t>) into a private impl struct in debug.cpp. This removes the includes of common.h and <regex> from debug.h, reducing transitive dependencies for any translation unit that includes the header. Assisted-by: llama.cpp:local pi --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 4 个月前 | |
common: fix missing exports in llama-common (#22340) * common: refactor common/debug to move abort_on_nan into base_callback_data Passing bool abort_on_nan as template parameter for common_debug_cb_eval is unnecessary and creates an issue with LTO. It should just be a member of the base_callback_data instead. * cont : cleanup * common : use pimpl in debug.h to reduce header dependencies Move common_debug_cb_user_data's data members (std::regex, std::vector<uint8_t>) into a private impl struct in debug.cpp. This removes the includes of common.h and <regex> from debug.h, reducing transitive dependencies for any translation unit that includes the header. Assisted-by: llama.cpp:local pi --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 4 个月前 | |
common: support the DSpark sidecar resolution (#26458) The dspark- files resolve like the other speculative sidecars: the -hfd tag applies to them, a requested sidecar resolves without a full model at the tag, and an explicit -md selection disables the discovery. When no type is requested, dspark outranks dflash in the auto-selection since its sidecar carries the extra Markov head. | 1 个月前 | |
common: support the DSpark sidecar resolution (#26458) The dspark- files resolve like the other speculative sidecars: the -hfd tag applies to them, a requested sidecar resolves without a full model at the tag, and an explicit -md selection disables the discovery. When no type is requested, dspark outranks dflash in the auto-selection since its sidecar carries the extra Markov head. | 1 个月前 | |
fit: Fix memory allocation for MTP layers (#26605) | 1 个月前 | |
fit : wrap llama_device_memory_data (#24522) | 2 个月前 | |
server: (router) add model management API (#23976) * wip * server: (router) add SSE realtime updates API * nits * wip * add download API * add download api * update docs * add delete endpoint * fix std::terminate * fix crash * fix 2 * add tests * nits | 2 个月前 | |
server: (router) add model management API (#23976) * wip * server: (router) add SSE realtime updates API * nits * wip * add download API * add download api * update docs * add delete endpoint * fix std::terminate * fix crash * fix 2 * add tests * nits | 2 个月前 | |
cli : move to HTTP-based implementation (#24948) * cli: move to HTTP-based implementation * wip * working * remote server ok * cli support router mode Co-authored-by: Piotr Wilkin <ilintar@gmail.com> * case: router with only one model * Apply suggestions from code review Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * remove outdated comment * use destructor instead * add ftype * cli-view --> cli-ui * pimpl * no more json in header * nits fixes * also show model aliases --------- Co-authored-by: Piotr Wilkin <ilintar@gmail.com> Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> | 1 个月前 | |
fix: check gguf array type before reading (#27075) * fix: check gguf array type before reading * update skill | 23 天前 | |
Move duplicated imatrix code into single common imatrix-loader.cpp (#22445) * Deduplicate imatrix loading code * Add back LLAMA_TRACE, early exit on quantize missing metadata | 3 个月前 | |
common/json-schema-to-grammar : align spacing rules with parsers (#24835) | 2 个月前 | |
common : add nemotron 3 parsing (#18077) * common : expose json-schema functionality to extract type info * common : fix peg parser negation during needs_more_input * common : add some defensive measures in constructed peg parser * common : add nemotron nano 3 support * common : add nemotron nano 3 tests * remove debug line | 8 个月前 | |
llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes * Rebase and address review comments * Address review comments * Address review comments * Update src/llama-sampler.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 28 天前 | |
common: update logging to enforce max_capacity and optimize queue resizing (#24490) * common: update logging to enforce max_capacity and optimize queue resizing logic * common/log: remove queue expansion logic | 2 个月前 | |
logs : reduce (#23021) * logs : reduce * args : fix envs * server : fix build * common : print verbosity level at start * server : clean-up logs * server : print prompt processing timings + sampling params * minor : whitespaces | 3 个月前 | |
spec : add self‑speculative decoding (no draft model required) + refactor (#18471) * server: introduce self-speculative decoding * server: moved self-call into speculative.cpp * can_speculate() includes self-speculation Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * server: can_speculate() tests self-spec * server: replace can_speculate() with slot.can_speculate() Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * common: use %zu format specifier for size_t in logging Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * server: can_speculate() requires a task instance * common: ngram map, config self-speculative decoding * common: add enum common_speculative_type * common: add vector of speculative states * common: add option --spec-draftless * server: cleanup (remove slot.batch_spec, rename) * common: moved self-spec impl to ngram-map * common: cleanup (use common_speculative_state_draft) * spec : refactor * cont : naming * spec: remove --spec-config * doc: (draftless) speculative decoding * common: print performance in spec decoding * minor : cleanup * common : better names * minor : cleanup + fix build * minor: comments * CODEOWNERS: add common/ngram-map.* (#18471) * common : rename speculative.draftless_type -> speculative.type * ngram-map : fix uninitialized values * ngram-map : take into account the input can become shorter * ngram-map : revert len check for now * arg : change --spec-draftless -> --spec-type * spec : add common_speculative_state::accept() * spec : refactor + add common_speculative_begin() * spec : fix begin() call with mtmd * spec : additional refactor + remove common_speculative_params --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> | 7 个月前 | |
spec : add self‑speculative decoding (no draft model required) + refactor (#18471) * server: introduce self-speculative decoding * server: moved self-call into speculative.cpp * can_speculate() includes self-speculation Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * server: can_speculate() tests self-spec * server: replace can_speculate() with slot.can_speculate() Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * common: use %zu format specifier for size_t in logging Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * server: can_speculate() requires a task instance * common: ngram map, config self-speculative decoding * common: add enum common_speculative_type * common: add vector of speculative states * common: add option --spec-draftless * server: cleanup (remove slot.batch_spec, rename) * common: moved self-spec impl to ngram-map * common: cleanup (use common_speculative_state_draft) * spec : refactor * cont : naming * spec: remove --spec-config * doc: (draftless) speculative decoding * common: print performance in spec decoding * minor : cleanup * common : better names * minor : cleanup + fix build * minor: comments * CODEOWNERS: add common/ngram-map.* (#18471) * common : rename speculative.draftless_type -> speculative.type * ngram-map : fix uninitialized values * ngram-map : take into account the input can become shorter * ngram-map : revert len check for now * arg : change --spec-draftless -> --spec-type * spec : add common_speculative_state::accept() * spec : refactor + add common_speculative_begin() * spec : fix begin() call with mtmd * spec : additional refactor + remove common_speculative_params --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> | 7 个月前 | |
speculative : fix out-of-bounds read in ngram-map on prompt shrink (#23936) * speculative : fix out-of-bounds read in ngram-map on prompt shrink * speculative : fix ngram-map cleanup cutoff after prompt shrink | 2 个月前 | |
fix: correct misspellings in code comments (#21217) - emdeddings → embeddings (gemma3.cpp, gemma3n-iswa.cpp, gemma-embedding.cpp) - imlpemented → implemented (llama-adapter.cpp) - interere → interfere (llama-graph.cpp) - overridde → overridden (chat.cpp) - stastistics → statistics (ngram-map.h) - layed → laid (llama-kv-cache.h) - worster → worst (llama-context.cpp) - sequantial → sequential (llama-batch.h) | 5 个月前 | |
ngram-mod : Add missing include (#23857) [no release] Signed-off-by: Omid Azizi <oazizi@gimletlabs.ai> | 3 个月前 | |
ngram-mod : fix build [no ci] (#19216) | 7 个月前 | |
common/peg : suppress incomplete escape sequences (#26780) | 28 天前 | |
common/peg : implement ac parser for stricter grammar generation (#24869) * common/peg : implement ac parser * cont : extract functions * cont : tidy up * cont : remove a test * cont : move ac() def | 2 个月前 | |
common: support --models-dir loading MTP assistant models (#24431) * common: support --models-dir loading MTP assistant models * common: preset: check for MTP models with strict prefix * common: preset: Take advantage of PR #27005 * handle other draft types * drop eagle3 * clean up --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> | 23 天前 | |
common: add system-level config file (#26118) * common: Add CLI > ENV > models-presets > INI precedence 1. CLI flags have the highest precedence 2. ENV vars have the second-highest precedence 3. System and User configs have the lowest precedence - Linux/BSD/Mac - /etc/llama.cpp/config.ini < ${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini - Windows - %PROGRAMDATA%\llama.cpp\config.ini < %APPDATA%\llama.cpp\config.ini * fix UB * use common_get_env * ignore_unknown_keys * nits * add docs --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> | 26 天前 | |
llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes * Rebase and address review comments * Address review comments * Address review comments * Update src/llama-sampler.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 28 天前 | |
common : add support for multiple end sequences in the reasoning budget sampler (#25544) * common : extract trie/ac to a separate file * common : support multiple token sequences in the reasoning budget sampler * common/trie : return matched word index * common/trie : rename "word" to "pattern" * common/reasoning-budget : expose matched end sequence * common/sampling : replay end sequence when reasoning budget is done * cont : update to use multiple end sequences * cont : clean up | 1 个月前 | |
llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes * Rebase and address review comments * Address review comments * Address review comments * Update src/llama-sampler.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 28 天前 | |
llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes * Rebase and address review comments * Address review comments * Address review comments * Update src/llama-sampler.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> | 28 天前 | |
model: support speculators-format checkpoints for DSpark (#26275) * dspark: support speculators-format checkpoints (SpecForge exports) Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the speculators draft class into DSparkModel One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5 * dspark: address review comments - rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the sample_from_anchor read into the block_size block * dspark: fix flake8 continuation indent * clean up * dspark: key the sample_from_anchor default off the export format Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: drop t2d in filter_tensors Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: map model.lm_head instead of bypassing the dflash prefix Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable --------- Co-authored-by: desovo7 <942845546@qq.com> Co-authored-by: ruixiang63 <wangruixiang07@outlook.com> | 21 天前 | |
common : auto-detect spec type from draft GGUF metadata (#26814) * common : auto-detect spec type from draft GGUF metadata When -md loads a local draft model without --spec-type, the sidecar inference in common_models_handler_apply only checks HF repo sidecars and misses local files. The draft model loads into VRAM but speculative decoding never activates (types stays NONE). Read general.architecture from the draft GGUF header and map: dflash + markov_w1.weight tensor -> draft-dspark dflash without markov head -> draft-dflash Assisted-by: opencode * common : address review feedback on spec-type auto-detect PR - Fix comment spacing to match surrounding style (/* .x = */ not /*.x =*/) - Add LOG_INF when auto-detection fires so users can see why spec decoding enabled - Document single-file assumption for split-GGUF edge case Addresses bot review feedback on #26814. * common : move spec-type GGUF auto-detect into speculative module - add common_speculative_types_from_gguf() in speculative.cpp/.h - use gguf_context_ptr (RAII) from ggml-cpp.h - reduce comments to a single line per AGENTS.md style Addresses review feedback on #26814 * common : add doc note and join SPC_INF line in spec-type auto-detect Assisted-by: opencode | 25 天前 | |
common: add subproc.h wrapper, disabled on android/ios (#26102) * add common/subproc.h|cpp * add compile flag LLAMA_SUBPROCESS * disabled by default on android and ios * test-jinja: use common subproc * mtmd: disable video if subproc is not set * disable subproc on wasm * make is_created atomic * migrate server-mcp | 1 个月前 | |
common: add subproc.h wrapper, disabled on android/ios (#26102) * add common/subproc.h|cpp * add compile flag LLAMA_SUBPROCESS * disabled by default on android and ios * test-jinja: use common subproc * mtmd: disable video if subproc is not set * disable subproc on wasm * make is_created atomic * migrate server-mcp | 1 个月前 | |
common : add support for multiple end sequences in the reasoning budget sampler (#25544) * common : extract trie/ac to a separate file * common : support multiple token sequences in the reasoning budget sampler * common/trie : return matched word index * common/trie : rename "word" to "pattern" * common/reasoning-budget : expose matched end sequence * common/sampling : replay end sequence when reasoning budget is done * cont : update to use multiple end sequences * cont : clean up | 1 个月前 | |
common : add support for multiple end sequences in the reasoning budget sampler (#25544) * common : extract trie/ac to a separate file * common : support multiple token sequences in the reasoning budget sampler * common/trie : return matched word index * common/trie : rename "word" to "pattern" * common/reasoning-budget : expose matched end sequence * common/sampling : replay end sequence when reasoning budget is done * cont : update to use multiple end sequences * cont : clean up | 1 个月前 | |
common/parser: handle reasoning budget (#20297) * v1 * Finished! * Handlie cli * Reasoning sampler * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Less explosive terminology :) * Add utf-8 case and tests * common : migrate reasoning budget sampler to common * cont : clean up * cont : expose state and allow passing as initial state * cont : remove unused imports * cont : update state machine doc string --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Alde Rojas <hello@alde.dev> | 5 个月前 | |
common/parser: handle reasoning budget (#20297) * v1 * Finished! * Handlie cli * Reasoning sampler * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Less explosive terminology :) * Add utf-8 case and tests * common : migrate reasoning budget sampler to common * cont : clean up * cont : expose state and allow passing as initial state * cont : remove unused imports * cont : update state machine doc string --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Alde Rojas <hello@alde.dev> | 5 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 22 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 1 个月前 | ||
| 2 年前 | ||
| 26 天前 | ||
| 26 天前 | ||
| 1 个月前 | ||
| 3 个月前 | ||
| 5 个月前 | ||
| 1 个月前 | ||
| 22 天前 | ||
| 25 天前 | ||
| 1 个月前 | ||
| 22 天前 | ||
| 1 个月前 | ||
| 20 天前 | ||
| 25 天前 | ||
| 4 个月前 | ||
| 6 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 23 天前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 8 个月前 | ||
| 28 天前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 2 个月前 | ||
| 5 个月前 | ||
| 3 个月前 | ||
| 7 个月前 | ||
| 28 天前 | ||
| 2 个月前 | ||
| 23 天前 | ||
| 26 天前 | ||
| 28 天前 | ||
| 1 个月前 | ||
| 28 天前 | ||
| 28 天前 | ||
| 21 天前 | ||
| 25 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 5 个月前 | ||
| 5 个月前 |