master

SShun_Yifeat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)

ff9c29f0创建于 6月11日历史提交

文件	最后提交记录	最后更新时间
kvc-aware-design.md	feat: add KV-cache aware routing (KVC-Aware) with token-prefix Trie Implement KV-cache aware request routing that directs requests to the vLLM worker holding the most relevant cached KV prefix, reducing recomputation and TTFT. Architecture: - ZMQ PUB/SUB subscriber connects to vLLM workers, parses msgpack KV events (BlockStored/BlockRemoved/AllBlocksCleared) - Token-prefix Trie index stores per-worker cached KV blocks keyed by raw token_ids, compatible with any vLLM configuration - HuggingFace chat_template rendering via minijinja with Python Jinja2 compatibility preprocessing (split, strip, items, etc.) - Scoring: combined_score = cache_weight × hit_ratio + tier_weight × tier_score + load_weight × load_score - Falls back to lowest-load selection when no KV prefix match found New crate: boom-kvindex (ZMQ subscriber, Trie index, tokenization) New routing policy: KVC-Aware (boom-routing/policy/kvc_aware.rs) Config: router_settings.kvc_aware (enabled, block_size, weights, tokenizer_dir, zmq_endpoints, zmq_topic_prefix) Debug endpoint: GET /internal/kv-index Docs: docs/kvc-aware-design.md Identity alignment required: - vLLM --served-model-name = gateway model_name - vLLM ZMQ worker_id = gateway model_info.id - vLLM block_size = gateway kvc_aware.block_size Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Jianhai Luan <luanjianhai@163.com>	1 个月前
kvc-aware-event-reporting-research.md	feat: add KV-cache aware routing (KVC-Aware) with token-prefix Trie Implement KV-cache aware request routing that directs requests to the vLLM worker holding the most relevant cached KV prefix, reducing recomputation and TTFT. Architecture: - ZMQ PUB/SUB subscriber connects to vLLM workers, parses msgpack KV events (BlockStored/BlockRemoved/AllBlocksCleared) - Token-prefix Trie index stores per-worker cached KV blocks keyed by raw token_ids, compatible with any vLLM configuration - HuggingFace chat_template rendering via minijinja with Python Jinja2 compatibility preprocessing (split, strip, items, etc.) - Scoring: combined_score = cache_weight × hit_ratio + tier_weight × tier_score + load_weight × load_score - Falls back to lowest-load selection when no KV prefix match found New crate: boom-kvindex (ZMQ subscriber, Trie index, tokenization) New routing policy: KVC-Aware (boom-routing/policy/kvc_aware.rs) Config: router_settings.kvc_aware (enabled, block_size, weights, tokenizer_dir, zmq_endpoints, zmq_topic_prefix) Debug endpoint: GET /internal/kv-index Docs: docs/kvc-aware-design.md Identity alignment required: - vLLM --served-model-name = gateway model_name - vLLM ZMQ worker_id = gateway model_info.id - vLLM block_size = gateway kvc_aware.block_size Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Jianhai Luan <luanjianhai@163.com>	1 个月前
vip-header-forwarding-design.md	feat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)	1 个月前