master

SShun_Yifeat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)

ff9c29f0创建于 6月11日历史提交

文件	最后提交记录	最后更新时间
.gitattributes	feat: import BooMGateway v1.0 — Rust LLM API Gateway Intelligence Boom Gateway: high-performance LLM API gateway with litellm key compatibility. Core modules: boom-core — core traits and shared types boom-auth — SHA-256 key auth, litellm table compatibility boom-config — YAML config parsing + env var expansion + hot reload boom-provider — 20+ providers (OpenAI/Anthropic/Bedrock/Azure/Gemini/DeepSeek/vLLM/etc.) boom-routing — DeploymentStore + KV-cache affinity scheduling + AliasStore boom-flowcontrol — per-deployment queue/concurrency/context-length flow control, VIP priority queuing boom-limiter — sliding-window RPM rate limiting + PlanStore + time-based schedule overrides boom-audit — request audit log (LoggedStream Drop-trait records real duration) boom-promptlog — prompt logging (team/key directory layout, gzip auto-compression) boom-dashboard — Web UI + REST API + JWT auth (AdminCommand channel decoupling) boom-main — route assembly, state management, hot reload, background tasks Key features: - KV-cache affinity routing: context-aware affinity scheduling with real-time hotspot detection and queue rebalancing, maximizing KV affinity while maintaining cluster load balance - Fine-grained traffic control: per-deployment concurrency/queue/context limits, per-key RPM + concurrency throttling - SLO guarantees: auto-disable on consecutive failures, load-aware least-loaded selection, VIP priority queue - Per-key routing control: model whitelist, public_models, team-level access, hidden aliases - Multi-provider load balancing: weighted routing, priority-based fallback, swappable round-robin/affinity policies - Zero-downtime hot reload: SIGHUP/API trigger, ArcSwap atomic swap, runtime counters preserved across reloads	2 个月前
01-background.png	feat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)	1 个月前
02-HTTP-Header-example.png	feat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)	1 个月前
03-priority-level-extend.png	feat: forward request priority to downstream scheduler via X-Gateway-Priority (off by default)	1 个月前