| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(routing): defer plan quota commit until upstream accepts request Plan quota was charged at handler entry (check_plan_or_default_limits), before flow control queuing and before the request reached the upstream provider. When a client disconnected mid-queue or the provider call failed, the quota was already consumed and the rollback path for flow control timeouts was missing — leading to quota burn with no upstream service and no visible log (FlowControlQueueTimeout is dedup'd). Split check_and_record into peek_only (read-only) + commit_record (write). check_plan_limits now returns a PlanCharge that is committed only after provider.chat_stream/chat returns Ok. Dropping PlanCharge without commit consumes zero quota; the ConcurrencyGuard releases via its own Drop. This removes the entire rollback machinery (PlanRollback / rollback_plan_quota / rollback_plan_windows / unrecord_window) since the "charged but failed" window no longer exists. Principle 2 (client disconnect cleanup) is already handled by the existing AcquireCleanup::drop in flow control — added tests to verify the future-drop path actually removes queued requests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: liqiang <liqiang64@huawei.com> | 1 个月前 | |
feat: import BooMGateway v1.0 — Rust LLM API Gateway Intelligence Boom Gateway: high-performance LLM API gateway with litellm key compatibility. Core modules: boom-core — core traits and shared types boom-auth — SHA-256 key auth, litellm table compatibility boom-config — YAML config parsing + env var expansion + hot reload boom-provider — 20+ providers (OpenAI/Anthropic/Bedrock/Azure/Gemini/DeepSeek/vLLM/etc.) boom-routing — DeploymentStore + KV-cache affinity scheduling + AliasStore boom-flowcontrol — per-deployment queue/concurrency/context-length flow control, VIP priority queuing boom-limiter — sliding-window RPM rate limiting + PlanStore + time-based schedule overrides boom-audit — request audit log (LoggedStream Drop-trait records real duration) boom-promptlog — prompt logging (team/key directory layout, gzip auto-compression) boom-dashboard — Web UI + REST API + JWT auth (AdminCommand channel decoupling) boom-main — route assembly, state management, hot reload, background tasks Key features: - KV-cache affinity routing: context-aware affinity scheduling with real-time hotspot detection and queue rebalancing, maximizing KV affinity while maintaining cluster load balance - Fine-grained traffic control: per-deployment concurrency/queue/context limits, per-key RPM + concurrency throttling - SLO guarantees: auto-disable on consecutive failures, load-aware least-loaded selection, VIP priority queue - Per-key routing control: model whitelist, public_models, team-level access, hidden aliases - Multi-provider load balancing: weighted routing, priority-based fallback, swappable round-robin/affinity policies - Zero-downtime hot reload: SIGHUP/API trigger, ArcSwap atomic swap, runtime counters preserved across reloads | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 3 个月前 |