main

GGitHub[Feat] Implement OpenAI external provider (#902)

cbab5434创建于 4月21日历史提交

文件	最后提交记录	最后更新时间
disaggregated_prefill	[bugfix] deprecate disable log request (#885) * bugfix: deprecate disable log request Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Update helm/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	5 个月前
disaggregated_prefill_orchestrated	[Feat][Router] Add disaggregated prefill orchestrated routing (#777) * [Feat][Router] Add disaggregated prefill orchestrated routing Implements support for disaggregated prefill as outlined in the 2025 Q1 roadmap. This enables prefill/decode disaggregation with router-orchestrated KV cache transfer. Closes #26 Signed-off-by: Yahav <yahavb@amazon.com> * [CI/Build] Lower Python version requirement to 3.10 for Neuron SDK compatibility Signed-off-by: Yahav <yahavb@amazon.com> * [Feat][Router] Address PR review feedback for disaggregated prefill orchestrated routing - Remove dead code (handle_orchestrated_request method in routing_logic.py) - Fix prefill request to use max_tokens=1 per proposal spec - Use shared aiohttp client instead of creating new session per request - Fix streaming to yield chunks immediately (true streaming) - Remove redundant isinstance check for DisaggregatedPrefillOrchestratedRouter - Use router's _find_endpoints method to avoid code duplication Signed-off-by: Yahav <yahavb@amazon.com> * fix: use kv_transfer_params instead of disagg_prefill_resp - Add kv_transfer_params to prefill request to enable disaggregated mode - Extract kv_transfer_params from prefill response and forward to decode - Set remote_host to prefill endpoint for KV cache retrieval Signed-off-by: Yahav <yahavb@amazon.com> * docs: add example for disaggregated_prefill_orchestrated mode - Add README with usage instructions and configuration notes - Add sanitized Kubernetes manifests (router, prefill, decode) - Include example curl command and expected router logs Signed-off-by: Yahav <yahavb@amazon.com> * style: fix black formatting Signed-off-by: Yahav <yahavb@amazon.com> * style: fix markdownlint errors in README.md Signed-off-by: Yahav <yahavb@amazon.com> * style: fix markdownlint errors in proposal doc Signed-off-by: Yahav <yahavb@amazon.com> * docs: clean up DisaggregatedPrefillOrchestratedRouter docstring Signed-off-by: Yahav <yahavb@amazon.com> * feat: return 503 with distinct codes for prefill/decode unavailability - PREFILL_SERVICE_UNAVAILABLE: No prefill endpoints discovered - DECODE_SERVICE_UNAVAILABLE: No decode endpoints discovered This allows automated tests to distinguish transient startup issues from real bugs. Signed-off-by: Yahav <yahavb@amazon.com> * revert: restore requires-python = 3.12 Signed-off-by: Yahav <yahavb@amazon.com> * fix: replace angle bracket placeholders with uppercase format Angle brackets like <your-pvc-name> are interpreted as shell redirections by shellcheck, causing CI failures. Use uppercase format instead: YOUR-PVC-NAME, YOUR-MODEL-PATH, etc. Signed-off-by: Yahav <yahavb@amazon.com> * fix: remove trailing whitespace from YAML files Signed-off-by: Yahav <yahavb@amazon.com> --------- Signed-off-by: Yahav <yahavb@amazon.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com>	5 个月前
batch.jsonl	feat: OpenAI batch API part 1 (#52) * feat: OpenAI batch API part 1 Signed-off-by: Ce Gao <cegao@tensorchord.ai>	1 年前
external_providers_config.yaml	[Feat] Implement OpenAI external provider (#902) * feat: Implement openai provider Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> * test: Update tests Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> * test: Add unit test for external providers Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> * fix: Increase timeout Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> --------- Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com>	4 个月前
openai_api_client_batch.py	[Router] Support Batch API part 2 (#109) * feat: Support batch Signed-off-by: Ce Gao <cegao@tensorchord.ai> * fix: Add TODO Signed-off-by: Ce Gao <cegao@tensorchord.ai> --------- Signed-off-by: Ce Gao <cegao@tensorchord.ai>	1 年前