性能测试 run_perf.sh --runs 3 在 small 场景(--recent_turns 10 --compress_message_cnt 50)Run 1 Batch 1 卡住不退出。两个测试子进程从 00:44 启动后一直存活,状态 S (sleeping)、wchan=wait_woken,CPU 累计仅 18s/11s(10h+ 内几乎不占用),日志在 01:00 后再无输出,永久挂死。
run_perf.sh --runs 3
small
--recent_turns 10 --compress_message_cnt 50
S (sleeping)
wchan=wait_woken
cd ~/workspace/scripts && bash ./run_perf.sh --runs 3 # large 场景 3 轮跑完;small 场景 Run 1 Batch 1 永久卡住
挂死时进程状态:
create_experiment
113.46.219.251:8080
find_recent_experiment
171.108.215.116:443
两个连接都是 ESTABLISHED 但远端收下连接后不返回响应。注意这是真实 LLM API(Qwen 8080 / DeepSeek 443),不是本地 mock(32000/32001,mock 只 mock 了 MetaVisor)。small 场景触发压缩需要额外 Planner LLM 调用,正卡在这一步。
dataagent/core/managers/llm_manager/llm_client.py:1694 的 _resolve_timeout:
dataagent/core/managers/llm_manager/llm_client.py:1694
_resolve_timeout
def _resolve_timeout(self, kwargs: dict[str, Any]) -> httpx.Timeout | None: timeout = kwargs.get("timeout", self._timeout) return httpx.Timeout(timeout) if timeout is not None else None
测试的 MODEL_PRESETS(tests/e2e/changping/test_performance.py:415)只设了 model/base_url/api_key,没设 timeout → self._timeout = None → _resolve_timeout 返回 None → httpx.{Async}Client(timeout=None) 即无限等待。
MODEL_PRESETS
tests/e2e/changping/test_performance.py:415
model/base_url/api_key
timeout
self._timeout = None
None
httpx.{Async}Client(timeout=None)
httpx 在 timeout=None 时禁用所有超时(其内置默认 5s 被显式覆盖)。当远端 LLM API 建连后不返回响应(服务端 stall / 限流静默),recv 永久阻塞。重试层 _with_transient_retry / _awith_transient_retry 只捕获异常,而挂死的 recv 不抛任何异常,重试永远不触发 → 进程永久挂死。
timeout=None
recv
_with_transient_retry
_awith_transient_retry
三层修复(分支 fix/llm-client-timeout,基于 0709 @ 117e522,修复 commit bd45f1a):
fix/llm-client-timeout
117e522
bd45f1a
dataagent/utils/constants.py
DEFAULT_LLM_TIMEOUT=600.0
DEFAULT_LLM_CONNECT_TIMEOUT=30.0
httpx.Timeout(connect=30, read=600, write=30, pool=30)
read=600
connect/write/pool=30
MODEL[*].params.timeout
test_performance.py
_apply_model_choice
params.timeout
~/workspace/scripts/run_perf.sh
timeout -k 5 $PERF_CALL_TTL uv run ...
DEFAULT_LLM_TIMEOUT
DEFAULT_LLM_CONNECT_TIMEOUT
dataagent/core/managers/llm_manager/llm_client.py
_astream_iter
httpx.Timeout | None
httpx.Timeout
tests/e2e/changping/test_performance.py
scripts/run_perf.sh
timeout -k 5 $TTL
0709
fix(llm): LLMClient 默认超时兜底,杜绝 httpx timeout=None 永久挂死
docs(cache): 主 Agent 缓存优化最终设计文档 + 性能验证数据与图表
/home/qianlong/.local/opencode/logs/perf_20260711_004416_small_r1_{compare_openai,0623_deepseek}.log
[Bug][LLM] LLMClient 无默认超时,httpx timeout=None 致测试进程永久挂死
问题描述
性能测试
run_perf.sh --runs 3在small场景(--recent_turns 10 --compress_message_cnt 50)Run 1 Batch 1 卡住不退出。两个测试子进程从 00:44 启动后一直存活,状态S (sleeping)、wchan=wait_woken,CPU 累计仅 18s/11s(10h+ 内几乎不占用),日志在 01:00 后再无输出,永久挂死。复现
cd ~/workspace/scripts && bash ./run_perf.sh --runs 3 # large 场景 3 轮跑完;small 场景 Run 1 Batch 1 永久卡住挂死时进程状态:
create_experiment113.46.219.251:8080(Qwen)find_recent_experiment171.108.215.116:443(DeepSeek)两个连接都是 ESTABLISHED 但远端收下连接后不返回响应。注意这是真实 LLM API(Qwen 8080 / DeepSeek 443),不是本地 mock(32000/32001,mock 只 mock 了 MetaVisor)。small 场景触发压缩需要额外 Planner LLM 调用,正卡在这一步。
根因
dataagent/core/managers/llm_manager/llm_client.py:1694的_resolve_timeout:def _resolve_timeout(self, kwargs: dict[str, Any]) -> httpx.Timeout | None: timeout = kwargs.get("timeout", self._timeout) return httpx.Timeout(timeout) if timeout is not None else None测试的
MODEL_PRESETS(tests/e2e/changping/test_performance.py:415)只设了model/base_url/api_key,没设timeout→self._timeout = None→_resolve_timeout返回None→httpx.{Async}Client(timeout=None)即无限等待。httpx 在
timeout=None时禁用所有超时(其内置默认 5s 被显式覆盖)。当远端 LLM API 建连后不返回响应(服务端 stall / 限流静默),recv永久阻塞。重试层_with_transient_retry/_awith_transient_retry只捕获异常,而挂死的recv不抛任何异常,重试永远不触发 → 进程永久挂死。解决方案
三层修复(分支
fix/llm-client-timeout,基于 0709 @117e522,修复 commitbd45f1a):1. 根治:LLMClient 默认超时 + 结构化 Timeout
dataagent/utils/constants.py新增DEFAULT_LLM_TIMEOUT=600.0(read)/DEFAULT_LLM_CONNECT_TIMEOUT=30.0(connect/write/pool)_resolve_timeout永不返回None,改用结构化httpx.Timeout(connect=30, read=600, write=30, pool=30):read=600容忍慢推理模型长响应,同时作为流式空闲断流阈值(无 chunk 超过 600s 即判定挂死并触发重试)connect/write/pool=30快速暴露 TCP 连接故障,不放大到 600sMODEL[*].params.timeout(YAML 可配)作为 read 超时覆盖,其余维度仍用 30s2. 测试预设补 timeout
test_performance.py的MODEL_PRESETS各预设补timeout字段(deepseek/openai=600, bailian=900),_apply_model_choice注入params.timeout,确保性能测试永不挂死3. OS 级兜底(仓库外脚本)
~/workspace/scripts/run_perf.sh用timeout -k 5 $PERF_CALL_TTL uv run ...包裹每个测试进程,即便 Python client 再挂也会被 SIGKILL(默认 2400s,可 env 覆盖)改动清单
dataagent/utils/constants.pyDEFAULT_LLM_TIMEOUT/DEFAULT_LLM_CONNECT_TIMEOUTdataagent/core/managers/llm_manager/llm_client.py_resolve_timeout永不返回 None + 结构化 Timeout;_astream_iter签名httpx.Timeout | None→httpx.Timeouttests/e2e/changping/test_performance.pyMODEL_PRESETS补timeout+_apply_model_choice注入scripts/run_perf.sh(仓库外,不进 commit)timeout -k 5 $TTL包裹环境
fix/llm-client-timeout(基于0709@117e522)bd45f1afix(llm): LLMClient 默认超时兜底,杜绝 httpx timeout=None 永久挂死117e522docs(cache): 主 Agent 缓存优化最终设计文档 + 性能验证数据与图表/home/qianlong/.local/opencode/logs/perf_20260711_004416_small_r1_{compare_openai,0623_deepseek}.log