| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(prompt): show phase + token progress for LLM prompt expansion (#9204) * feat(prompt): show phase + token progress for LLM prompt expansion Both /utilities/expand-prompt and /utilities/image-to-prompt were blocking calls with only a spinner — users had no signal whether the model was still loading or already generating, which is rough on larger LLMs. The pipelines now stream tokens via HuggingFace's TextIteratorStreamer, and the routes emit new llm_task_progress / complete / error socket events correlated to a client-supplied task_id and routed privately to the requesting user. The Expand Prompt and Image-to-Prompt popovers render a phase label ("Loading model…" / "Generating…") plus a token progress bar (current / max_tokens) while the request is in flight. * Chore Ruff * Chore openapi * fix(utilities): restore missing events binding in image_to_prompt The merge with upstream/main dropped `events = ApiDependencies.invoker.services.events` from `image_to_prompt` when inserting the new image-read access check, leaving five `events.emit_llm_task_*` calls dangling as NameErrors whenever a task_id is sent. Restore the binding to match `expand_prompt`. * fix(prompt): address review findings for LLM prompt progress Critical: prevent a deadlock when model.generate() raises in the producer thread. transformers only calls streamer.end() on the normal exit of the generation loop, so on failure (e.g. CUDA OOM) the consuming loop blocked forever, leaking the worker and pinning the model on device. Both pipelines now call streamer.end() in the worker's except block and pass a generous STREAM_TIMEOUT to the streamer as a backstop for hangs that don't raise; queue.Empty is surfaced as an error instead of blocking on thread.join(). Adds regression tests that drive generate().side_effect through the real streamer for both the text and LLaVA pipelines. Medium: fix a per-request store leak. llm_task_complete/llm_task_error now delete the store key instead of writing a terminal state, so a socket event arriving after the mutation's finally-clear can't orphan an entry. The now dead error branch of LLMTaskProgressDisplay is removed (errors surface via the RTK Query toast); the LLMTaskState union collapses to the progress case. Minor: throttle progress emissions (>=100ms) with a guaranteed final emit to bound the O(n^2) re-encode + socket cost at large max_tokens; use the generated S['LLMTask*Event'] schema types instead of hand-typed payloads; extract a shared _make_progress_callback helper to keep the two endpoints in sync; add LLaVA pipeline test coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | 1 个月前 | |
feat(prompt): show phase + token progress for LLM prompt expansion (#9204) * feat(prompt): show phase + token progress for LLM prompt expansion Both /utilities/expand-prompt and /utilities/image-to-prompt were blocking calls with only a spinner — users had no signal whether the model was still loading or already generating, which is rough on larger LLMs. The pipelines now stream tokens via HuggingFace's TextIteratorStreamer, and the routes emit new llm_task_progress / complete / error socket events correlated to a client-supplied task_id and routed privately to the requesting user. The Expand Prompt and Image-to-Prompt popovers render a phase label ("Loading model…" / "Generating…") plus a token progress bar (current / max_tokens) while the request is in flight. * Chore Ruff * Chore openapi * fix(utilities): restore missing events binding in image_to_prompt The merge with upstream/main dropped `events = ApiDependencies.invoker.services.events` from `image_to_prompt` when inserting the new image-read access check, leaving five `events.emit_llm_task_*` calls dangling as NameErrors whenever a task_id is sent. Restore the binding to match `expand_prompt`. * fix(prompt): address review findings for LLM prompt progress Critical: prevent a deadlock when model.generate() raises in the producer thread. transformers only calls streamer.end() on the normal exit of the generation loop, so on failure (e.g. CUDA OOM) the consuming loop blocked forever, leaking the worker and pinning the model on device. Both pipelines now call streamer.end() in the worker's except block and pass a generous STREAM_TIMEOUT to the streamer as a backstop for hangs that don't raise; queue.Empty is surfaced as an error instead of blocking on thread.join(). Adds regression tests that drive generate().side_effect through the real streamer for both the text and LLaVA pipelines. Medium: fix a per-request store leak. llm_task_complete/llm_task_error now delete the store key instead of writing a terminal state, so a socket event arriving after the mutation's finally-clear can't orphan an entry. The now dead error branch of LLMTaskProgressDisplay is removed (errors surface via the RTK Query toast); the LLMTaskState union collapses to the progress case. Minor: throttle progress emissions (>=100ms) with a guaranteed final emit to bound the O(n^2) re-encode + socket cost at large max_tokens; use the generated S['LLMTask*Event'] schema types instead of hand-typed payloads; extract a shared _make_progress_callback helper to keep the two endpoints in sync; add LLaVA pipeline test coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 1 个月前 |