main

TTekniumfeat(computer-use): cua-driver backend, universal any-model schema

850413f1创建于 26 天前历史提交

文件	最后提交记录	最后更新时间
apple-notes	docs(skills): compress 74 built-in skill descriptions to <=60 chars Target: every skill's description fits in a one-line gateway menu and leads with trigger keywords an agent would match on. Drops filler like 'Use this skill to', 'A skill for', 'This skill provides'. Before: max description length was 791 chars (architecture-diagram), 74 of 81 built-in skills were >60 chars. After: max 60, mean 54, all 81 built-in skills <=60. Rewritten with double-quoted YAML scalars to preserve Chinese/arrow glyphs (baoyu-comic, yuanbao, youtube-content).	1 个月前
apple-reminders	docs(skills): compress 74 built-in skill descriptions to <=60 chars Target: every skill's description fits in a one-line gateway menu and leads with trigger keywords an agent would match on. Drops filler like 'Use this skill to', 'A skill for', 'This skill provides'. Before: max description length was 791 chars (architecture-diagram), 74 of 81 built-in skills were >60 chars. After: max 60, mean 54, all 81 built-in skills <=60. Rewritten with double-quoted YAML scalars to preserve Chinese/arrow glyphs (baoyu-comic, yuanbao, youtube-content).	1 个月前
findmy	docs(skills): compress 74 built-in skill descriptions to <=60 chars Target: every skill's description fits in a one-line gateway menu and leads with trigger keywords an agent would match on. Drops filler like 'Use this skill to', 'A skill for', 'This skill provides'. Before: max description length was 791 chars (architecture-diagram), 74 of 81 built-in skills were >60 chars. After: max 60, mean 54, all 81 built-in skills <=60. Rewritten with double-quoted YAML scalars to preserve Chinese/arrow glyphs (baoyu-comic, yuanbao, youtube-content).	1 个月前
imessage	feat: secure skill env setup on load (core #688) When a skill declares required_environment_variables in its YAML frontmatter, missing env vars trigger a secure TUI prompt (identical to the sudo password widget) when the skill is loaded. Secrets flow directly to ~/.hermes/.env, never entering LLM context. Key changes: - New required_environment_variables frontmatter field for skills - Secure TUI widget (masked input, 120s timeout) - Gateway safety: messaging platforms show local setup guidance - Legacy prerequisites.env_vars normalized into new format - Remote backend handling: conservative setup_needed=True - Env var name validation, file permissions hardened to 0o600 - Redact patterns extended for secret-related JSON fields - 12 existing skills updated with prerequisites declarations - ~48 new tests covering skip, timeout, gateway, remote backends - Dynamic panel widget sizing (fixes hardcoded width from original PR) Cherry-picked from PR #723 by kshitijk4poor, rebased onto current main with conflict resolution. Fixes #688 Co-authored-by: kshitijk4poor <kshitijk4poor@users.noreply.github.com>	2 个月前
macos-computer-use	feat(computer-use): cua-driver backend, universal any-model schema Background macOS desktop control via cua-driver MCP — does NOT steal the user's cursor or keyboard focus, works with any tool-capable model. Replaces the Anthropic-native computer_20251124 approach from the abandoned #4562 with a generic OpenAI function-calling schema plus SOM (set-of-mark) captures so Claude, GPT, Gemini, and open models can all drive the desktop via numbered element indices. - tools/computer_use/ package — swappable ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary). - Universal computer_use tool with one schema for all providers. Actions: capture (som/vision/ax), click, double_click, right_click, middle_click, drag, scroll, type, key, wait, list_apps, focus_app. - Multimodal tool-result envelope (_multimodal=True, OpenAI-style content: [text, image_url] parts) that flows through handle_function_call into the tool message. Anthropic adapter converts into native tool_result image blocks; OpenAI-compatible providers get the parts list directly. - Image eviction in convert_messages_to_anthropic: only the 3 most recent screenshots carry real image data; older ones become text placeholders to cap per-turn token cost. - Context compressor image pruning: old multimodal tool results have their image parts stripped instead of being skipped. - Image-aware token estimation: each image counts as a flat 1500 tokens instead of its base64 char length (~1MB would have registered as ~250K tokens before). - COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset is active. - Session DB persistence strips base64 from multimodal tool messages. - Trajectory saver normalises multimodal messages to text-only. - hermes tools post-setup installs cua-driver via the upstream script and prints permission-grant instructions. - CLI approval callback wired so destructive computer_use actions go through the same prompt_toolkit approval dialog as terminal commands. - Hard safety guards at the tool level: blocked type patterns (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash, force delete, lock screen, log out). - Skill apple/macos-computer-use/SKILL.md — universal (model-agnostic) workflow guide. - Docs: user-guide/features/computer-use.md plus reference catalog entries. 44 new tests in tests/tools/test_computer_use.py covering schema shape (universal, not Anthropic-native), dispatch routing, safety guards, multimodal envelope, Anthropic adapter conversion, screenshot eviction, context compressor pruning, image-aware token estimation, run_agent helpers, and universality guarantees. 469/469 pass across tests/tools/test_computer_use.py + the affected agent/ test suites. - model_tools.py provider-gating: the tool is available to every provider. Providers without multi-part tool message support will see text-only tool results (graceful degradation via text_summary). - Anthropic server-side clear_tool_uses_20250919 — deferred; client-side eviction + compressor pruning cover the same cost ceiling without a beta header. - macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that can break on any macOS update. Pin with HERMES_CUA_DRIVER_VERSION. - Requires Accessibility + Screen Recording permissions — the post-setup prints the Settings path. Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic- native schema). Credit @0xbyt4 for the original #3816 groundwork whose context/eviction/token design is preserved here in generic form.	26 天前
DESCRIPTION.md	feat(computer-use): cua-driver backend, universal any-model schema Background macOS desktop control via cua-driver MCP — does NOT steal the user's cursor or keyboard focus, works with any tool-capable model. Replaces the Anthropic-native computer_20251124 approach from the abandoned #4562 with a generic OpenAI function-calling schema plus SOM (set-of-mark) captures so Claude, GPT, Gemini, and open models can all drive the desktop via numbered element indices. - tools/computer_use/ package — swappable ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary). - Universal computer_use tool with one schema for all providers. Actions: capture (som/vision/ax), click, double_click, right_click, middle_click, drag, scroll, type, key, wait, list_apps, focus_app. - Multimodal tool-result envelope (_multimodal=True, OpenAI-style content: [text, image_url] parts) that flows through handle_function_call into the tool message. Anthropic adapter converts into native tool_result image blocks; OpenAI-compatible providers get the parts list directly. - Image eviction in convert_messages_to_anthropic: only the 3 most recent screenshots carry real image data; older ones become text placeholders to cap per-turn token cost. - Context compressor image pruning: old multimodal tool results have their image parts stripped instead of being skipped. - Image-aware token estimation: each image counts as a flat 1500 tokens instead of its base64 char length (~1MB would have registered as ~250K tokens before). - COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset is active. - Session DB persistence strips base64 from multimodal tool messages. - Trajectory saver normalises multimodal messages to text-only. - hermes tools post-setup installs cua-driver via the upstream script and prints permission-grant instructions. - CLI approval callback wired so destructive computer_use actions go through the same prompt_toolkit approval dialog as terminal commands. - Hard safety guards at the tool level: blocked type patterns (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash, force delete, lock screen, log out). - Skill apple/macos-computer-use/SKILL.md — universal (model-agnostic) workflow guide. - Docs: user-guide/features/computer-use.md plus reference catalog entries. 44 new tests in tests/tools/test_computer_use.py covering schema shape (universal, not Anthropic-native), dispatch routing, safety guards, multimodal envelope, Anthropic adapter conversion, screenshot eviction, context compressor pruning, image-aware token estimation, run_agent helpers, and universality guarantees. 469/469 pass across tests/tools/test_computer_use.py + the affected agent/ test suites. - model_tools.py provider-gating: the tool is available to every provider. Providers without multi-part tool message support will see text-only tool results (graceful degradation via text_summary). - Anthropic server-side clear_tool_uses_20250919 — deferred; client-side eviction + compressor pruning cover the same cost ceiling without a beta header. - macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that can break on any macOS update. Pin with HERMES_CUA_DRIVER_VERSION. - Requires Accessibility + Screen Recording permissions — the post-setup prints the Settings path. Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic- native schema). Credit @0xbyt4 for the original #3816 groundwork whose context/eviction/token design is preserved here in generic form.	26 天前