已开启
feat: add camera-observation skill #1
Lilili创建于 8月14日
feat: add camera-observation skill #1
已开启
共 3 个文件变更+305-0
| @@ -0,0 +1,128 @@ | |||
| 1 | +--- | ||
| 2 | +name: camera-observation | ||
| 3 | +description: Use when the user wants to observe live camera footage — watching what happens outside/around a camera, checking the state of hardware (LED lights, LCD screens, PCBA boards) in view, or continuously describing/commenting on real-time camera scenes. Triggers include "watch the camera", "看下摄像头/外面发生了什么", "灯在闪吗/LCD 显示什么", live video commentary requests. | ||
| 4 | +--- | ||
| 5 | + | ||
| 6 | +# Camera Observation | ||
| 7 | + | ||
| 8 | +## Overview | ||
| 9 | + | ||
| 10 | +Observe a live camera and describe what is happening, using a two-process pipeline: a Windows-side capture script writes short video segments, and a WSL-side agent analyzes each new segment with a resident (loaded-once) vision-language model. Core principle: **load the model once, keep it resident, and analyze each new segment — never re-load weights per segment.** | ||
| 11 | + | ||
| 12 | +## Environment Requirements (先读这里，环境已就绪，不要从零安装) | ||
| 13 | + | ||
| 14 | +**两个 venv 已搭好，直接使用，禁止重新安装 torch/transformers。** | ||
| 15 | + | ||
| 16 | +| 角色 | 位置 | 已含依赖 | | ||
| 17 | +| :--- | :--- | :--- | | ||
| 18 | +| 捕获端（Windows） | `C:\Users\TL\Documents\Visio\.venv\Scripts\python.exe` | opencv-python 5.x（cv2） | | ||
| 19 | +| 分析端（WSL） | `/root/mage/.venv/bin/python` | torch 2.10.0+cu128（CUDA 可用）、transformers 5.x、codec-video-prep、mamba-ssm、decord | | ||
| 20 | +| 模型 | `/mnt/c/Users/TL/Documents/Visio/models/Mage-VL`（11GB，ModelScope 下载） | 完整 checkpoint | | ||
| 21 | +| GPU | RTX 3080 20GB（Mage-VL BF16 需 8–10GB） | — | | ||
| 22 | +| 推理代码 | `/mnt/c/Users/TL/Documents/Visio/Mage/mage_vl/` | inference_base.py 等 | | ||
| 23 | + | ||
| 24 | +**必须执行的检查顺序（本技能标准流程）：** | ||
| 25 | + | ||
| 26 | +```bash | ||
| 27 | +# 1. 分析端 venv 就绪（WSL） | ||
| 28 | +wsl -e bash -c "/root/mage/.venv/bin/python -c 'import torch; print(torch.__version__, torch.cuda.is_available())'" | ||
| 29 | +# 2. 模型存在（WSL） | ||
| 30 | +wsl -e bash -c "ls /mnt/c/Users/TL/Documents/Visio/models/Mage-VL/config.json" | ||
| 31 | +# 3. 捕获端 venv 就绪（Windows） | ||
| 32 | +python -c "import cv2; print(cv2.__version__)" # 或用 .venv\Scripts\python.exe | ||
| 33 | +``` | ||
| 34 | + | ||
| 35 | +**If any check fails:** | ||
| 36 | +- torch/transformers 缺失 → 检查是否用了 `/root/mage/.venv/bin/python`（不是系统 `python3`）。若 venv 损坏，按 `camera-observation` 之外的环境修复流程处理，**不要在系统 python 里 `pip install torch`**（PEP 668 会拦截，且会破坏已装环境）。 | ||
| 37 | +- 模型缺失 → 从 ModelScope 重新下载 `microsoft/Mage-VL` 到 `models/Mage-VL`。 | ||
| 38 | +- GPU 不可用 → 无法运行 Mage-VL，改用帧采样 + 描述性 fallback 或提示用户。 | ||
| 39 | + | ||
| 40 | +**运行命令必须用 venv 解释器：** | ||
| 41 | +- 捕获：`C:\Users\TL\Documents\Visio\.venv\Scripts\python.exe scripts/capture_live.py ...` | ||
| 42 | +- 分析：`wsl -e bash -c "/root/mage/.venv/bin/python /mnt/c/Users/TL/.atomcode/skills/camera-observation/scripts/realtime_analyze.py --model /mnt/c/Users/TL/Documents/Visio/models/Mage-VL --live-dir /mnt/c/Users/TL/Documents/Visio/cam_live"` | ||
| 43 | + | ||
| 44 | +## When to Use | ||
| 45 | + | ||
| 46 | +- User wants to watch live camera footage and understand what is happening. | ||
| 47 | +- User asks about hardware state visible to a camera (LED blinking/steady, LCD screen content, PCBA board condition). | ||
| 48 | +- User wants continuous or periodic narration/commentary of a camera scene. | ||
| 49 | +- Any request where the camera view must be fed to a vision-language model for real-time understanding. | ||
| 50 | + | ||
| 51 | +Do NOT use for: analyzing a saved video file (use the offline inference scripts instead), or when no camera is attached. | ||
| 52 | + | ||
| 53 | +## Core Pattern | ||
| 54 | + | ||
| 55 | +``` | ||
| 56 | +[Windows] capture_live.py → cam_live/seg_*.mp4 (every N seconds) | ||
| 57 | + ↓ | ||
| 58 | +[WSL] realtime_analyze.py → Mage-VL (loaded once) → live narration | ||
| 59 | +``` | ||
| 60 | + | ||
| 61 | +Two long-running processes. The Windows side captures; the WSL side polls for new segments, analyzes each with frame sampling, and prints the narration. | ||
| 62 | + | ||
| 63 | +## Model Hanging (常驻模型) | ||
| 64 | + | ||
| 65 | +The analyzer keeps the model resident so segments are analyzed in seconds instead of re-loading weights (≈1 min) each time. | ||
| 66 | + | ||
| 67 | +- **How it works**: `realtime_analyze.py` loads the model ONCE at startup (`AutoModelForCausalLM.from_pretrained(...).eval()`), then loops forever, watching `cam_live/` for new `seg_*.mp4` files. | ||
| 68 | +- **Verify it is resident**: the log line `模型就绪 <device>，开始实时监听 <dir>/` appears after the ~1 min load; afterwards every segment is processed without re-loading. | ||
| 69 | +- **Keep it alive**: run it with `nohup ... &` or in a dedicated terminal — do NOT start it per-segment. | ||
| 70 | +- **Stop it**: `pkill -f realtime_analyze` (WSL). The capture side: stop the python process on Windows (`Get-Process python | Stop-Process` if needed). | ||
| 71 | +- **Restart after edits**: edit the script, then restart the analyzer so the changes take effect. | ||
| 72 | +- **Memory**: Mage-VL 4B BF16 ≈ 8–10 GB VRAM; ensure the GPU is free (RTX 3080 20GB works). | ||
| 73 | + | ||
| 74 | +## Using the Model (如何使用) | ||
| 75 | + | ||
| 76 | +Both scripts are parameterized — pass paths instead of editing them: | ||
| 77 | + | ||
| 78 | +```bash | ||
| 79 | +# Terminal 1 (Windows) | ||
| 80 | +python scripts/capture_live.py --segment 10 --out cam_live | ||
| 81 | + | ||
| 82 | +# Terminal 2 (WSL) — point MODEL/LIVE_DIR at your setup | ||
| 83 | +HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ | ||
| 84 | + python scripts/realtime_analyze.py \ | ||
| 85 | + --model /mnt/c/Users/TL/Documents/Visio/models/Mage-VL \ | ||
| 86 | + --live-dir /mnt/c/Users/TL/Documents/Visio/cam_live | ||
| 87 | +``` | ||
| 88 | + | ||
| 89 | +Parameters for `realtime_analyze.py`: | ||
| 90 | +- `--model`: path to the Mage-VL checkpoint (default `microsoft/Mage-VL`; use local path + offline env vars to avoid network). | ||
| 91 | +- `--live-dir`: directory the capture script writes segments into. | ||
| 92 | +- `--num-frames`: frames sampled per segment (default 12 — enough to catch animations/LED blinks). | ||
| 93 | +- `--max-new-tokens`: narration length (default 120). | ||
| 94 | +- `--question`: the prompt used to analyze each segment (LED/LCD-aware by default). | ||
| 95 | + | ||
| 96 | +The question prompt defaults to asking about glowing LCD screens playing GIFs, LED blink vs steady, and scene content — keep it if your hardware watch targets LEDs/LCDs, or change it for general scene description. | ||
| 97 | + | ||
| 98 | +## Quick Reference | ||
| 99 | + | ||
| 100 | +| Concern | Command / Setting | | ||
| 101 | +| :--- | :--- | | ||
| 102 | +| Start capture (Windows) | `python scripts/capture_live.py --segment 10 --out cam_live` | | ||
| 103 | +| Start analyzer (WSL) | `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python scripts/realtime_analyze.py --model <path> --live-dir <dir>` | | ||
| 104 | +| Segment length | `--segment` (seconds; 10 is a good default) | | ||
| 105 | +| Frames sampled per segment | `--num-frames` (default 12) | | ||
| 106 | +| Live log | analyzer prints `[解说] ...` per segment | | ||
| 107 | +| Camera not readable | Often "device in use" — kill stale python/capture processes or ask user to replug USB camera | | ||
| 108 | +| Stop analyzer | `pkill -f realtime_analyze` (WSL) | | ||
| 109 | + | ||
| 110 | +## Implementation | ||
| 111 | + | ||
| 112 | +Both scripts live in `scripts/`: | ||
| 113 | + | ||
| 114 | +- **`scripts/capture_live.py`** (Windows, OpenCV): opens the camera (DSHOW preferred, falls back to default), writes a video segment every N seconds, retries failed frame grabs, and discards segments smaller than 50 KB (invalid). | ||
| 115 | +- **`scripts/realtime_analyze.py`** (WSL, PyTorch + transformers): loads Mage-VL once with `attn_implementation="sdpa"`, then polls `--live-dir` for new segments and analyzes each unseen segment by frame-sampling `--num-frames` frames and asking the model to describe the scene (LED/LCD-aware question prompt). | ||
| 116 | + | ||
| 117 | +## Common Mistakes | ||
| 118 | + | ||
| 119 | +- **Re-loading the model per segment**: kills latency (each load ≈ 1 min). Keep the analyzer resident — this is the whole point of the two-process design. | ||
| 120 | +- **Camera "device in use"**: a stale capture process or another app holds the camera. Kill leftover python/capture processes, or ask the user to unplug/replug the camera. | ||
| 121 | +- **Empty/invalid segments**: OpenCV can write a truncated file if the camera glitches; the capture script already discards <50 KB segments — keep that guard. | ||
| 122 | +- **Forgetting offline env vars**: run the analyzer with `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` so it never tries the network. | ||
| 123 | +- **Hardcoded paths**: pass `--model` / `--live-dir`; do not edit the script per machine. | ||
| 124 | +- **Starting the analyzer per segment**: wrong — one resident process serves all segments. | ||
| 125 | + | ||
| 126 | +## Real-World Impact | ||
| 127 | + | ||
| 128 | +Live observation successfully identified a round glowing LCD screen playing an animated GIF (a cartoon face changing expressions) plus a blinking green LED — details that single-frame analysis had repeatedly missed, because the 12-frame sampling captured the animation. | ||
| @@ -0,0 +1,65 @@ | |||
| 1 | +"""Windows 侧：持续抓取摄像头，每 N 秒保存一个片段供 WSL 模型实时分析。 | ||
| 2 | + | ||
| 3 | +修复：强制 DSHOW 后端（避免 MSMF 抓帧失败）、抓帧失败自动重试、 | ||
| 4 | +片段过小视为无效丢弃。产出: cam_live/seg_<timestamp>.mp4 | ||
| 5 | +""" | ||
| 6 | +import argparse | ||
| 7 | +import cv2 | ||
| 8 | +import time | ||
| 9 | +import os | ||
 | |||
| 10 | +from pathlib import Path | ||
| 11 | + | ||
| 12 | + | ||
| 13 | +def main(): | ||
| 14 | + p = argparse.ArgumentParser() | ||
| 15 | + p.add_argument("--camera", type=int, default=0) | ||
| 16 | + p.add_argument("--segment", type=float, default=10.0, help="每段秒数") | ||
| 17 | + p.add_argument("--fps", type=int, default=10) | ||
| 18 | + p.add_argument("--out", default="cam_live") | ||
| 19 | + args = p.parse_args() | ||
| 20 | + | ||
| 21 | + out = Path(args.out) | ||
| 22 | + out.mkdir(exist_ok=True) | ||
| 23 | + | ||
| 24 | + # 只使用 DSHOW 后端，避免回退到 MSMF 导致抓帧失败 | ||
| 25 | + cap = cv2.VideoCapture(args.camera, cv2.CAP_DSHOW) | ||
| 26 | + if not cap.isOpened(): | ||
| 27 | + print("[错误] DSHOW 无法打开摄像头，尝试默认后端 ...", flush=True) | ||
| 28 | + cap = cv2.VideoCapture(args.camera) | ||
| 29 | + if not cap.isOpened(): | ||
| 30 | + print("[错误] 无法打开摄像头", flush=True) | ||
| 31 | + return | ||
| 32 | + | ||
| 33 | + w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH) or 640) | ||
| 34 | + h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT) or 480) | ||
| 35 | + print(f"[信息] 摄像头 {args.camera} 打开 ({w}x{h})，每 {args.segment}s 一段", flush=True) | ||
| 36 | + | ||
| 37 | + seg = 0 | ||
| 38 | + min_size = 50 * 1024 # 小于 50KB 视为无效片段 | ||
| 39 | + while True: | ||
| 40 | + stamp = time.strftime("%H%M%S") | ||
| 41 | + path = out / f"seg_{seg:04d}_{stamp}.mp4" | ||
| 42 | + writer = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), args.fps, (w, h)) | ||
🟠 High Priority 变更行：capture_live.py:42 直接用最终文件名 建议：先写入临时文件名（如   不准确？ | |||
| 43 | + t0 = time.time() | ||
| 44 | + written = 0 | ||
| 45 | + while time.time() - t0 < args.segment: | ||
| 46 | + ok, frame = cap.read() | ||
| 47 | + if not ok: | ||
| 48 | + # 抓帧失败：短暂等待重试（摄像头可能被占用/信号抖动） | ||
| 49 | + time.sleep(0.2) | ||
| 50 | + continue | ||
| 51 | + writer.write(frame) | ||
| 52 | + written += 1 | ||
| 53 | + writer.release() | ||
| 54 | + size = path.stat().st_size if path.exists() else 0 | ||
| 55 | + if size < min_size or written < 5: | ||
| 56 | + # 无效片段：删除并重试 | ||
| 57 | + path.unlink(missing_ok=True) | ||
| 58 | + print(f"[警告] {path.name} 无效({size}B/{written}帧)，丢弃", flush=True) | ||
| 59 | + else: | ||
| 60 | + print(f"[抓取] {path.name} ({size//1024}KB, {written}帧)", flush=True) | ||
| 61 | + seg += 1 | ||
| 62 | + | ||
| 63 | + | ||
| 64 | +if __name__ == "__main__": | ||
| 65 | + main() | ||
| @@ -0,0 +1,112 @@ | |||
| 1 | +"""WSL 侧：模型常驻加载一次，循环分析摄像头片段并输出解说。 | ||
| 2 | + | ||
| 3 | +配合 Windows 侧 capture_live.py 使用：它持续把摄像头画面存成片段， | ||
| 4 | +本脚本轮询新片段 -> Mage-VL 帧采样分析 -> 打印解说。 | ||
| 5 | + | ||
| 6 | +用法示例: | ||
| 7 | + HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ | ||
| 8 | + python realtime_analyze.py \ | ||
| 9 | + --model /mnt/c/Users/TL/Documents/Visio/models/Mage-VL \ | ||
| 10 | + --live-dir /mnt/c/Users/TL/Documents/Visio/cam_live | ||
| 11 | +""" | ||
| 12 | +import argparse | ||
| 13 | +import os | ||
| 14 | +import sys | ||
🔵 Low Priority 变更行：realtime_analyze.py:14 建议：删除第 14 行未使用的 不准确？ | |||
| 15 | +import time | ||
| 16 | +import glob | ||
| 17 | +from pathlib import Path | ||
| 18 | + | ||
| 19 | +os.environ["HF_HUB_OFFLINE"] = "1" | ||
| 20 | +os.environ["TRANSFORMERS_OFFLINE"] = "1" | ||
| 21 | + | ||
| 22 | +import torch # noqa: E402 | ||
| 23 | +from transformers import AutoModelForCausalLM, AutoProcessor # noqa: E402 | ||
| 24 | + | ||
| 25 | +DEFAULT_QUESTION = ( | ||
| 26 | + "实时解说这段摄像头画面：请描述画面里发生了什么。" | ||
| 27 | + "特别注意：有没有发光的 LCD 屏幕在播放 GIF 动图？上面显示什么？" | ||
| 28 | + "有没有 LED 灯？它在闪烁还是常亮？" | ||
| 29 | +) | ||
| 30 | + | ||
| 31 | + | ||
| 32 | +def parse_args() -> argparse.Namespace: | ||
| 33 | + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) | ||
| 34 | + p.add_argument("--model", default="microsoft/Mage-VL", | ||
| 35 | + help="Mage-VL 模型路径（建议本地路径）") | ||
| 36 | + p.add_argument("--live-dir", default="/mnt/c/Users/TL/Documents/Visio/cam_live", | ||
| 37 | + help="capture_live.py 输出片段的目录") | ||
| 38 | + p.add_argument("--num-frames", type=int, default=12, | ||
| 39 | + help="每个片段采样的帧数") | ||
| 40 | + p.add_argument("--max-new-tokens", type=int, default=120, | ||
| 41 | + help="解说最大 token 数") | ||
| 42 | + p.add_argument("--question", default=DEFAULT_QUESTION, | ||
| 43 | + help="分析每个片段时使用的提示词") | ||
| 44 | + return p.parse_args() | ||
| 45 | + | ||
| 46 | + | ||
| 47 | +def analyze_segment(model, processor, video_path: str, num_frames: int, | ||
| 48 | + max_new_tokens: int, question: str) -> str: | ||
| 49 | + import cv2 | ||
| 50 | + import numpy as np | ||
| 51 | + from PIL import Image | ||
| 52 | + | ||
| 53 | + # 帧采样 | ||
| 54 | + cap = cv2.VideoCapture(video_path) | ||
| 55 | + total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) | ||
| 56 | + if total <= 0: | ||
| 57 | + cap.release() | ||
| 58 | + return "(无法读取片段)" | ||
| 59 | + indices = np.linspace(0, total - 1, min(num_frames, total), dtype=int) | ||
| 60 | + frames = [] | ||
| 61 | + for i in indices: | ||
| 62 | + cap.set(cv2.CAP_PROP_POS_FRAMES, int(i)) | ||
| 63 | + ok, f = cap.read() | ||
| 64 | + if ok: | ||
| 65 | + frames.append(Image.fromarray(cv2.cvtColor(f, cv2.COLOR_BGR2RGB))) | ||
| 66 | + cap.release() | ||
| 67 | + | ||
| 68 | + messages = [{"role": "user", "content": [ | ||
| 69 | + {"type": "video"}, {"type": "text", "text": question}, | ||
| 70 | + ]}] | ||
| 71 | + text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | ||
| 72 | + inputs = processor(text=[text], videos=[frames], return_tensors="pt", padding=True) | ||
| 73 | + inputs = {k: (v.to(model.device) if hasattr(v, "to") else v) for k, v in inputs.items()} | ||
| 74 | + if "pixel_values" in inputs: | ||
| 75 | + inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype) | ||
| 76 | + with torch.inference_mode(): | ||
| 77 | + out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False) | ||
| 78 | + return processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip() | ||
| 79 | + | ||
| 80 | + | ||
| 81 | +def main(): | ||
| 82 | + args = parse_args() | ||
| 83 | + live_dir = Path(args.live_dir) | ||
| 84 | + live_dir.mkdir(parents=True, exist_ok=True) | ||
| 85 | + | ||
| 86 | + print(f"[信息] 加载模型（首次约 1 分钟，之后常驻）: {args.model} ...", flush=True) | ||
| 87 | + processor = AutoProcessor.from_pretrained(args.model, trust_remote_code=True) | ||
| 88 | + model = AutoModelForCausalLM.from_pretrained( | ||
| 89 | + args.model, trust_remote_code=True, torch_dtype="auto", | ||
| 90 | + device_map="auto", attn_implementation="sdpa", | ||
| 91 | + ).eval() | ||
| 92 | + print(f"[信息] 模型就绪 {model.device}，开始实时监听 {live_dir}/", flush=True) | ||
| 93 | + | ||
| 94 | + seen = set() | ||
| 95 | + while True: | ||
| 96 | + segs = sorted(glob.glob(str(live_dir / "seg_*.mp4"))) | ||
| 97 | + new = [s for s in segs if s not in seen] | ||
| 98 | + for s in new: | ||
| 99 | + seen.add(s) | ||
| 100 | + t = time.strftime("%H:%M:%S") | ||
| 101 | + print(f"\n[{t}] 分析 {Path(s).name} ...", flush=True) | ||
| 102 | + try: | ||
| 103 | + desc = analyze_segment(model, processor, s, | ||
| 104 | + args.num_frames, args.max_new_tokens, args.question) | ||
| 105 | + print(f"[解说] {desc}", flush=True) | ||
| 106 | + except Exception as e: | ||
| 107 | + print(f"[错误] {e}", flush=True) | ||
| 108 | + time.sleep(1) | ||
| 109 | + | ||
| 110 | + | ||
| 111 | +if __name__ == "__main__": | ||
| 112 | + main() | ||
🔵 Low Priority
变更行：capture_live.py:9
import os在文件中从未被引用，属于死代码/未引用导入，lint 会告警且无任何用途。建议：删除第 9 行未使用的
import os。