Automated Penetration Testing Agentic Framework Powered by Large Language Models
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
chore: remove XBOW product integration (#496) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration | 1 个月前 | |
chore: remove XBOW product integration (#496) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration | 1 个月前 | |
Legacy multi llm base (#470) * fix: 🐛 minor typo and build process * feat: 🎸 [WIP] Pentest mode * feat: 🎸 code abstraction * feat: modernize legacy PentestGPT with native multi-LLM support (#469) Rebuild the classic USENIX-2024 interactive PentestGPT (reasoning / generation / parsing sessions + Pentesting Task Tree + REPL) as a standalone `pentestgpt_legacy` package on a native per-provider LLM layer that supports the latest 2026 models. - llm/: BaseProvider + OpenAI-compatible / Anthropic / Gemini connectors, a web-verified model registry (OpenAI, Anthropic, Gemini, DeepSeek, xAI, Qwen, Moonshot, local Ollama), a factory, and an LLMClient bridging async providers to the core's synchronous send_new_message/send_message session API. - CLI `pentestgpt-legacy`: --list-models and --smoke-test (live per-model round-trip matrix), plus --reasoning-model / --parsing-model / --base-url. - Tests: 25 unit tests (mocked, no network). Live smoke test verified 22/22 models with a configured key respond. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(backend): address review on ClaudeCodeBackend subprocess handling - _build_env: pop ANTHROPIC_API_KEY instead of setting it to "", so an empty value can't shadow the CLI's own auth fallback (e.g. subscription login). - _kill_process: reap the force-killed process with os.waitpid(.., WNOHANG) instead of calling the proc.wait() coroutine without awaiting it (removes the "coroutine was never awaited" warning). - query/_drain_stderr: drain subprocess stderr in a background task so its pipe buffer can't fill and deadlock the child. Also reformats backend.py, fixing the failing Lint (ruff format) check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docker-test): assert uv instead of Poetry in container health check The project migrated from Poetry to uv (the Dockerfile installs uv to /home/pentester/.local/bin, which is on PATH), so test_poetry_installed failed with exit 127. Replace it with test_uv_installed checking `uv --version`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 2 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: 🎸 improve langfuse logging results (#388) * feat: 🎸 improve langfuse logging results * style: 💄 lint fix | 7 个月前 | |
chore: remove XBOW product integration (#496) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration | 1 个月前 | |
chore: remove XBOW product integration (#496) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
Create LICENSE.md | 3 年前 | |
chore: remove XBOW product integration (#496) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
docs: mark XBOW as reference-only (#497) * chore: promote unified-agent to 0.3 * chore: remove XBOW product integration * docs: mark XBOW as reference-only | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 | |
feat: ship the durable multi-model autonomous PentestGPT runtime (#493) * first refactor * feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for BOTH Claude Code and Codex, and add a multi-model benchmark harness. Backend (multi-model): - Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/ backend.py) wraps unified_agent's Codex backend and translates its events into AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction, and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1). Docker tool (tool-only image; the benchmark stays OUTSIDE the image): - Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/ pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore (keeps creds/benchmark/workspace out of the build context). - Persistent dual login (the hard part) — asymmetric by token model: * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS host creds live in the Keychain and can't be copied). * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455 OAuth callback is forwarded into the container via a socat hop (-p 1455:8455). * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s). - docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/ run/shell/down/nuke). - Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login. Benchmark (multi-model, host-side): - benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers the loopback port, runs the pipeline across the 4 model combos, judges by the baked FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates). Includes the partial pilot's results (results.jsonl + REPORT.md). Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fail controller on backend error messages * fix: allow listing sessions without target * docs: add docker xbow benchmark report * fix: infer concrete backend constructor type * docs: refresh docker benchmark documentation * feat(benchmark): add pure single-agent baseline + pipeline comparison Add a "pure single agent" benchmark variant -- one bare `claude -p` / `codex exec` call per target (no pipeline) -- to quantify what the 3-stage PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets. - pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the pipeline's shared fragments collapsed into ONE turn, so prompt content is held constant and the only variable is the multi-stage decomposition. - benchmark/pilot/run_docker_bench.py: docker-network runner (--variant single|pipeline). Brings the target up, discovers the container's internal IP+network (skips DB side-cars/ports), docker-runs the tool image on that network, and scores the ground-truth flag against the agent's *assistant text* only (parity with the pipeline's raw streaming). Reads stdout in chunks to handle >64KB JSON lines. Resumable; --dry-run supported. - benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head pipeline-vs-single per model on the common non-infra set. - tests/unit/test_single_agent_prompt.py: prompt-builder coverage. - docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated. Recorded result (10 medium/hard targets x 4 models, container-to-container, same baseline image digest 0c4c0f3e..., commit dca0019 image): Model Pipeline Single Claude Opus 5/10 7/10 (single +2) Claude Sonnet 6/10 4/10 (pipeline +2) Codex gpt-5.5 7/10 7/10 (tie) Codex gpt-5.4-mini 3/10 4/10 (single +1) TOTAL 21/40 22/40 Single agent matches the pipeline on solve rate (55% vs 52%) while using ~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline only clearly helps Claude Sonnet (which times out solo); Opus is better solo. Full per-challenge grid in DOCKER_COMPARISON.md; raw records in docker_single_results.jsonl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(benchmark): add pentestgpt_agent docker harness * bench: refresh pentestgpt_agent smoke result * fix(benchmark): make repeat rows variant-aware * fix(agent): fall back for semantic executor labels * fix(agent): tolerate executor prose evidence * fix(benchmark): score accepted framework findings * bench: append partial framework repeat results * bench: complete framework repeat sweep * bench: expose framework executor concurrency * bench: add extended parallel framework sweep * checkpoint: preserve working agent and benchmark state * feat: harden durable agent loop and xbow qualification * fix: reserve an exploit result turn * docs: record clean xbow qualification * build: consume unified-agent from the git wrapper repo Repoint pentestgpt_agent_new's unified-agent dependency from the local editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock and update test_dependency.py to assert the external package is installed from that VCS URL (not the repo-root vendored copy) at version 0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: make pentestgpt_agent_new the sole framework Remove the retired ledger-based pentestgpt_agent package (instructor/executor/ judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent from the git wrapper) is now the single maintained framework. Repoint the top-level tooling to it: - pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from the wheel packages. - Makefile: lint/format target parent code only; typecheck/check/ci now run the nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new / check-agent-new, so `make check` finally covers it; `make run` delegates to the pentestgpt-agent-new CLI. - Dockerfile: stop copying the removed package (kept the build working); note the framework is not baked into the image yet. - docker container-health test: import the substrate packages that actually ship. - CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and the deprioritized benchmark/Docker rewire. The XBOW `--variant framework` path and docker-bench Makefile targets still point at the old in-image framework and are left as a pending rewire (benchmarks deprioritized); the naive `--variant single` path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename pentestgpt_agent_new -> pentestgpt_agent The framework reclaims the clean name now that the old ledger-based package is gone. Rename the nested project folder, its src package, the distribution (pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in the package, the umbrella Makefile, the Dockerfile, the docker health test, and CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit; the git-sourced unified-agent dependency is unchanged. `make check` is green (108 nested tests). The two historical *_REPORT.md files keep the old name as dated records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: extract benchmark harness to sibling xbow-benchmark repo Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this project clean. The harness was decoupled from the framework code (it scores container output, never imports pentestgpt_agent/unified_agent), so only operational ties remain and they now live in the sibling repo. - Remove benchmark/ and the 4 harness unit tests (relocated + repointed there). - Strip the docker-bench-*/bench-* targets and their config vars from the Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a help pointer to `make -C ../xbow-benchmark help`. The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and runs the pentestgpt:latest image built here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: harden autonomous framework and runtime integration --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 1 个月前 |
以下内容由 AI 翻译，如有问题请 点此提交 issue 反馈
PentestGPT
人工智能驱动的自主渗透测试代理
发表于 USENIX Security 2024
官方网站：pentestgpt.com »
研究论文
·
报告漏洞
·
请求功能
演示
安装
PentestGPT 实际应用
v1.0（智能代理升级）新特性
- 多阶段工作流 - 代理按阶段执行任务（CTF 场景：信息收集 → 漏洞利用 → 渗透报告；渗透测试场景：资产发现 → 漏洞识别 → 报告生成），每个阶段的成果将作为下一阶段的输入。
- 自主代理 - 驱动 Claude Code 或 Codex 运行工具并进行推理，无需人工干预。
- 会话持久化 - 保存和恢复渗透测试会话。
自主 CTF 工作流支持后端接入 Claude Code 和 Codex。交互式现代化 legacy模式（
pentestgpt-legacy）支持更多提供商：OpenAI、Anthropic、Google Gemini、DeepSeek、xAI、Qwen、Moonshot 以及本地 Ollama。详见交互式多 LLM 模式。
功能特点
- AI驱动的挑战解决器 - 利用LLM的高级推理能力执行渗透测试和CTF竞赛
- 实时操作演示 - 在智能体解决挑战的过程中实时跟踪步骤
- 多类别支持 - Web、密码学、逆向工程、取证分析、PWN、权限提升
- 实时反馈 - 通过实时活动更新观察AI的工作过程
- 可扩展架构 - 简洁的模块化设计，为未来增强功能做好准备
快速开始
前提条件
- Python 3.12+
- uv - Python包管理器
- Claude Code CLI (
claude) - 已安装并通过身份验证，用于本地Claude运行。参见Claude Code文档 - Codex CLI (
codex) - 已安装并通过身份验证，用于本地Codex运行。下方的Docker流程已捆绑这两个CLI。
安装
git clone https://github.com/GreyDGL/PentestGPT.git
cd PentestGPT
make install # runs uv sync
命令参考
| 命令 | 描述 |
|---|---|
make install |
安装依赖 |
make test |
运行所有测试 |
make check |
运行代码检查 + 类型检查 |
make build |
构建可分发软件包 |
使用方法
# Run against a target (CTF mode by default)
pentestgpt --target 10.10.11.234
# With challenge context
pentestgpt --target 10.10.11.50 --instruction "WordPress site, focus on plugin vulnerabilities"
# Penetration-test mode (asset discovery → vulnerabilities → report)
pentestgpt --target 10.10.11.234 --mode pentest
# List previously saved sessions
pentestgpt --list-sessions
该代理通过多阶段流水线运行，将每个阶段的发现结果传递到下一阶段——CTF场景下为侦察→漏洞利用→通关指南，渗透测试场景下为资产发现→漏洞识别→报告生成。
在Docker中运行（安装一次，登录一次）
一个独立的镜像捆绑了工具 + Claude Code 和 Codex 命令行界面。您只需登录一次，会话便会保存在命名卷中——后续运行无需重新登录。
make docker-build # build the tool image
make docker-login # ONE-TIME, idempotent: checks logins, logs in only what's missing
make docker-auth-status # check both are logged in (ROUNDTRIP=1 for a live 1-token check)
# Run the pipeline against a target (any backend / model / mode):
make docker-run TARGET=http://127.0.0.1:8000 BACKEND=codex MODEL=gpt-5.5 MODE=ctf
make docker-run TARGET=10.10.11.234 BACKEND=claude MODEL=opus MODE=pentest
make docker-login 用于登录 Claude（通过 setup-token 将令牌存储在卷中）和 Codex（通过容器内自带的 codex login 命令，OAuth 回调通过 socat 转发——由于 ChatGPT 刷新令牌为一次性使用，因此不进行预配置）。该命令具有幂等性：重新运行时会跳过仍然有效的登录步骤。登录状态在容器重建后仍然保留；make docker-down 会保留登录状态，make docker-nuke 则会删除登录卷（以强制重新登录或轮换令牌）。设计与详细信息：docs/docker-dev-plan.md。
交互式多LLM模式（现代化的旧版本）
USENIX 2024 论文中经典的、人机协作的 PentestGPT 被保留下来，并作为 pentestgpt-legacy 进行了现代化处理。它运行三个协作的 LLM 会话——推理/生成/解析——在您通过交互方式（next、more、todo、discuss）驱动会话时，维护一个渗透测试任务树（PTT）。自主固定阶段的流水线支持 Claude 和 Codex 后端；此旧版本模式通过官方 SDK 直接与许多提供商进行原生通信。
配置提供商
为您想要使用的任何提供商设置 API 密钥（在您的环境变量或 .env 文件中——参见 .env.example）。只有您配置的提供商会被启用。
OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GEMINI_API_KEY=... # or GOOGLE_API_KEY
DEEPSEEK_API_KEY=... GROK_API_KEY=... QWEN_API_KEY=... KIMI_API_KEY=...
运行
# Auto-pick the best available models for each session
pentestgpt-legacy
# Choose models per session
pentestgpt-legacy --reasoning-model claude-opus-4-8 --parsing-model gemini-3.5-flash
# Local model via Ollama (OpenAI-compatible)
pentestgpt-legacy --reasoning-model ollama:qwen3 --base-url http://localhost:11434/v1
# List every supported model (shows which providers are configured)
pentestgpt-legacy --list-models
# Live round-trip every configured model and print a pass/fail matrix
pentestgpt-legacy --smoke-test
支持的模型（2026年6月网络验证）
pentestgpt-legacy --list-models 始终会显示最新的注册表。当模型ID发生变化后，请重新运行 --smoke-test。当前快照：
| 提供商 | 当前模型 | 保留的旧模型 | 环境变量键 |
|---|---|---|---|
| OpenAI | gpt-5.5、gpt-5.5-pro、gpt-5.4-mini、gpt-5.4-nano、gpt-5.2、gpt-5.3-codex |
gpt-4o、gpt-4o-mini、o3、o4-mini |
OPENAI_API_KEY |
| Anthropic | claude-opus-4-8、claude-sonnet-4-6、claude-haiku-4-5-20251001 |
— | ANTHROPIC_API_KEY |
| Google Gemini | gemini-3.1-pro、gemini-3.5-flash、gemini-3-pro、gemini-3.1-flash-lite |
gemini-2.5-pro、gemini-2.5-flash |
GEMINI_API_KEY / GOOGLE_API_KEY |
| DeepSeek | deepseek-v4-flash、deepseek-v4-pro |
deepseek-chat、deepseek-reasoner |
DEEPSEEK_API_KEY |
| xAI Grok | grok-4.3 |
— | GROK_API_KEY / XAI_API_KEY |
| Alibaba Qwen | qwen3.7-max、qwen3.5-flash |
qwen3-max |
QWEN_API_KEY / DASHSCOPE_API_KEY |
| Moonshot Kimi | kimi-k2.6 |
— | KIMI_API_KEY（默认 .cn；若使用 .ai 请设置 MOONSHOT_BASE_URL） |
| 本地（Ollama） | ollama:<model>（例如 ollama:qwen3） |
— | 无（OLLAMA_BASE_URL） |
注册表位于
pentestgpt_legacy/llm/registry.py（唯一的事实来源）。添加模型只需一个ModelSpec条目；与 OpenAI 兼容的提供商可复用一个连接器。
遥测
PentestGPT 会收集匿名使用数据以帮助改进工具。这些数据会发送至我们的 Langfuse 项目，包括：
- 会话元数据（目标类型、持续时间、完成状态）
- 工具执行模式（使用了哪些工具，而非实际命令）
- 标志检测事件（仅记录发现标志这一事件，不包含标志内容）
不会收集任何敏感数据——命令输出、凭据或实际标志值绝不会被传输。
选择退出
# Via command line flag
pentestgpt --target 10.10.11.234 --no-telemetry
# Via environment variable
export LANGFUSE_ENABLED=false
基准测试历史
在2025年12月的XBOW验证套件实验中，PentestGPT取得了86.5%的成功率（104项基准测试中完成90项）。该数据为历史研究结果，不构成当前pentestgpt-agent的回归保证。
XBOW测试工具和结果档案作为仅供参考的研究工件，维护在本产品仓库之外。受支持的PentestGPT CLI、Makefile、CI和Docker运行时不提供XBOW运行器。未来的评估可能会通过独立所有的适配器复用该语料库，但不会将其作为产品依赖项。
引用
如果您在研究中使用了PentestGPT，请引用我们的论文：
@inproceedings{299699,
author = {Gelei Deng and Yi Liu and Víctor Mayoral-Vilches and Peng Liu and Yuekang Li and Yuan Xu and Tianwei Zhang and Yang Liu and Martin Pinzger and Stefan Rass},
title = {{PentestGPT}: Evaluating and Harnessing Large Language Models for Automated Penetration Testing},
booktitle = {33rd USENIX Security Symposium (USENIX Security 24)},
year = {2024},
isbn = {978-1-939133-44-1},
address = {Philadelphia, PA},
pages = {847--864},
url = {https://www.usenix.org/conference/usenixsecurity24/presentation/deng},
publisher = {USENIX Association},
month = aug
}
许可协议
本项目基于 MIT 许可协议进行分发。详情请参见 LICENSE.md。
免责声明：本工具仅用于教育目的和获得授权的安全测试。作者不鼓励任何非法使用行为。使用本工具的风险由使用者自行承担。
致谢
- 本研究得到 Quantstamp 和 新加坡南洋理工大学 的支持
(返回顶部)