canary

GGitHub✨ feat: per-call llm_generation_tracing observability (#15124)

cce14911创建于 19 小时前历史提交

文件	最后提交记录	最后更新时间
benchmarks	✨ feat(desktop): implement subscription pages embedding with webview (#12114) * ✨ feat(desktop): implement subscription pages embedding with webview - Add SubscriptionIframeWrapper component for embedding subscription pages in Electron webview - Replace compile-time ENABLE_BUSINESS_FEATURES with runtime serverConfigSelectors - Add useIsCloudActive hook to detect cloud sync status - Add setupSubscriptionWebviewSession service for webview session management Resolves LOBE-4589 * ✨ feat: enhance SubscriptionIframeWrapper for improved configuration - Updated iframe URL to use OFFICIAL_URL for better maintainability. - Integrated server configuration to conditionally enable business features. - Cleaned up code by removing commented-out lines and unnecessary eslint suppressions. This update improves the flexibility and readability of the SubscriptionIframeWrapper component. Signed-off-by: Innei <tukon479@gmail.com> * ✨ feat: enhance link interception in SubscriptionIframeWrapper - Updated the script to intercept both link clicks and window.open calls for better URL handling. - Added functionality to resolve relative URLs to absolute before logging. This improvement enhances the tracking of external link interactions within the iframe. Signed-off-by: Innei <tukon479@gmail.com> * 🔧 chore: clean up useIsCloudActive hook - Removed unnecessary eslint suppressions related to react-hooks rules. - Improved code readability by eliminating commented-out lines. This update enhances the clarity and maintainability of the useIsCloudActive hook. Signed-off-by: Innei <tukon479@gmail.com> --------- Signed-off-by: Innei <tukon479@gmail.com>	3 个月前
src	✨ feat: per-call llm_generation_tracing observability (#15124) * ✨ feat(database): add llm_generation_tracing schema + tracing package (LOBE-9462) Foundation layer for per-call observability of `generateObject` calls. - New Drizzle table `llm_generation_tracing` with identity / context / model / result / usage / storage / feedback / audit columns and full single-column index coverage (Postgres bitmap-scan friendly). Migration 0103 is idempotent (CREATE TABLE/INDEX IF NOT EXISTS) for safe re-runs. - `LlmGenerationTracingModel` with `record` / `updateFeedback` / `findById` / `listRecent`, all userId-scoped to prevent cross-user leaks. - New package `@lobechat/llm-generation-tracing` mirroring agent-tracing's shape: `ITracingStore` interface, `FileTracingStore` (local/dev, scenario subfolders + latest.json symlink), `computePromptHash` (6-char sha256 of systemPrompt + schema), and `TRACING_SCENARIO_REGISTRY` + `resolveScenario` with explicit scenario override. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(model-runtime): wire llm_generation_tracing into ModelRuntime.generateObject (LOBE-9462) Per-call interception layer — one hook covers all generateObject callers. - New `onGenerateObjectComplete` hook on `ModelRuntimeHooks`: always fires (success or failure) with latency, usage, output/error. Fixes the gap where `onGenerateObjectFinal` only fires when the runtime invokes `onUsage`. - `S3TracingStore` (zstd level 3, key `llm-generation-tracing/{scenario}/{v}-{hash}/{date}/{id}.json.zst`) and `LLMGenerationTracingService` that does DB insert → store.save → patch storage_key. Store failures preserve the row with `metadata.store_error`. - `createLLMGenerationTracingHook` + `mergeModelRuntimeHooks` wired into `initModelRuntimeFromDB`; tracing runs alongside business (billing) hooks via `next/server.after()` when available, microtask fallback otherwise. Unknown metadata keys (e.g. `parent_memory_trace_key`) pass through. - Memory extractor accepts `parentMemoryTraceKey` option for the job-level backlink. Follow-up-action caller given an explicit `scenario: 'follow_up'` metadata override — it was the only OSS caller missing trigger metadata. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✅ test(llm-generation-tracing): type vi.fn mocks so tsgo accepts mock.calls indexing The hook + service tests destructured `mock.calls[0][0]` and accessed nested fields, which tsgo flagged as TS2493 / TS18046 because `vi.fn()` defaults to a zero-arg signature. Add explicit type parameters to the mocks so tsgo can infer the call tuple, and cast `call.payload` at the access point. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ♻️ refactor(model-runtime): move mergeModelRuntimeHooks into the package It's a generic utility for composing `ModelRuntimeHooks` instances — same import surface as `ModelRuntime` and the hooks interface — so it belongs alongside them rather than tucked under a server-side consumer. - New `packages/model-runtime/src/core/mergeHooks.ts` exports `mergeModelRuntimeHooks` and is re-exported from the package index. - Move the unit tests to `packages/model-runtime/src/core/mergeHooks.test.ts`, including a new case covering the "a throws → b is skipped" load-bearing semantics. - `src/server/services/llmGenerationTracing/hook.ts` drops the local copy and the consumer (`src/server/modules/ModelRuntime/index.ts`) imports from `@lobechat/model-runtime`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ♻️ refactor(llm-generation-tracing): version lives with the prompt, not in a central table `promptVersion` was baked into `TRACING_SCENARIO_REGISTRY`, far from any prompt definition — editing a prompt + forgetting to bump the entry in a completely different file was an obvious foot-gun. - Registry is now `Record<string, string>` mapping trigger → scenario only; it's the stable concern that rarely changes. - `resolveScenario` always passes `promptVersion` through from the caller, defaulting to `UNKNOWN_PROMPT_VERSION` ('v0') when absent. - Each call site declares its own `*_PROMPT_VERSION` constant next to the prompt it describes. `followUpAction` ships the first one: `FOLLOW_UP_PROMPT_VERSION` in `prompts/index.ts`, threaded through `metadata.promptVersion` at the `generateObject` call. Other callers can add the same constant when they next touch their prompts. The 6-char prompt hash on the row still catches forgotten bumps. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(input-completion): wire prompt-version metadata at the auto-complete call site Aligns input auto-complete with the FOLLOW_UP_PROMPT_VERSION convention so each prompt iteration is recordable as the chat-side tracing lands. - `INPUT_COMPLETION_PROMPT_VERSION = 'v1.0'` declared next to `chainInputCompletion` — bump together with the prompt body. - `fetchPresetTaskResult` accepts optional `metadata` and forwards it to `getChatCompletion`; the existing chat path already plumbs metadata to `ModelRuntime.chat` options. - `InputEditor` call site passes `{ scenario: 'input_completion', promptVersion }`. Note: `llm_generation_tracing` currently only fires from `onGenerateObjectComplete`. Input completion is a `chat` call, so this metadata is forward-looking until a chat-side tracing hook lands. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * 🐛 fix(llm-generation-tracing): collapse bucketDir path.join args to silence turbopack glob warning Turbopack's static analyzer treats `path.join(root, dyn1, dyn2)` as a multi-segment glob pattern and warned that it could match ~12k files in the project. Compose the relative subdir as a single string first, so `path.join` only sees one dynamic segment. Behavior unchanged — the resulting path is identical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(input-completion): route auto-complete through generateObject for tracing Auto-complete is the first preset-task caller migrated to the structured- output path so it lands in `llm_generation_tracing` via the existing `onGenerateObjectComplete` hook. No new server hook, no global chat-side tracing. - `chainInputCompletion` now returns `{ messages, schema }` with a minimal `{ completion: string }` schema and a stable `INPUT_COMPLETION_SCHEMA_NAME` constant. JSON wrapping costs ~15-30 tokens against a 100-token completion budget — negligible for the observability win. - `StructureOutputSchema` / `StructureOutputParams` accept optional `metadata`; `aiChatRouter.outputJSON` merges caller metadata over the default trigger so `{ scenario, promptVersion, schemaName }` reach `ModelRuntime.generateObject` options unchanged. - `IStructureSchema.description` is now optional to match the zod schema — previously the TS type was stricter than runtime validation accepted. - `InputEditor` switches from `chatService.fetchPresetTaskResult` to `aiChatService.generateJSON`, reading `response.completion`. Streaming is dropped because auto-complete already buffers the full result before inserting; no UX change. - Reverts the unused `metadata` field that was added to `fetchPresetTaskResult` in the previous commit — no current caller needs it now that input completion uses the generateObject path. Bumps `INPUT_COMPLETION_PROMPT_VERSION` to v2.0 because the system prompt gained an "output the completion field" instruction. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ♻️ refactor(aiGeneration): extract the runtime-init + generateObject dance into a service Every server-side caller that produces structured output was repeating the same two-step ritual: `initModelRuntimeFromDB(...)` → `runtime.generateObject(payload, { metadata })`. `AiGenerationService` collapses it into one call so future cross-cutting concerns (default metadata, retry, observability hooks) have one place to land. - New `src/server/services/aiGeneration/index.ts` exposes `generateObject<T>(input, options)` and is unit-tested for provider resolution + payload/metadata pass-through. - `aiChatRouter.outputJSON` and `FollowUpActionService.extract` migrated to the service (other callers move organically when next touched). - Drops the unused `keyVaultsPayload` field from `StructureOutputParams` and the placeholder at the InputEditor call site — key vaults are server-resolved from DB, the client never supplies them. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ♻️ refactor(tracing): centralize TRACING_SCENARIOS const + inject AiGenerationService via trpc ctx - New `packages/const/src/llmGenerationTracing.ts` exports `TRACING_SCENARIOS` + `TracingScenario` type — the single directory where every known scenario name lives. Adds `@lobechat/const` as a workspace dep on llm-generation- tracing so `TRACING_SCENARIO_REGISTRY` can reference the same literals. - Callers (FollowUpActionService, InputEditor) replace `'follow_up'` / `'input_completion'` string literals with `TRACING_SCENARIOS.FollowUp` / `.InputCompletion`, so a typo or a rename fails the type-check instead of silently drifting on the row. - `AiGenerationService` is now injected into the `aiChatProcedure` ctx middleware alongside `aiChatService`; `outputJSON` consumes it via `ctx.aiGenerationService` instead of new-ing it inside the handler. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ✨ feat(llm-generation-tracing): add lt/llm-tracing CLI + drop local-only storage_key - Add `lt` / `llm-tracing` CLI under @lobechat/llm-generation-tracing with `list` (recent records, --scenario filter, --json) and `inspect` (by tracing_id prefix or latest, --full, --json). - `FileTracingStore.save` now returns `{ key: null }` so dev DB rows leave `storage_key` empty instead of recording a non-resolvable local path; S3 store remains the source of truth for the real key. Add helpers `findByTracingId` / `getLatest` used by the CLI. - Wire `agentId` and `topicId` into `input_completion` tracing metadata from the chat input auto-complete call site. - Default `FileTracingStore` whenever NODE_ENV=development (drop the ENABLE_LLM_GENERATION_TRACING_LOCAL opt-in env var). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * 💄 style(llm-generation-tracing): prettier CLI output (tree + colors) Mirror the @lobechat/agent-tracing viewer style: - Inline ANSI color helpers (dim/bold/cyan/magenta/green/yellow/red). - Compact single-line header with id, scenario, version, model, status, time — replaces the multi-line bullet list. - Tree structure with `├─`/`└─` connectors instead of `── section ──` banners. - input arrays render per-message (role + char count + preview) rather than dumping raw JSON. - Small single-key outputs (e.g. `{ completion: "怎么样" }`) collapse to inline `key: "value"`. - `lt list` switches to a colored, properly padded table. Default view stays compact; --full expands system_prompt / input / schema bodies. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ♻️ refactor(llm-generation-tracing): split `tracing` config out of `metadata` `options.metadata` was overloaded — half tracing-specific structured fields (scenario / promptVersion / schemaName / agentId / topicId / ...), half free-form jsonb passthrough. Callers couldn't tell which was which, and the inputHint was always auto-extracted (useless when the prompt wraps the user's text in a template). This commit introduces a dedicated `tracing` option: - Add `TracingOptions` to @lobechat/llm-generation-tracing — the typed shape callers import (agentId / topicId / inputHint / scenario / promptVersion / schemaName / systemPrompt / parentTracingId / metadata). - Add loose `tracing?: Record<string, unknown>` to GenerateObjectOptions and StructureOutputParams / StructureOutputSchema so the field flows through the runtime + TRPC. - Tracing hook now reads `context.options.tracing` for structured fields; it still falls back to `metadata.trigger` for the cross-cutting trigger string (ModelRuntime itself uses metadata.trigger for timing logs, so trigger stays on metadata). - Service `record()` accepts an explicit `inputHint`; otherwise falls back to auto-extraction from the first user message. Always truncated. - Free-form jsonb fields move to `tracing.metadata` (was unknown-key passthrough on `metadata`). - Call sites updated: - FollowUpAction now passes `tracing: { scenario, promptVersion, schemaName, topicId }` (previously `metadata`). - InputCompletion now passes `tracing: { agentId, topicId, inputHint: input, scenario, promptVersion, schemaName }` — `inputHint` is the user's actual typed text, not the wrapper prompt's first user message. - `aiChat.outputJSON` router forwards both metadata and tracing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Update inputCompletion.ts * 🐛 fix(llm-generation-tracing): stop duplicating provider into the row's metadata jsonb `provider` is already a first-class column on the `llm_generation_tracing` row, so auto-stamping it into the `metadata` jsonb column on every call was pure noise. The hook now writes the caller-supplied `tracing.metadata` verbatim — empty/undefined when the caller had nothing to add. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	19 小时前
.gitignore	🔨 chore: update agent coding rule (#10856) update agent coding rule	5 个月前
package.json	♻️ refactor: remove promptfoo configs and dependencies (#13665) ♻️ refactor: remove promptfoo configs and dependencies from packages Migrate all prompt evaluation tests to the cloud repo's agent-evals framework. Remove promptfoo directories, configs, dependencies, and generator scripts from @lobechat/prompts, @lobechat/memory-user-memory, and @lobechat/builtin-tool-memory. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 个月前
vitest.config.ts	♻️ refactor: implement SkillResolver, BaseSystemRoleProvider, and agent document injection pipeline (#13315) * ♻️ refactor: implement SkillResolver to replace ad-hoc skill assembly Introduces a two-layer skill resolution architecture mirroring ToolsEngine + ToolResolver: - SkillEngine (assembly layer): accepts raw skills + enableChecker, outputs OperationSkillSet - SkillResolver (resolution layer): merges operation + step delta + accumulated activations Key changes: - Add SkillResolver, OperationSkillSet, StepSkillDelta, ActivatedStepSkill types - Enhance SkillEngine with enableChecker and generate() method - Wire SkillResolver into RuntimeExecutors call_llm - Replace manual skillMetas assembly in aiAgent with SkillEngine.generate() - Update client-side skillEngineering to use SkillEngine + enableChecker - Add activatedStepSkills to AgentState for step-level skill accumulation Fixes: agent-browser content injected into non-desktop scenarios (Discord bot) due to missing filterBuiltinSkills call in aiAgent LOBE-6410 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: extract agent-templates to standalone package and inject documents server-side - Create @lobechat/agent-templates package with types, templates, and registry - Move DocumentLoadPosition, DocumentLoadFormat, DocumentLoadRule, etc. to new package - Move claw templates (AGENTS, BOOTSTRAP, IDENTITY, SOUL) with .md file imports - Add BOOTSTRAP.md as new onboarding template (priority 1, system-append) - Fix template positions: AGENTS→before-system, IDENTITY/SOUL→system-append - Update database package to re-export from @lobechat/agent-templates - Migrate all consumers to import directly from @lobechat/agent-templates - Add agent documents injection in server-side RuntimeExecutors (was missing) - Support -p CLI flag in devStartupSequence for port configuration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 🐛 fix: correct import statement for non-type exports from agent-templates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 📦 build: add @lobechat/agent-templates to root dependencies Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: remove template proxy files from database package Stop re-exporting template/templates from database — consumers import directly from @lobechat/agent-templates. Keep types.ts re-exports for internal database code only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: create BaseSystemRoleProvider to unify system message append pattern All providers that append to the system message now inherit from BaseSystemRoleProvider and only implement buildSystemRoleContent(). The base class handles find-or-create and join logic. Migrated providers: - EvalContextSystemInjector - BotPlatformContextInjector - SystemDateProvider - ToolSystemRoleProvider - HistorySummaryProvider - SkillContextProvider Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 🐛 fix: restore metadata tracking in BaseSystemRoleProvider via onInjected hook Add onInjected() callback to BaseSystemRoleProvider so subclasses can update pipeline metadata after successful injection. Also add raw-md plugin to context-engine vitest config for .md imports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ✨ feat: add enabled field to AgentDocumentInjector config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: add enabled field to all providers, remove spread conditionals in MessagesEngine All providers now accept an `enabled` config field. MessagesEngine pipeline is a flat array with no spread conditionals — each provider is always instantiated and uses `enabled` to skip internally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 💄 style: clean up MessagesEngine pipeline comments Remove numbered prefixes, keep descriptive comments for each provider. Only phase headers use separator blocks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: reorganize MessagesEngine pipeline phases by injection target Phase 1: History Truncation Phase 2: System Message Assembly (all BaseSystemRoleProvider) Phase 3: Context Injection (before first user message, BaseFirstUserContentProvider) Phase 4: User Message Augmentation (last user message injections) Phase 5: Message Transformation (flatten, template, variables) Phase 6: Content Processing & Cleanup (multimodal, tool calls, cleanup) Moved SkillContext, ToolSystemRole, HistorySummary from Phase 3 to Phase 2 since they append to system message, not user context. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 💄 style: split Phase 6 into Content Processing (6) and Cleanup (7) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: split AgentDocumentInjector into three position-based injectors - AgentDocumentSystemInjector (Phase 2): before-system, system-append, system-replace - AgentDocumentContextInjector (Phase 3): before-first-user - AgentDocumentMessageInjector (Phase 4): after-first-user, context-end Shared utilities (filterByRules, formatDocument, sortByPriority) extracted to AgentDocumentInjector/shared.ts. Old monolithic injector removed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ♻️ refactor: split AgentDocumentSystemInjector into three separate injectors - AgentDocumentBeforeSystemInjector: prepends as separate system message (before-system) - AgentDocumentSystemAppendInjector: appends to system message (system-append) - AgentDocumentSystemReplaceInjector: replaces entire system message (system-replace) Each has distinct semantics and correct pipeline placement: - BeforeSystem → before SystemRoleInjector - SystemAppend → after HistorySummary (end of Phase 2) - SystemReplace → last in Phase 2 (destructive) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ✨ feat: auto-enable agent-documents tool when agent has documents - Add AgentDocumentsManifest to defaultToolIds - Add hasAgentDocuments rule in server createServerAgentToolsEngine - Query agent documents in AiAgentService.execAgent to determine flag - Pattern matches KnowledgeBase auto-enable via enableChecker rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * 🔨 chore: add agent documents status to execAgent operation log Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * update content * fix tests * 🐛 fix: add raw-md plugin to database vitest configs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	1 个月前