| ✨ Remove mem0 and support new memory architecture (#3497) * ✨ feat(context): Add fine-grained context management infrastructure (PR-0) Add foundational context management module under sdk/nexent/core/agents/context/ to support W8/W12/W13 workstreams for progressive component reduction, history projections, and unified context policy. Core components: - ContextItem: Fine-grained context unit with type, authority tier, and fidelity levels - ContextItemHandler: Pluggable handler interface for scoring and reducing context items - ItemHandlerRegistry: Registry mapping context item types to their handlers - 10 built-in handlers (system_prompt, tool, skill, memory, knowledge_base, etc.) - ContextProjector: Converts ContextComponent to ContextItem with proper authority/fidelity - Policy models: SelectionDecision, MemoryDecision for traceable policy decisions - ReductionResult: Immutable record of context reduction operations - Reason codes: 11 standardized codes for decision traceability Testing: - 77 unit tests covering all data models, handlers, and registry - All tests pass with 0.59s execution time This is infrastructure-only (no existing code modified) and provides the foundation for subsequent PRs implementing context projection, policy engines, and handlers. * feat(context): integrate ContextItem projection into ContextManager (PR-1) - Add use_context_items config field (default False for backward compatibility) - Add context_items field to ContextEvidence for traceability - Implement project_context_items() method in ContextManager - Integrate projection logic into assemble_final_context() - Store _source_component reference in metadata for semantic equivalence - Use component.to_messages() to produce formatted text (not JSON dumps) - Add 15 projection tests covering all 7 component types - Add 3 ManagedContextRuntime integration tests - Remove test __init__.py files to fix namespace collision All 112 tests pass. Oracle verified complete and correct. * feat(context): add DB history projection for ReAct process persistence (PR-2) Implement W12 DB History Projection to persist and reconstruct ReAct execution details from conversation history. Database layer: - Add 5 nullable columns: run_id, step_id, tool_call_id, event_time on conversation_message_t and conversation_message_unit_t - Add indexes on run_id and tool_call_id for query performance - Extend conversation_db.py with optional history projection params - Add get_message_units_by_run() and get_max_run_id_for_conversation() SDK layer: - Create HistoryProjector with dependency injection for DB queries - Support 3 projection modes: model_context, resume, chat - Produce HISTORY_TURN, TOOL_CALL_RESULT, WORKING_MEMORY ContextItems - Wire into ContextManager.use_context_items=True path via config Service layer: - Track run_id/step_id/tool_call_id/event_time in _stream_agent_chunks() - Pass tracking fields through save_message/save_message_unit wrappers - Compute run_id before message creation for proper persistence Tests: 21 new tests, 114 total context tests passing, zero regressions. * feat(context): close PR-2 acceptance gaps — production wiring + integration tests Production wiring: - Thread conversation_id through execution chain: agent_service → create_agent_run_info → AgentConfig → ManagedContextRuntime → assemble_final_context → HistoryProjector - Inject HistoryProjector into ContextManagerConfig before get_or_create_context_manager() so both run-scoped and conversation-scoped ContextManager paths receive it - use_context_items remains False by default (opt-in) Integration tests (7 new, 28 total): - TestChatProjectionCompleteness: unit type coverage, ordering, metadata, source_refs completeness - TestEndToEndIntegration: component + history projection → FinalContext, graceful failure handling, conversation_id gating 121 context tests + 28 HistoryProjector tests passing, zero regressions. * fix(context): resolve 3 activation blockers for use_context_items pipeline Blocker 0: Call register_all() in ContextManager.__init__ so ItemHandlerRegistry is populated before any projection occurs. Blocker 1: Add to_messages() method to ContextItemHandler base class with default implementation, plus type-specific overrides in HistoryTurnHandler, ToolCallResultHandler, and WorkingMemoryHandler. Blocker 2: Remove mock of ItemHandlerRegistry.get in integration test so the end-to-end path exercises real handlers. 198 tests passing (121 context agent + 77 context module), zero regressions. * fix(context): move register_all() to lazy initialization to avoid import regression Move register_all() call from ContextManager.__init__ to lazy initialization in project_context_items() to avoid breaking isolated test loading. Changes: - Remove register_all() from __init__ (line 301-302) - Add _ensure_handlers_registered() helper with idempotency flag - Call helper at start of project_context_items() - Handlers now register only when actually needed This fixes the 124-test regression caused by eager import in __init__ breaking test_agent_context/test_pure_functions.py isolated loading. All 254 context tests now pass (121 context + 77 module + 56 agent_context). * feat(context): add OpenTelemetry instrumentation to context module Add comprehensive OTel tracing to provide visibility into the context assembly pipeline: - ManagedContextRuntime: trace prepare_step() and prepare_final_answer() - ContextManager: trace assemble_final_context(), project_context_items(), compress_if_needed(), and _do_generate_summary() with nested LLM spans - HistoryProjector: trace project() with nested DB query spans All spans include relevant attributes (conversation_id, purpose, counts) and span events for key milestones. Backward compatible - spans are no-ops when monitoring is disabled. Testing: All 254 existing tests pass, no syntax errors. * fix(context): correct compress_if_needed span scope and improve project_items attributes Critical fix: - Move 'with self._lock:' block inside trace_operation context manager - Ensures all compression work (including LLM calls) is properly traced - Fixes broken parent-child span hierarchy for compression path Minor improvements: - Add component_count as span attribute to project_context_items - Remove unused 'as span' from assemble_final_context All 254 tests pass. * fix(context): use ContextManager uncompressed baseline for save% metric Previously _last_uncompressed_est was set from input_messages which are already compressed by prepare_step, making est_raw_i always equal est_i and save% structurally ~0%. Now pull the truly-uncompressed token count from ContextManager.get_token_counts()['last_uncompressed'], recorded in compress_if_needed from the raw memory before compression. Co-Authored-By: Claude <noreply@anthropic.com> * chore: expose temp_scripts test harness for cross-machine sync Add test scripts and fixtures used during context-manager refactor development. Only .py and .md files are tracked; .out run artifacts remain ignored via .git/info/exclude. Co-Authored-By: Claude <noreply@anthropic.com> * chore: add .gitignore in temp_scripts to suppress .out/.log artifacts Co-Authored-By: Claude <noreply@anthropic.com> * chore: allow common ops by default, keep rm/mv/git push as ask Co-Authored-By: Claude <noreply@anthropic.com> * refactor: extract SummaryTaskStep and ManagedRunContext into summary_step.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract pure budget/cache/fingerprint helpers into budget.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract LLMSummary class into llm_summary.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract StepRenderer and compress_history_offline into step_renderer.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract PreviousCompressor into previous_compression.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract CurrentCompressor into current_compression.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract compression stats functions into stats_export.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract ContextManager orchestrator into manager.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: correct relative import depth for utils.token_estimation The token_estimation module lives at core/utils/, not agents/utils/. From agent_context/ sub-package, three dots (...utils) are needed instead of two (..utils). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: wire agent_context package with __init__.py re-exports and remove monolith Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: update test loader for agent_context package structure Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: update tests to use standalone functions from decomposed agent_context package Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: align extra compression tests with v2 compressor behavior The v2 PreviousCompressor and CurrentCompressor do not fall through from incremental to fresh when LLM returns None - they return PreviousCompressResult/CurrentCompressResult(summary_text=None) immediately. Updated tests P3, C4 to match this behavior. Updated P4 and C6_asymmetry to patch _summarize_pairs with PreviousCompressResult instead of raw tuples. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: correct relative import depth for context_runtime.contracts The context_runtime package lives at core/context_runtime/, not agents/context_runtime/. From agent_context/ sub-package, three dots (...context_runtime) are needed instead of two (..context_runtime). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * chore: untrack temp_scripts and settings.local before PR Co-Authored-By: Claude <noreply@anthropic.com> * chore: restore .claude/settings.local.json to match upstream Co-Authored-By: Claude <noreply@anthropic.com> * test: add coverage for stats_export, step_renderer, and budget modules - New test_stats_export.py: 21 tests covering all pure functions (100%) - New test_step_renderer.py: 24 tests covering truncation, rendering, compress_history_offline with mock LLM (49%→86%) - Extended test_pure_functions.py: tests for _is_context_length_error, has_invoked_tools, message_role, trim_pairs_to_budget (92%→97%) Overall agent_context package coverage: 71% → 81% Co-Authored-By: Claude <noreply@anthropic.com> * test: add llm_summary error-handling and output-format coverage (72%→100%) Co-Authored-By: Claude <noreply@anthropic.com> * test: cover _step_stream uncompressed estimation path in core_agent Adds two tests: - test_step_stream_uses_context_manager_for_uncompressed_est: verifies _last_uncompressed_est is pulled from ContextManager.get_token_counts() - test_step_stream_falls_back_without_context_manager: verifies fallback to msg_token_count when context_manager is None Co-Authored-By: Claude <noreply@anthropic.com> * test: add fingerprint, change detection, and manager utility tests Cover the largest untested blocks in manager.py: - _normalize_for_fingerprint, _fingerprint - _change_reasons, _stable_component_fingerprints - _purpose_messages, _messages_from_memory - _without_leading_stable_messages, _canonical_tools - _estimate_tools_tokens, build_compressed_snapshot - init keep_recent_steps cap, token estimation delegates Manager.py coverage: 69% → 91%, overall: 82% → 92% Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * chore: fix SonarCloud issues in test and llm_summary files - test_pure_functions.py: use ValueError instead of generic Exception (S112) - test_step_renderer.py: remove separator comments flagged as code (S125) - llm_summary.py: use logger.exception() in except blocks (S8572) - test_manager_fingerprint.py: replace unused variable with _ (S1481) Co-Authored-By: Claude <noreply@anthropic.com> * fix(context): resolve strategy filtering bypass in use_context_items path Critical bug fix: When use_context_items=True, assemble_final_context() was passing ALL registered components to project_context_items() instead of only strategy-selected components, causing token budget overflow and behavioral divergence from use_context_items=False path. Changes: - Add selected_components field to ManagedRunContext to track filtered subset - Modify prepare_run_context() to explicitly call strategy.select_components() and store both all components and selected components - Fix assemble_final_context() to use run_context.selected_components - Fix _stable_component_fingerprints to use filtered components - Propagate selected_components through rebuilt ManagedRunContext Additional improvements: - Add OpenInference input/output value recording to all context module spans - Fix trace_operation() to process input.value through payload preview - Fix history_projector output scope to set on correct span Verification: - 5 new strategy filtering tests (all pass) - 5 PR-0/1/2 comprehensive tests (all pass) - 87 existing tests (all pass) - Verified behavioral equivalence between use_context_items=True and False - Verified OTel traces show correct input/output in Langfuse Production ready: backward compatible, no breaking changes * refactor: reduce duplication in step_renderer and fix Sonar warnings in compression and test files - Extract _build_offline_user_prompt and _call_model_for_summary helpers from compress_history_offline - Rename unused cache parameter to _cache with alias in current/previous compression (Sonar S1172) - Replace unused idx variables with _ in test_cache_valid (Sonar S1481) - Rename test methods to lowercase convention in test_compress_with_cache_extra (Sonar S100) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * chore: clean up root-level test files Moved to test/sdk/core/agents/: - test_pr0_1_2_comprehensive.py → test_context_pr0_1_2.py - test_strategy_filtering_fix.py → test_strategy_filtering.py Deleted (redundant or not suitable for CI): - test_otel_smoke.py (manual smoke test, not a proper unit test) - test_budget_equivalence.py (covered by test_strategy_filtering.py) - test_assemble_equivalence.py (covered by test_strategy_filtering.py) - test_agent_context_items.py (integration test requiring API keys) * refactor: remove PR references from test names - test_pr0_handlers -> test_handlers - test_pr0_projector -> test_projector - test_pr1_equivalence -> test_equivalence - test_pr2_history_projector -> test_history_projector - Updated all test output labels to remove PR-0/1/2 prefixes * fix(tests): update test assertions to match new API signatures - Add conversation_id parameter to create_agent_config and create_agent_run_info assertions - Add run_id, step_id, tool_call_id, event_time parameters to create_message_unit assertions - Update fake_save_message to accept **kwargs for additional parameters - All tests now pass individually (177 + 327 + 68 = 572 tests) - Note: test__stream_agent_chunks_captures_final_answer_and_adds_memory has a pre-existing issue with background task execution * fix(tests): add missing ProcessType attributes to MockProcessType The test was failing because MockProcessType was missing STEP_COUNT, TOOL, and EXECUTION_LOGS attributes that are used in agent_service.py. This caused an exception in the chunk processing loop that prevented captured_final_answer from being set, which in turn prevented the background memory task from running. Added the missing attributes to MockProcessType and added a small sleep to ensure the background task has time to complete before assertions. * fix: handle None history in agent request Fixed TypeError when agent_request.history is explicitly None instead of absent. Changed getattr(agent_request, 'history', []) to handle None case properly. This bug prevented message persistence when is_debug=false, causing all event log fields (run_id, step_id, tool_call_id, event_time) to remain NULL. * test: add unit tests for conversation_db new functions and fix SonarCloud floating point comparison - Add 5 tests for get_message_units_by_run (with/without run_id, empty result, string coercion, dict mapping) - Add 4 tests for get_max_run_id_for_conversation (found, none, string coercion, zero edge case) - Update stubs to include run_id and step_id attributes - Fix floating point comparison in test_handlers.py using pytest.approx (SonarCloud S1244) This improves patch coverage for conversation_db.py and fixes SonarCloud reliability rating. * test: add SDK integration test with real model and LangFuse tracing Add comprehensive integration test that verifies: - Real LLM model execution through the SDK - OpenTelemetry instrumentation captures spans correctly - Context management with use_context_items=True works end-to-end - LangFuse receives and displays traces properly Test includes: - test_context_items_with_real_model: Basic context management with real model - test_context_compression_with_real_model: Compression with low token threshold - test_history_projector_integration: History projection with real model Requires OPENAI_API_KEY environment variable to run. * test: fix SDK integration tests and mark as local_only - Fix OpenAIModel initialization (add model_id parameter) - Fix ActionStep initialization (use correct parameters) - Fix HistoryProjector initialization (use query_units_fn) - Fix ManagedContextRuntime.prepare_step call (remove conversation_id) - Add local_only marker to pytest.ini - Mark all SDK integration tests as local_only to skip in CI These tests require OPENAI_API_KEY and network access, so they should only run locally, not in CI environments. * chore: remove temporary dev plan from git tracking - Remove CONTEXT_MANAGEMENT_DEV_PLAN.md from git (keep locally) - Add to .gitignore to prevent future commits This is a temporary development document that should not be tracked in version control. * docs: add spec coding workflow * refactor: merge tool_call rows into JSON, remove run_id/event_time, remove WorkingMemoryHandler - Merge tool+execution_logs into single tool_call row with JSON unit_content - Remove run_id from conversation_message_t and conversation_message_unit_t - Remove tool_call_id and event_time from conversation_message_unit_t - Rename step_id to step_index for clarity - Replace run_id grouping with message_id-based algorithm in HistoryProjector - Remove WorkingMemoryHandler and WORKING_MEMORY context item type - Remove resume projection from HistoryProjector - Remove dead components field from ManagedRunContext - Fix parameter ordering in create_agent_run_info() - Add TOOL_CALL handling in frontend for both streaming and history - Rewrite migration to be idempotent with proper DROP/ADD COLUMN - Update all affected tests (backend + SDK) * test: add tool_call merge tests and fix SonarCloud type mismatch - Add 3 tests for tool_call merge logic in _stream_agent_chunks - Fix string-to-int type mismatch in test_conversation_db.py * chore: add spec-coding skill, update AGENTS.md and benchmark scripts * chore: add .agents file and update .gitignore to exclude it * Remove old doc * ♻️ Clean up lagecy implementation of mem0 memory * Merge origin/develop into dev/feature/memory_0709 (#3412) Update feature branch with mainstream develop branch. * 📃 Update spec-coding skills * ✨ [WIP] Memory implementation Phase 1 * ✨ [WIP] Memory implementation Phase 2 * ✨ [WIP] Memory implementation Phase 4 * ✨ Memory settings and management frontend * ✨ Memory settings and management frontend * ✨ Support memory management and memory tool calling 🐛 Bugfix: add pdfinfo in data-process image to resovle files with images * 🧪 Add test files * 🧪 Add test files ♻️ Change memory seperate key --------- Co-authored-by: Jinglong Wang <jasonwong2019@outlook.com> Co-authored-by: liudongfei <744532452@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: liudongfei <liudongfei4@huawei.com> Co-authored-by: Jason Wang <56037774+JasonW404@users.noreply.github.com> Co-authored-by: Nexent Agent <agent@nexent.local> | 1 个月前 |