| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Add new logo | 1 年前 | |
Update iOS launch icon | 1 年前 | |
fix(app): disable silence pause under INV-CAP-1 and cut capture-path CPU work (#20882) * fix(app): preserve pendant audio while reducing capture CPU work Failure-Class: new Disable silence-driven unsubscribe on current firmware, recover historical automatic pauses, and reduce redundant policy decoding, payload copies, and WAL serialization allocations. Preserve INV-CAP-1, reconnect timing, and durable write cadence. * refactor(capture): keep helpers bounded and enforce retention guards Move restore effects into the existing uplink extension and extract unchanged phone WAL serialization. Wait for observed sync-drop reconciliation in legacy-state tests and wire INV-CAP-1 path citations to the continued-delivery guards. * docs(app): record battery preservation design and verification | 6 天前 | |
feat(app): pendant audio-loss integrity — unresolved outages + per-day provenance (#21151) * feat(app): credible audio-loss accounting — persist unresolved outages, per-day provenance * fix(app): fence durable audio evidence and reconcile daily outage exposure * fix(app): rekey OmiDeviceHealthStore pbxproj entries to unique UUIDs * test(app): gate depends_on_21152 delivery tests out of shared CI runs * fix(app): buffer BLE outage writes when SharedPreferences binding is absent * test(app): enable depends_on_21152 delivery tests after #21152 merge | 2 天前 | |
fix(app): verify pendant capture from arriving audio and recover a silent link (#20378) * fix(app): verify pendant capture from arriving audio and recover a silent link A pendant could stay Bluetooth-connected for hours while no audio reached the app: a restored link was treated as capturing, nothing detected zero ingress, and nothing forced a fresh connection. Capture health on iOS now starts unverified on a restored or replayed link and is established only by a confirmed audio subscription plus audio on the current connection. When evidence is absent or stops, native code runs one persisted-budget recovery: repair the subscription, then a real disconnect and reconnect. The Listening claim and timer follow that evidence; the card, transcript and controls stay. Subscription results are reported to Dart on both platforms, and Recording Requested keeps the telemetry denominator now that Recording Started waits for audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(app): run native capture recovery tests in CI and tighten review-flagged tests Register the capture health and recovery transaction harnesses in the checks manifest so CI runs them, and let the BLE policy test run on its own. Make four tests assert what their names claim, keep the subscription-failure event in the scorecard export, and word the changelog as an attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(app): bound Android CCCD waits and harden the recovery reconnect path Address the remaining review findings on the pendant capture link: - Bound the accepted CCCD write with a 15s native timeout mirroring iOS OmiCaptureHealth.Policy.operationTimeout, so a missing onDescriptorWrite fails the pending completion and releases the GATT queue instead of blocking Dart's retry behind it forever. The pending map entry is the ownership token: a late ack after timeout or cleanup settles nothing and cannot double-release a later command's slot. - unsubscribeCharacteristic gets the same sentinel, timeout, and try/catch treatment (same unbounded-wait defect class). - cleanupPeripheral now fails and removes pending CCCD completions and their timeout callbacks for the torn-down peripheral. - subscribeCharacteristic never throws synchronously (pre-check and queued body wrapped); the generated Pigeon handler cannot strand Dart without a reply. BleHostApiImpl adds the same guard. - Recovery resume goes through existing.connect() instead of the bare transport: transport failures surface as DeviceConnectionException like the cold-connect path, and the pendant time sync runs after the link returns. The service state callback is re-passed so subscribers keep receiving state changes. - Recheck native ingress ownership after the feature-gate await so a legacy episode cannot land invisibly and block later declarations. - _resolveIfActive on ingress-failed episodes releases the telemetry slot while retaining the native-owned ingress failure. - The verified ingress start survives a lease expiry (only withdrawn proof clears it) so the capture timer continues instead of restarting at zero when audio returns. * fix(app): isolate CCCD write ownership and retire timed-out GATTs Serialize descriptor settlement and queue callbacks on the main handler. Arm each write token before issuing its CCCD operation, and release only the current command owner. Retire a GATT with a missing ACK, fail its queued subscription replies, and hand reconnection to the existing managed service so late ENABLE or DISABLE callbacks cannot settle a newer write. Add JVM regressions for late ACKs, stale timers, ENABLE/DISABLE ownership, retired connections, queue-owner changes, rejected writes, and cleanup. * fix(app): bound reconnects after CCCD timeout retirement Distinguish missing CCCD acknowledgements from ordinary GATT disconnects. Persist a per-device budget for two reconnects with backoff, and reset it only after a current successful descriptor ACK. Stop companion and sticky restoration from bypassing exhaustion. Project the exhausted result through existing capture health without a competing Dart retry. Preserve capture intent, replay stored failures when capture binds, and keep healthy Android recording telemetry unchanged. Cover the policy, ACK ownership, and recovery presentation with production-seam tests. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: David Zhang <9387252+Git-on-my-level@users.noreply.github.com> | 10 天前 | |
feat(app): show the capture Live Activity only when capture needs you (#20957) * feat(app): show the capture Live Activity only when capture needs you Healthy pendant capture is Omi's default, so the card and Dynamic Island no longer restate it with a pendant and a running clock. A card appears only when the user is paused, the pendant is still unheard two minutes after connecting, or its battery is at 15% or less; a phone recording the user started keeps the recording card. Compact trailing views fall back to a dot in landscape instead of clipping the clock to "2:1". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(ci): allowlist the sign-language files #20946 left unreachable main has failed the dead-code ratchet since #20946: SignVideo.tsx, signAssets.ts and signPoses.ts have no importer, so every push is rejected. Allowlist them with a reason so the owner can wire them up or delete them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(app): keep the Omi pendant in the island and close review gaps Compact notices show the pendant and a tinted status glyph, with values only in the expanded island and Lock Screen. A tapped Start no longer loses its card before the intent replies. The battery notice is limited to Omi pendants and goes stale like the other cards. Swiped-away notice kinds are remembered as a set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "chore(ci): allowlist the sign-language files #20946 left unreachable" #20958 fixed the dead code on main, so this PR no longer needs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> | 5 天前 | |
chore(app): annotate vendored decoder note | 16 天前 | |
fix(ios): type-safe native sinks and a SwiftLint safety ratchet (#19586) * fix(ios): add typed Foundation sink boundaries Guard defaults, JSON serialization, and floating-point conversion against type-erased native traps. Prove the boundaries with hostile Swift harness inputs and representative Runner callers. * ci(ios): ratchet native Swift safety lint Pin SwiftLint through the verified desktop binary, baseline existing violations, reject baseline growth, and run macOS-scoped manifest checks in the hosted iOS compile job. * docs(app): note iPhone diagnostics reliability improvement * ci(ios): catch main sync call syntax in SwiftLint * test(ios): scope legacy plist scan to handwritten Swift CocoaPods places third-party Swift under app/ios/Pods; retain the Optional-to-Any guard without treating installed dependencies as first-party source. * fix(ios): guard high-risk native crash paths * refactor(ios): route defaults and numeric sinks safely * fix(ios): guard missing watch audio start time * chore(ios): shrink native safety lint baseline * fix(ios): preserve watch battery value type * fix(ios): type Siri telemetry records explicitly * refactor(ios): centralize epoch millisecond conversion | 14 天前 | |
fix(ios): type-safe native sinks and a SwiftLint safety ratchet (#19586) * fix(ios): add typed Foundation sink boundaries Guard defaults, JSON serialization, and floating-point conversion against type-erased native traps. Prove the boundaries with hostile Swift harness inputs and representative Runner callers. * ci(ios): ratchet native Swift safety lint Pin SwiftLint through the verified desktop binary, baseline existing violations, reject baseline growth, and run macOS-scoped manifest checks in the hosted iOS compile job. * docs(app): note iPhone diagnostics reliability improvement * ci(ios): catch main sync call syntax in SwiftLint * test(ios): scope legacy plist scan to handwritten Swift CocoaPods places third-party Swift under app/ios/Pods; retain the Optional-to-Any guard without treating installed dependencies as first-party source. * fix(ios): guard high-risk native crash paths * refactor(ios): route defaults and numeric sinks safely * fix(ios): guard missing watch audio start time * chore(ios): shrink native safety lint baseline * fix(ios): preserve watch battery value type * fix(ios): type Siri telemetry records explicitly * refactor(ios): centralize epoch millisecond conversion | 14 天前 | |
fix(app): connect Ray-Ban Meta mic by stable iOS UID Ray-Ban audio-only discovery used a renameable Bluetooth port name and a synthetic constant device ID, so real or renamed glasses could be filtered out and could not reconnect through the normal stored-device path. Expose AVAudioSession HFP input UIDs, add an iOS microphone picker, persist the selected UID as BtDevice.id, and require that UID in discovery, reconnect, and capture routing. Keep product-name and EL AI matching only as first-selection conveniences. Add translated picker states for all 48 non-English locales, regression/widget coverage, and connection documentation. Verification: - PATH=/tmp/omi-rayban-flutter.uNe7A0/flutter/bin:$PATH bash app/test.sh (917 tests passed) - PATH=/tmp/omi-rayban-flutter.uNe7A0/flutter/bin:$PATH app/scripts/analyze_ratchet.sh (passed) - Flutter 3.41.9: flutter build ios --debug --no-codesign --flavor dev (Runner.app built) - Signed iPhone launch attempted; blocked by missing team 9536L8KLMP account/profiles before installation Failure-Class: none | 2 个月前 | |
Improve Ask Omi Siri routing and keep spoken answers in Siri (#19946) * fix(app): improve Siri Ask routing and spoken answers * fix(app): address Siri review feedback - AskOmiIntent reports its outcome through performReportingOutcome so SearchOmiIntent telemetry records delegated failures (auth, network, quota, server) instead of always 'ok' - OpenOmiFolderIntent/OpenOmiListIntent classify thrown auth failures via SiriTelemetry.outcome before the deferred telemetry write - Launch telemetry drain waits for the analytics identity bind so native Siri records are not destructively taken into a pre-auth epoch that bindIdentity then discards (P1) - Data & Privacy Siri hint and onboarding guidance move to l10n keys (siriShortcutsSetupHint/siriShortcutsSearchHint); phrases stay literal - Shortcuts hint/button gate to iOS 16 (platform view registration); Search phrase sentence gates to iOS 27 (searchInApp schema) * fix(app): address Siri review feedback round 2 - SearchOmiIntent now emits the askOmi telemetry event around the delegated performReportingOutcome call, which bypassed AskOmiIntent.perform()'s defer and lost the ask event for every search-originated ask. - iOS version gates parse the first integer run of operatingSystemVersion (iOS reports 'Version 18.2 (Build 22C150)'; split('.').first captured 'Version 18' and parsed null, so the Shortcuts settings card and onboarding hint never rendered). Shared PlatformService.isIOSAtLeast helper + tests. - Entity-open intents (conversation, memory, task, donated UI) fence authentication before the Boolean entity check so Firebase-owner failures report auth instead of server, matching the folder/list open pattern. - Truncated spoken answers keep a continuation route: short answers stay spoken-only in Siri, long ones end with 'Open Omi chat for the rest.' (iOS and macOS). * fix(app): translate Siri setup guidance and keep the CTA reachable at large text - siriShortcutsSetupHint and siriShortcutsSearchHint carried raw English copy in 32 non-English catalogs (be bg bn bs ca cs da el et fa fi hr hu kn lt lv mk mr ms no ro sk sl sr sv ta te th tl uk ur vi); supplied translations through scripts/l10n.py set so every catalog renders real localized guidance on onboarding and Data & Privacy. - Onboarding completion content is now scrollable (LayoutBuilder + SingleChildScrollView + ConstrainedBox + IntrinsicHeight) so 200% text with the Siri hint no longer overflows the fixed Column and pushes 'Start Using Omi' off-screen; centered Spacer layout preserved when content fits (ux-contract §16). - OnboardingCompleteScreen.showSiriHint test seam overrides the iOS 16 App Shortcuts gate off-device. - New hermetic widget test pins the CTA reachable (scroll + tap fires onComplete) at 2x text on a 320x568 surface with the hint shown; fails on the pre-fix fixed Column layout. Verification: flutter test test/widgets/onboarding_complete_large_text_test.dart (+2); python3 scripts/l10n.py check (49 locales, 3497 keys); scripts/l10n.py set through the repository tool; targeted fault injection (scroll wrapper removed) fails the new test. * fix(app): fail closed on Darwin version strings in iOS gates Darwin uname strings lead with the kernel major (Darwin 21 is iOS 15, Darwin 24 is iOS 18), so the previous first-integer parse over-reported the marketing version and could pass isIOSAtLeast(16)/isIOSAtLeast(27) on devices that do not satisfy them. Anchor parsing on the leading 'Version N' NSProcessInfo form; Darwin and other unrecognized formats now fail closed to 0, which keeps the Shortcuts and Search hints hidden on hosts whose iOS version cannot be determined. * fix(app): gate the Shortcuts card on native Siri availability Stable-compiler (Xcode 26.6, the required CI pin) builds compile the omi/shortcuts_button registration out of the Runner, while Data & Privacy requested the platform view purely on the runtime iOS version. Siri-capable devices running such a build therefore hit an unregistered platform-view type when opening Data & Privacy. Add a SiriIndexApi.appShortcutsAvailable capability probe: the Siri toolchain bridge answers true; the stable-compiler fallback answers false. Dart fails closed (hidden card) on Android, timeout, or any bridge error, so the card renders only in builds that actually registered the view. Regenerate the Pigeon Dart/Swift surfaces for the new method. * fix(app): keep Ask Omi read-only under the weaker auth policy Remember saves require local-device authentication (contracts/siri), but Ask Omi runs under .requiresAuthentication so a companion device (unlocked Watch, locked phone) can invoke it. The delegated 'remember ...' phrase in Ask therefore persisted a private memory without local-device auth on both iOS and macOS. Ask no longer writes: remember-phrases get a spoken redirect to the strict Remember shortcut, recorded as a cancelled ask. Addresses Codex P2 'Preserve local authentication for Remember delegation' (thread PRRT_kwDOLkKqys6nzHKv). * fix(app): gate onboarding Siri hint on availability * fix(app): regenerate localization output after merge --------- Co-authored-by: David Zhang <9387252+Git-on-my-level@users.noreply.github.com> | 11 天前 | |
fix(ios): clarify Bluetooth purpose strings for App Review 5.1.1(ii) Apple rejected the build under Guideline 5.1.1(ii) because the Bluetooth usage description did not sufficiently explain how the data is used and gave no concrete example. Rewrite NSBluetoothAlwaysUsageDescription and NSBluetoothPeripheralUsageDescription (and the Japanese localization) to state that Bluetooth connects the Omi wearable and streams its microphone audio to the phone, with a specific example (transcribing meetings into notes, checking device battery). Kept short so it renders fully in the iOS permission dialog without truncation. Verification: reviewed rendered strings for length; both are ~180 chars, well within the iOS system alert display limit. No code paths changed. Failure-Class: none | 2 个月前 | |
feat(app): pendant audio-loss integrity — unresolved outages + per-day provenance (#21151) * feat(app): credible audio-loss accounting — persist unresolved outages, per-day provenance * fix(app): fence durable audio evidence and reconcile daily outage exposure * fix(app): rekey OmiDeviceHealthStore pbxproj entries to unique UUIDs * test(app): gate depends_on_21152 delivery tests out of shared CI runs * fix(app): buffer BLE outage writes when SharedPreferences binding is absent * test(app): enable depends_on_21152 delivery tests after #21152 merge | 2 天前 | |
fix(ios): type-safe native sinks and a SwiftLint safety ratchet (#19586) * fix(ios): add typed Foundation sink boundaries Guard defaults, JSON serialization, and floating-point conversion against type-erased native traps. Prove the boundaries with hostile Swift harness inputs and representative Runner callers. * ci(ios): ratchet native Swift safety lint Pin SwiftLint through the verified desktop binary, baseline existing violations, reject baseline growth, and run macOS-scoped manifest checks in the hosted iOS compile job. * docs(app): note iPhone diagnostics reliability improvement * ci(ios): catch main sync call syntax in SwiftLint * test(ios): scope legacy plist scan to handwritten Swift CocoaPods places third-party Swift under app/ios/Pods; retain the Optional-to-Any guard without treating installed dependencies as first-party source. * fix(ios): guard high-risk native crash paths * refactor(ios): route defaults and numeric sinks safely * fix(ios): guard missing watch audio start time * chore(ios): shrink native safety lint baseline * fix(ios): preserve watch battery value type * fix(ios): type Siri telemetry records explicitly * refactor(ios): centralize epoch millisecond conversion | 14 天前 | |
ios: apply same default-calendar fallback to getReminders/completeReminder | 4 个月前 | |
feat: on-device tool surface for macOS and iOS (#10949) * feat(desktop): add an on-device tool surface for messaging and automation The floating-bar execute prompt already told the agent it had "shell + osascript" access to drive Messages, Telegram and friends. Nothing backed that claim: there was no messaging tool in the manifest, and actuation was gated behind desktop.automation.act_dev_only, which release bundles deny. The agent could only reach those apps by improvising raw shell, with no policy, no approval card, and no ledger entry. This adds five kernel-governed tools: search_contacts resolve a name to phone/email handles (Contacts) list_message_chats recent conversations from chat.db (read-only) read_message_history one thread's recent messages (read-only) send_message iMessage/SMS through Messages.app run_applescript AppleScript actuation against local apps and three capability bundles: desktop.contacts.read, desktop.messaging.read and desktop.messaging.send, plus desktop.automation.act — a production sibling of act_dev_only that is reachable in release builds but can never resolve without a dispatch or an unexpired scoped grant. Both reads and sends are classified sensitive, so reading someone's message history requires the same approval record as sending. Scoped grants carry a resourceRef, so approving a send to one recipient does not authorize another. The Messages store is opened read-only through GRDB, mirroring AppleNotesReaderService, and OMI_MESSAGES_DB overrides the path so contract tests run against a fixture instead of real message history. Recipient handles, message bodies, and attachment paths reach osascript as argv rather than interpolated script source: a body can come from a thread the agent just read, so it must never reach the AppleScript compiler. request_permission/check_permission_status grow four types — contacts, calendars, reminders, photos — the grants macOS demands once AppleScript starts driving those apps. Info.plist gains the matching usage strings. Verification: desktop/macos/Desktop $ swift build -> Build complete (73.63s) desktop/macos/agent $ bunx vitest run -> 630 passed, 1 failed (run-attempt-lifecycle artifact assertion; verified failing identically on the base commit 4e879ac7eaf via git stash, unrelated to this change) desktop/macos/agent $ bunx vitest run tests/device-tool-policy.test.ts tests/desktop-tool-policy.test.ts tests/omi-tool-manifest.test.ts -> 42 passed $ check_product_file_line_count_ratchet.py -> OK Generated Swift surfaces were regenerated with agent/scripts/generate-tool-surfaces.mjs, not hand-edited. * feat(app): add the iOS on-device tool surface iOS has no API for sending a message without the user seeing it, so the mobile surface exposes a propose verb rather than a send verb. MFMessageComposeViewController is presented prefilled with the recipient and body, and the system sheet is the approval UI — the delegate reports sent, cancelled or failed, and `ok` reflects what the user actually did. A cancelled proposal never reads as a delivered message. The app previously reached messaging through share_to_contacts_sheet.dart, which builds an `sms:` URI and hands it to url_launcher. That leaves the app and reports nothing back, so nothing could act on the outcome. Tool names and result shapes mirror the macOS device tools, including the `permission`/`next_tool` hint on permission failures, so one model-facing contract covers both platforms. What differs is declared rather than discovered: capabilities() reports can_read_messages=false and can_run_scripts=false, because reading history and running scripts have no iOS equivalent at all. `limited` Contacts authorization (iOS 18) is treated as a usable grant. An earlier revision guarded on `== .authorized`, which would have broken contact search for every user who granted partial access; the iOS-SDK typecheck caught it. Android has no implementation, so the surface advertises no tools there instead of failing at call time. Not included: a transport that lets the backend model call these mid-turn. /v2/messages is a one-way SSE stream, and a client round-trip means restructuring that streaming generator into a suspend/resume protocol with durable state. That is deliberately a separate change; the surface is the half that must exist first. Recorded in the doc rather than left as a TODO. Verification: app $ flutter test test/services/device_tool_surface_test.dart -> 15 passed app $ dart analyze lib/services/device_tools/ test/services/ -> clean $ xcrun -sdk iphoneos swiftc -typecheck DeviceToolsService.swift -> clean (against a Flutter stub; a full iOS build needs pods plus the gitignored env/firebase generated files, so it was not run) $ check_agents_md_lean.py -> ok * docs(desktop): add the changelog fragment for the device tool surface * test(desktop): cover the device tool surface with a hermetic bridge probe * feat(chat): let the model call device tools mid-turn Closes the loop the previous commits left open: the iOS device tool surface existed and was tested, but nothing could invoke it. The model can now call propose_message and search_contacts during a chat turn. No suspend/resume of the streaming generator was needed. The SSE stream for the turn is already open and already carries mid-turn `think:` frames, so a device tool call rides the same stream as a new `tool:` frame and the turn simply stays live while the user answers: client POST /v2/messages { text, device_tools: [...] } backend model calls propose_message -> tool coroutine emits `tool: <base64 json>` on the open stream -> coroutine polls Redis for the result client runs MFMessageComposeViewController, POSTs the outcome to /v2/messages/device-tool/{call_id}/result backend poll observes it, the tool returns, the model finishes the turn The client declares device_tools per request rather than the server assuming them: the capability set depends on the device in hand, so an iPad with no messaging service or an Android build never has the model offered a tool it cannot run. Redis polling rather than an in-process future, because the result POST can land on a different worker than the one holding the awaiting coroutine. The key is uid-scoped so one user cannot answer another's call, and it is deleted on read so a later call cannot reuse a stale result. Cancellation is preserved end to end. A dismissed compose sheet returns status=cancelled verbatim, and the tool description tells the model never to report a message as sent unless the result says sent. An unanswered call times out at 180s with an explicit instruction not to retry, because the user may still be looking at the sheet. Device tool names are added to STANDARD_TOOL_NAMES: they are named like core tools, and _extract_app_id would otherwise read "propose" as an app id and route the call to a non-existent app. A regression test covers that. Registered chat:device_tool_result in the central rate-limit policy table — with_rate_limit rejects unknown policy names at import, which caught this. Verification: backend $ .venv/bin/python -m pytest tests/unit/test_device_tools.py -> 12 passed backend $ python -c "import routers.chat" (with ENCRYPTION_SECRET set) -> imports clean; route /v2/messages/device-tool/{call_id}/result registered app $ bash test.sh -> 995 passed app $ bash scripts/analyze_ratchet.sh -> Analyzer ratchet passed $ check_agents_md_lean.py -> ok Not verified: no live end-to-end run against a real device or a deployed backend. The transport is covered by unit tests on both sides of the wire, not by an integration test. * chore(backend): declare route policy for the device tool result endpoint New backend routes must carry a reviewed policy rather than grow the legacy missing-route baseline. Firebase ID token via dependency, uid-keyed chat:device_tool_result rate limit, chat data domain, first-party surface. * test(desktop): cover the new device tools in the required-field table The seven tools this PR projects into pi-mono were absent from the expected map, so search_contacts failed the deepEqual. The lookup also fell back to [] for any absent tool, which let a new tool with no required fields pass without ever being listed; it now asserts the tool is in the table first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(api): export the device tool result endpoint into the app-client spec Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(backend): wire the device-tool imports into the chat stubs routers.chat now reaches utils.device_tools, which binds StructuredTool and the redis handle at import. These suites stub bare modules on purpose so an unwired name fails loudly, so the names are added rather than the import deferred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(backend): count the device tool result rate limit chat.py carries eight rate-limited endpoints now that the device tool result handler declares chat:device_tool_result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(api): regenerate the TypeScript clients for the device tool types Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(api): model the device tool result response The handler returned a bare dict, so the endpoint shipped an unmodeled success response and the app-client schema inventory refused it. Every neighbouring route on this router declares a response model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(api): regenerate the Swift and TypeScript clients Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(api): move the device tool response model out of the router routers/chat.py is frozen by the product-file line-count ratchet, and the response model belongs beside its request type anyway. Only the import lands in the router, which the baseline is raised by one to cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(backend): hoist the agentic import out of the call phase The fast-unit guard measures call-phase CPU, and importing utils.retrieval.agentic inside the test body spent 1.00s against a 0.12s limit. At module scope the cost lands in collection instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: drop the stray pnpm lockfile and workspace manifest These landed by accident in a test commit. The repo root installs with npm, nothing outside two Windows workflows mentions pnpm, and the workspace manifest's allowBuilds values were never filled in — they are still the literal "set this to true or false" placeholder, which pnpm reads as "not approved" rather than as a decision. Between them they were 9,263 of the diff's lines. * chore: ignore the pnpm files the lefthook probe leaves behind * fix(app): compile DeviceToolsService into the iOS app The service was added to app/ios/Runner/ and referenced from AppDelegate, but never registered in the Xcode project. Runner is not a PBXFileSystemSynchronizedRootGroup — only omiWatchApp is — so a file that is not listed explicitly is not compiled. The whole iOS device-tool surface was dead code. Both app targets build AppDelegate, so both need the service: Runner and RunnerRayBanDat each get a PBXBuildFile entry against one shared PBXFileReference. Verified: plutil -lint project.pbxproj -> OK both targets' PBXSourcesBuildPhase now list it -> 38 files each xcrun -sdk iphoneos swiftc -typecheck -> clean (Flutter stub) * fix(chat): let a device tool time out before the turn does The wait was written down as 180s next to a stream that cancels its producer at AGENT_STREAM_MAX_DURATION_SECONDS = 150s. The tool never reached its own timeout: at 150s the stream died with idle_timeout while the user was still looking at the compose sheet, and they could go on to tap Send afterwards. That is precisely the ambiguous "was it sent?" outcome this surface was built to remove, and the model never got the timed_out result the tool description tells it to explain. The wait is now derived from the stream budget instead of written beside it, reserving headroom for the reasoning before the call and the sentence after it. build_device_tools no longer defaults timeout_seconds: only the caller knows the turn's budget, and a default is what let these two drift. Verified: stream cap 150.0 -> device tool wait 120 BACKEND_UNIT_TEST_FILE_LIST=... bash test.sh -> 15 passed in 2.83s black --check -> unchanged Failure-Class: none * fix(app): make the iOS device tool surface answer every call Six defects, all of which either stranded a Flutter call or told the model something untrue about what it was allowed to do: - proposeMessage checked pendingComposeResult on the calling thread but assigned it inside a later main-queue block. Two proposals could both see a nil slot; the second overwrote the first, and the first Flutter call never got a reply. The check and the reservation now happen together on the main queue, which is also the only place that field is touched. - The same block captured self weakly and returned without calling result when self was gone, hanging the Dart future. Every exit path replies now. - contactsPermissionStatus returned no `ok`, and the common result contract reads a missing `ok` as failure — so every successful status check looked like a failed one. - requestContactsPermission inferred the grant from requestAccess's boolean, which is true for both full and iOS 18 limited access. It re-reads the status instead, so limited is no longer reported as a full grant. - A fresh install sits at notDetermined, and nothing in the mobile flow can call request_permission — the surface exposes only search_contacts and propose_message. Named-recipient lookup could therefore never succeed. The first lookup now prompts. - Permission failures omitted next_tool/next_tool_arguments that the Dart wrapper and the macOS executors both send, so the model lost the recovery hint on iOS only. The deep-link branch is also moved to the end of didFinishLaunching. It returns early to stop propagation, so a cold launch from a URL skipped every method channel registration for the rest of the session. Verified: xcrun -sdk iphoneos swiftc -typecheck -> clean (Flutter stub) * fix(app): stop a bad tool frame or a stale account from hanging the turn Four ways the Dart side could leave the server waiting or lie to the model: - A malformed tool: frame was logged and dropped. The server stays blocked on that call id, so the user waited out the whole timeout for nothing. When the id itself survives the damage the frame is now answered with a malformed_request failure; only a frame with no recoverable id is silent, because there is no address to reply to. - decodeBase64 throws on a corrupt frame, and it ran inside the stream loop, so one bad frame tore down the entire turn rather than one tool call. - device_tools was built from the platform's tool list, never the device's. A simulator, an iPod touch, or an iPad with no messaging service still offered propose_message, and the round trip could only return messaging_unavailable. The list now comes from the native capabilities() response, which is what the per-request contract promised. - The result was posted with whatever account was current when the sheet closed. A sign-out or account switch mid-sheet attributed the outcome to the wrong uid or failed outright, timing out a turn whose message may well have been sent. The dispatcher binds the owner at dispatch and drops the result if it changed. Verified: flutter test test/services/device_tool_* -> 27 passed dart analyze lib/services/device_tools lib/backend/http/api/messages.dart -> No issues found * fix(chat): accept a device tool result only for a call in flight Two ways the Redis handoff could go wrong: The result endpoint wrote whatever it was given, for whatever call_id it was given. It verified the caller was authenticated but never that anyone was waiting, so at the configured 600 requests/hour any user could invent call ids and park unbounded nested payloads in shared Redis, each held for the full TTL. The dispatching coroutine now marks the call in flight before the tool: frame goes out, and the endpoint consumes that marker with a DELETE — whose return count makes the check and the consume one atomic step, so a replayed submission cannot land twice. The marker is dropped when the wait ends, so a late answer is refused rather than stored. Results are also size-bounded before any write, and an oversized one does not consume the marker, so it cannot lock the real client out of answering. A failed Redis GET during the wait raised straight out of the tool, which the model sees as a generic execution error — free to retry. But the frame was already on the wire and the compose sheet may already be open, so a poll failure says nothing about whether the message was sent; retrying could send it twice. Poll failures are now recorded through record_fallback and the wait continues, so the outcome is either the real result or the typed timed_out that tells the model not to retry. The chat router harness stubs utils.observability as a plain module, so it gains a fallback stub alongside the journeys and transcription ones. Verified: BACKEND_UNIT_TEST_FILE_LIST=... bash test.sh -> 21 passed chat/rate-limit/prompt-cache files -> pass; the fast-unit duration guard trips on this host for test_chat_stream_error_fallback and test_chat_quota_counting_router, verified identical on the unmodified tree via git stash, and green in CI black --check -> unchanged Failure-Class: none * fix(desktop): correct the Messages reads, the sends, and the permissions they ask for Reads returned the wrong conversation, or none at all: - A handle can belong to a direct thread and any number of group chats, and the lookup took LIMIT 1 with no ORDER BY. "Read my thread with Sam" could return an unrelated group — other people's messages, and the wrong context to reply from. Direct chats now sort first, then most recent activity. - A stale or invented chat_id read back as ok with an empty message list, indistinguishable from a quiet thread. It is now validated against the chat table and raises the existing chat_not_found. - chat_id was parsed only as a number, so a quoted "12345" fell through to a handle lookup or a missing-reference error. - Modern Messages stores leave message.text null and keep the content in attributedBody, so ordinary threads read as blank. The typedstream is now decoded, with the empty body as the fallback when it does not match. - fileExists cannot tell a missing store from one TCC is withholding, so a Mac without Full Disk Access was told Messages had never been signed in, with no permission and no next_tool. Presence is read from errno instead. Sends could go to the wrong person, or twice: - The preferred handle was chosen by searching the *localized* phone label for "mobile". That only matches on an English system; everywhere else it fell through to whatever number came first, often a home or work line. Matched against CNLabelPhoneNumberMobile now, on macOS and iOS both. - The body and the attachment are two effects. A rejected attachment failed the whole script, and `auto` moved on to SMS and delivered the body a second time. The script reports a partial send and the caller stops. - PipeProcessRunner throws on timeout rather than returning timedOut, so the ambiguous-send branch was unreachable and a hung osascript surfaced as a clean failure the model was free to retry. Permissions pointed somewhere useless: - An Accessibility denial was reported as needing Automation. The model would call request_permission(type: "automation"), which probes System Events, get "granted", and fail again identically. Assistive-access denials are their own classification and ask for accessibility. - contacts/calendars/reminders/photos were advertised by request_permission but missing from the external surface's PERMISSION_TYPES, so every such request was rejected before reaching Swift. And two that were not about correctness: - osascript and Messages sends ran on the main actor, holding the UI thread for up to the full 120s timeout — freezing the desktop, including the cancellation controls. They run off the main actor now. - Int(_: Double) traps on 1e100 or NaN, both legal under a bare `type: number` schema, so a model typo could kill the app. - send_message and run_applescript arguments were logged in full, writing recipients, message bodies, and whole scripts to the production log. Verified: swift build -> Build complete agent npx vitest run -> 631 passed xcrun -sdk iphoneos swiftc -typecheck -> clean (Flutter stub) swift-format -> applied Failure-Class: none * fix(desktop): tell a pending Full Disk Access grant from a missing one macOS binds Full Disk Access at process start, so a user who grants it while Omi is running stays locked out until Omi relaunches — and the in-process read fails exactly as it would if they had never granted anything. checkFullDiskAccessDirectly returned false for both, so the app sent them back to System Settings to toggle something already on, and would keep doing so forever. FullDiskAccessProbe asks a child process the same question. A freshly spawned child inherits the current grant rather than the one in force at launch, so child-can-read plus parent-cannot means the grant is live and only the restart is missing. Any probe failure answers "denied", because wrongly claiming a grant exists would send the user to restart the app and land them back where they started. Also drops the fileExists check that decided which path to test: for a TCC-protected path fileExists reports false for a directory that is really there, so the loop could stop on the wrong answer. It uses the same errno-based presence check the Messages reader now does. Verified: swift build -> Build complete * feat(desktop): let the agent read Apple Mail headers Omi could read Notes and Messages but had nothing for Mail, so "what's waiting for me" was a question it could not answer from local data. omi-v4 already solved this against the same store; this ports the approach. AppleMailReaderService opens ~/Library/Mail/V*/MailData/Envelope Index read-only through GRDB and returns subject, sender, date, and read/flagged/replied state. Bodies live beside that index as .emlx files and this reader never opens them — answering who wrote and about what does not need the full text of someone's mail, and the tool description tells the model to say a body is unavailable rather than infer one. Three things the store forces: - It picks the highest-numbered V{n} directory. Mail leaves old format versions in place, so a hardcoded version reads stale mail after an upgrade. - It probes the schema with PRAGMA table_info before querying. Mail's schema is Apple's private business; an OS update should degrade to a legible schema_unavailable rather than a raw SQLite error mid-query. flags and addresses.comment are optional and substituted when absent. - date_received is plain Unix seconds here, unlike the Core Data reference dates in Notes and the Apple absolute times in Messages. Reusing a sibling reader's conversion would shift every message by 31 years, so the conversion is stated rather than inherited. list_mail_messages sits behind its own desktop.mail.read bundle, classified sensitive like desktop.messaging.read. Separate bundles because agreeing to let the agent read a text thread is not agreeing to let it read the inbox — there is a test that a messaging grant does not authorize a mail read. Generated Swift surfaces were regenerated with generate-tool-surfaces.mjs, not hand-edited. Verified: swift build -> Build complete agent npx vitest run -> 634 passed scripts/test-tool-surfaces.sh -> 108 pass, 0 fail backend device tools -> 21 passed app bash test.sh -> 1000 passed Not verified: no read against a real Envelope Index. This environment has no Full Disk Access, so the SQL, the schema probe, and the V{n} selection are unit- and typecheck-level only. The hermetic probe covers the absent-store classification; it cannot cover a populated mailbox. * refactor(desktop): keep the device-tool helpers out of ChatToolExecutor The file's own ratchet justification says every executor body lives in ChatToolExecutor+DeviceTools.swift and only dispatch cases stay here. The argument redaction and the Full Disk Access status helper are device-tool concerns, so they move to match, which puts ChatToolExecutor back under its baseline instead of raising it again. backend/routers/chat.py does grow by 13 and its baseline is raised with the reason: the in-flight claim and the size bound live in utils/device_tools.py, and only the HTTP status mapping belongs at the route that owns the response contract. Verified: swift build -> Build complete ChatToolExecutor.swift -> 2966 lines, baseline 2973 * chore(desktop): cover the new readers in the device-tool e2e flow The mail reader and the Full Disk Access probe are Swift sources with no e2e coverage entry, which desktop-e2e-flow-coverage rejects. Both are exercised by the existing hermetic probe, so they join its covers list rather than getting a flow of their own. Also drops a force unwrap swiftlint flagged in the argument redaction helper. Verified: scripts/pr-preflight --base upstream/main -> passed, 36 checks * chore(api): re-export the app-client spec after the endpoint docstring The device tool result endpoint's docstring is its OpenAPI description, so documenting the in-flight requirement made the checked-in spec stale. Regenerated with backend/scripts/export_openapi.py, not hand-edited. * fix(agent): register desktop.mail.read in coordinator bundle schemas Adds desktop.mail.read to the desktopCoordinatorBundleSchema enum in control-tools.ts and both manifest bundle enums so the policy layer's MAIL_READ classification of list_mail_messages is selectable/requestable end-to-end instead of only inside evaluateDesktopToolPolicy. Includes list_mail_messages in the Swift chat tool executor manifest coverage test. Addresses Git-on-my-level review on 2a8ad8c1 (PR 10949). Verification: cd desktop/macos/agent && npm ci && npm run build && npx vitest run => 56 files, 634 tests passed. * feat(device-tools): make the contacts permission prompt reachable on iOS The search_contacts failure path tells the model to call request_permission, but that tool was not declared in the iOS surface, not routed in the Dart dispatcher surface, and not in the backend DEVICE_TOOL_SPECS — so on a fresh install with Contacts notDetermined, named-recipient lookup always failed with no way to recover. Adds the tool end to end: backend spec (type arg), iOS surface descriptor + execute routing to the native prompt, and docs. Addresses chatgpt-codex-connector P1 on 59534f38 (PR 10949): contacts permission request unreachable through the chat flow. Verification: - cd app && flutter test test/services/device_tool_surface_test.dart => 20 passed - cd app && flutter analyze <files> => no issues - cd backend && pytest tests/unit/test_device_tools.py => 21 passed - pytest tests/unit/test_rate_limiting.py => 69 passed * fix(desktop): make the contacts prompt wait cancellation-aware ContactsReaderService.requestAccess used a plain checked continuation, so a TCC prompt still open when the tool is cancelled or the owner changes left the continuation suspended until the user answered, blocking owner-bound work drain. Routes the first-use prompt through the existing awaitCancellablePermissionRequest once-guard adapter used by every other permission wait. Addresses chatgpt-codex-connector P1 on 59534f38 (PR 10949). Verification: cd desktop/macos && xcrun swift build -c debug --package-path Desktop => Build complete; swift-format lint strict => clean; dev-feedback --once swift 'ChatToolExecutorRowIntTests|ChatToolExecutorPolicyTests' => 9 tests passed * fix(desktop): enforce device tool approval boundaries Harden owner-bound device effects, cancellation, output capture, permission requests, and mail/contact filtering. Persist atomic backend tool results and enforce scoped desktop approvals before runtime authorization. Add focused policy, bridge, and process tests.\n\nFailure-Class: none\n\nVerification: Swift 6 package build completed; SwiftLint and formatter lint passed; Dart analyze passed; Bun build and device-tool-policy tests passed; desktop test-quality and diff checks passed. Full XCTest execution is blocked locally by missing Sparkle runtime resolution, and the backend unit runner is blocked by missing Google Python dependencies. * fix(device-tools): honor the remaining stream budget Evaluate the device-tool wait timeout when the tool starts and clamp it to the actual remaining agent stream budget. Add a focused timeout-provider contract test.\n\nFailure-Class: none\n\nVerification: black, Python bytecode compilation, and git diff check passed; pytest is blocked by the local missing google dependency in backend/tests/unit/conftest.py. * fix(device-tools): scope automation and contact recovery Keep AppleScript output-limit failures out of permission recovery, require exact script resources for automation grants, and route permanent iOS Contacts denial to Settings instead of request loops. Add a scoped automation policy regression test.\n\nFailure-Class: none\n\nVerification: Swift format/lint, Bun desktop policy tests, and git diff check passed; iOS build was not available in this desktop workspace. * fix(backend): keep agentic gateway harness importable under the langchain stub test_chat_agent_provider_retry.py failed with ImportError when test_prompt_cache_integration.py stubbed langchain_core.tools first: perplexity_tools' import-time @tool decorator resolved against a bare stub missing `tool`, and the gateway_client stub lacked feature_auto_lane_id / get_llm_gateway_base_url / llm_gateway_headers. Only install the passthrough @tool when this file created the stub (leaving the real decorator intact otherwise), mirror the gateway surface the tool module names, and let the perplexity proxy fall back to its static schema when the passthrough leaves a bare function. 83 tests green. * test(backend): reload the mocked agentic when test_device_tools imported the real one Rebase seam: on current main the real look_at_frame tool's runtime name diverges from its export identifier (like close_fact), and the entity timeline schema carries pydantic titles that _convert_tools strips. The prompt-cache harness assumed it always loads agentic against its own stubs; test_device_tools.py imports the production module at collection time, so the cache tests silently asserted against real tool objects and failed on both divergences. Cache the stub-loaded module and drop any real utils.retrieval.agentic from sys.modules before loading. * fix(app): deliver device tool results through the typed ApiResult path Rebase seam: current main requires new app files to use executeApi with exhaustive ApiResult handling instead of nullable makeApiCall (api-result-boundaries). The dispatcher's fire-and-forget result POST now classifies failures through ApiProblem instead of guessing from a nullable response; the log-only behavior is unchanged — the server-side call still times out on its own when delivery fails. * docs: keep the device tool surface sections inside the AGENTS.md lean budgets Rebase seam: main tightened the AGENTS.md size ratchets after this PR branched, leaving no headroom for the new sections. Condense the device tool surface routing to its sharpest invariants and pay for the bytes by tightening equally verbose neighboring bullets (phone-mic capture, dev-run fast lane, agent runtime cache) without dropping any fact. * chore(app): add the unreleased changelog fragment for the iOS device tool surface Rebase seam: main now requires an unreleased app changelog fragment for any app/ change; the PR predated the rule. * fix(app): register the device tools channel on the shared messenger Rebase seam: main's SwiftLint now rejects force unwrapping; the neighboring channels already register on the engine-bridge messenger. * fix(desktop): declare the device tools probe action's effects Rebase seam: main's automation action registry now requires an explicit effects set on every registration; the read-only probe declares none. * fix(app): make DeviceToolsService compile in the real iOS app build Rebase seam surfaced by the ci:full iOS Compile Check: the original typecheck ran against a Flutter stub, so two real-SDK errors survived — an instance-method call in a closure without explicit self, and the sync UIApplication.open overload that no longer exists (the async overload was the only visible one). Use explicit self and the completion-based open. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Max Carter 祁明思 <undivisible@users.noreply.github.com> Co-authored-by: David Zhang <david@scalingforever.com> | 8 天前 | |
fix(app): address review on the untethered launch fix - Notice on the phone names the canonical `bash setup.sh ios` invocation from app/ instead of a repo-root path. - setup.sh warning describes the shipped behavior (engine-unavailable notice, not a crash) and only fires for the dev flavor, since its remedy builds the dev flavor. - Drop the GOOGLE_REVERSE_CLIENT_ID pin from devProfile/devRelease so Custom.xcconfig keeps owning that value. Failure-Class: none Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YU5BMEAfcFxNSWmLLDNfZ3 | 1 个月前 | |
feat(app): tiered pendant capture — pause the live uplink after silence, resume on foreground or charge (#20837) * fix(app): close pendant listen socket on pause and reopen on resume * feat(app): pause silent pendant uplink until foreground or charge start * fix(app): stop synthetic pendant battery ticks and redundant widget reloads * feat(app): add opportunistic background backlog sync without live uplink * fix(app): preserve timeout settings and isolate uplink policy effects * chore(app): fix failure-class scope hints to tracked globs * fix(app): preserve charging observations across failed GATT reads Treat failed and empty charging reads as unknown, retaining the tracker's last successful state and UI/widget value. Only successful observations may create charge-start edges. Audit confirms notifications are real characteristic payloads, with empty notifications ignored. Cover read/notify recovery, clean reconnects, and genuine charging transitions through DeviceProvider. * fix(app): fence background reconnect backlog sync from live capture Keep deviceConnected background-eligible to honor generous reconnect sync without weakening immediate BLE link recovery. Like periodic wakes, its discovery and backlog drain now run inside SyncWakeScope. Update stale foreground-only comments and prove capture start attempts cannot open sockets or BLE audio, reconnect bursts stop at two serial passes, and failures release holds without scheduling background retries. * fix(app): defer resume commits until sync drains finish Choose shape (a): keep pause intent intact until SyncWakeScope.whenIdle, then dispatch the existing resume transition. Apply the same fence to Home, pendant double-tap, Live Activity, foreground, and charging resume paths; re-arm the existing keepalive after successful resume. No background retry timer or sync admission bypass is added. Regressions reproduced the old failure for foreground and background reconnect drains and assert connected socket, active BLE audio, and armed keepalive after completion. * fix(app): gate the first charging notification after a failed read Firmware force-notifies charging state when notifications are enabled. If the read failed, preserve its per-connect resume gate for the first valid notification, even after the sync fence drops. Always record that sample; later genuine false-to-true edges still resume. Cover explicitly gated connects, notifications during/after sync, empty payloads, and ordinary first charging observations. * fix(app): preserve authorized charging edges across later sync scopes Latch the first-observation resume gate per connection. A scope that starts later defers capture work rather than consuming the first real charging edge after an unknown read. Connections initially admitted only for sync still observe without resuming. Add a provider regression for one initial edge and later genuine charge starts. * fix(app): reconcile fenced pendant transports on sync scope drop Enforce live-session liveness at the transport layer instead of relying on a safe gap before resume dispatch. Coalesce skipped transport attempts, await scope drop and capture commit, then revalidate current ownership and idempotently restore missing transports plus keepalive. A new scope during the re-drive records fresh demand; no retry timer, policy write, or session mint is introduced. Clear cancelled audio handles and make already-unmuted resume a no-op. Keep dispatch deferral as an optimization only. Add the numbered eight-case resume/fence suite, including a real coalesced coordinator with a controlled microtask yield at depth zero. Case 3 fails on fbb0ba0483 with a disconnected socket after the whole wake, and passes with reconciliation. Pure sync still opens zero transports. | 6 天前 | |
feat(app): tiered pendant capture — pause the live uplink after silence, resume on foreground or charge (#20837) * fix(app): close pendant listen socket on pause and reopen on resume * feat(app): pause silent pendant uplink until foreground or charge start * fix(app): stop synthetic pendant battery ticks and redundant widget reloads * feat(app): add opportunistic background backlog sync without live uplink * fix(app): preserve timeout settings and isolate uplink policy effects * chore(app): fix failure-class scope hints to tracked globs * fix(app): preserve charging observations across failed GATT reads Treat failed and empty charging reads as unknown, retaining the tracker's last successful state and UI/widget value. Only successful observations may create charge-start edges. Audit confirms notifications are real characteristic payloads, with empty notifications ignored. Cover read/notify recovery, clean reconnects, and genuine charging transitions through DeviceProvider. * fix(app): fence background reconnect backlog sync from live capture Keep deviceConnected background-eligible to honor generous reconnect sync without weakening immediate BLE link recovery. Like periodic wakes, its discovery and backlog drain now run inside SyncWakeScope. Update stale foreground-only comments and prove capture start attempts cannot open sockets or BLE audio, reconnect bursts stop at two serial passes, and failures release holds without scheduling background retries. * fix(app): defer resume commits until sync drains finish Choose shape (a): keep pause intent intact until SyncWakeScope.whenIdle, then dispatch the existing resume transition. Apply the same fence to Home, pendant double-tap, Live Activity, foreground, and charging resume paths; re-arm the existing keepalive after successful resume. No background retry timer or sync admission bypass is added. Regressions reproduced the old failure for foreground and background reconnect drains and assert connected socket, active BLE audio, and armed keepalive after completion. * fix(app): gate the first charging notification after a failed read Firmware force-notifies charging state when notifications are enabled. If the read failed, preserve its per-connect resume gate for the first valid notification, even after the sync fence drops. Always record that sample; later genuine false-to-true edges still resume. Cover explicitly gated connects, notifications during/after sync, empty payloads, and ordinary first charging observations. * fix(app): preserve authorized charging edges across later sync scopes Latch the first-observation resume gate per connection. A scope that starts later defers capture work rather than consuming the first real charging edge after an unknown read. Connections initially admitted only for sync still observe without resuming. Add a provider regression for one initial edge and later genuine charge starts. * fix(app): reconcile fenced pendant transports on sync scope drop Enforce live-session liveness at the transport layer instead of relying on a safe gap before resume dispatch. Coalesce skipped transport attempts, await scope drop and capture commit, then revalidate current ownership and idempotently restore missing transports plus keepalive. A new scope during the re-drive records fresh demand; no retry timer, policy write, or session mint is introduced. Clear cancelled audio handles and make already-unmuted resume a no-op. Keep dispatch deferral as an optimization only. Add the numbered eight-case resume/fence suite, including a real coalesced coordinator with a controlled microtask yield at depth zero. Case 3 fails on fbb0ba0483 with a disconnected socket after the whole wake, and passes with reconciliation. Pure sync still opens zero transports. | 6 天前 | |
Fix opening exact iOS Spotlight results (#20156) * fix: open iOS Spotlight entities in their exact Home destination * fix(ios): dedupe Spotlight opens and expire pending routes | 12 天前 | |
fix(app): verify pendant capture from arriving audio and recover a silent link (#20378) * fix(app): verify pendant capture from arriving audio and recover a silent link A pendant could stay Bluetooth-connected for hours while no audio reached the app: a restored link was treated as capturing, nothing detected zero ingress, and nothing forced a fresh connection. Capture health on iOS now starts unverified on a restored or replayed link and is established only by a confirmed audio subscription plus audio on the current connection. When evidence is absent or stops, native code runs one persisted-budget recovery: repair the subscription, then a real disconnect and reconnect. The Listening claim and timer follow that evidence; the card, transcript and controls stay. Subscription results are reported to Dart on both platforms, and Recording Requested keeps the telemetry denominator now that Recording Started waits for audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(app): run native capture recovery tests in CI and tighten review-flagged tests Register the capture health and recovery transaction harnesses in the checks manifest so CI runs them, and let the BLE policy test run on its own. Make four tests assert what their names claim, keep the subscription-failure event in the scorecard export, and word the changelog as an attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(app): bound Android CCCD waits and harden the recovery reconnect path Address the remaining review findings on the pendant capture link: - Bound the accepted CCCD write with a 15s native timeout mirroring iOS OmiCaptureHealth.Policy.operationTimeout, so a missing onDescriptorWrite fails the pending completion and releases the GATT queue instead of blocking Dart's retry behind it forever. The pending map entry is the ownership token: a late ack after timeout or cleanup settles nothing and cannot double-release a later command's slot. - unsubscribeCharacteristic gets the same sentinel, timeout, and try/catch treatment (same unbounded-wait defect class). - cleanupPeripheral now fails and removes pending CCCD completions and their timeout callbacks for the torn-down peripheral. - subscribeCharacteristic never throws synchronously (pre-check and queued body wrapped); the generated Pigeon handler cannot strand Dart without a reply. BleHostApiImpl adds the same guard. - Recovery resume goes through existing.connect() instead of the bare transport: transport failures surface as DeviceConnectionException like the cold-connect path, and the pendant time sync runs after the link returns. The service state callback is re-passed so subscribers keep receiving state changes. - Recheck native ingress ownership after the feature-gate await so a legacy episode cannot land invisibly and block later declarations. - _resolveIfActive on ingress-failed episodes releases the telemetry slot while retaining the native-owned ingress failure. - The verified ingress start survives a lease expiry (only withdrawn proof clears it) so the capture timer continues instead of restarting at zero when audio returns. * fix(app): isolate CCCD write ownership and retire timed-out GATTs Serialize descriptor settlement and queue callbacks on the main handler. Arm each write token before issuing its CCCD operation, and release only the current command owner. Retire a GATT with a missing ACK, fail its queued subscription replies, and hand reconnection to the existing managed service so late ENABLE or DISABLE callbacks cannot settle a newer write. Add JVM regressions for late ACKs, stale timers, ENABLE/DISABLE ownership, retired connections, queue-owner changes, rejected writes, and cleanup. * fix(app): bound reconnects after CCCD timeout retirement Distinguish missing CCCD acknowledgements from ordinary GATT disconnects. Persist a per-device budget for two reconnects with backoff, and reset it only after a current successful descriptor ACK. Stop companion and sticky restoration from bypassing exhaustion. Project the exhausted result through existing capture health without a competing Dart retry. Preserve capture intent, replay stored failures when capture binds, and keep healthy Android recording telemetry unchanged. Cover the policy, ACK ownership, and recovery presentation with production-seam tests. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: David Zhang <9387252+Git-on-my-level@users.noreply.github.com> | 10 天前 | |
Omi Apple Watch Integration + Comm arch changes (#3012) * init watch app and audio streaming * watch pairing check + build fixes * apple watch integration and cleanup * missing import * rename watch folder * add app icon * improve connection ui stuff and cleanup * cleanup and prepend 3 dummy bytes * misc * move file from src to gen folder * introduce discoverers and communicators * introduce transporters * use transporter for communication in connection and device service * remove communicators and cleanup * run discoverers parallely | 1 年前 | |
feat(app): stream mobile voice reply playback | 16 天前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
fix(app): make profile and release dev builds signable on a device `OMI_MOBILE_BUILD_MODE=profile bash setup.sh ios` failed at code signing on a physical iPhone for two reasons that never affected debug builds: - devProfile.xcconfig and devRelease.xcconfig did not pin APP_BUNDLE_IDENTIFIER the way devDebug.xcconfig does, so the app fell back to Custom.xcconfig's hostname-suffixed id, which may not be registrable to the team and no longer matches the debug install or its keychain session. Pin both to the development id, with the same Google reverse client id. - RunnerProfile-dev.entitlements declared aps-environment "Devdevelopment", which no provisioning profile can satisfy. Failure-Class: none Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YU5BMEAfcFxNSWmLLDNfZ3 | 1 个月前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 个月前 | |
fix(ios): type-safe native sinks and a SwiftLint safety ratchet (#19586) * fix(ios): add typed Foundation sink boundaries Guard defaults, JSON serialization, and floating-point conversion against type-erased native traps. Prove the boundaries with hostile Swift harness inputs and representative Runner callers. * ci(ios): ratchet native Swift safety lint Pin SwiftLint through the verified desktop binary, baseline existing violations, reject baseline growth, and run macOS-scoped manifest checks in the hosted iOS compile job. * docs(app): note iPhone diagnostics reliability improvement * ci(ios): catch main sync call syntax in SwiftLint * test(ios): scope legacy plist scan to handwritten Swift CocoaPods places third-party Swift under app/ios/Pods; retain the Optional-to-Any guard without treating installed dependencies as first-party source. * fix(ios): guard high-risk native crash paths * refactor(ios): route defaults and numeric sinks safely * fix(ios): guard missing watch audio start time * chore(ios): shrink native safety lint baseline * fix(ios): preserve watch battery value type * fix(ios): type Siri telemetry records explicitly * refactor(ios): centralize epoch millisecond conversion | 14 天前 | |
Speaker identification: measured threshold + margin, live clip pooling, SpeechBrain retirement; carries #12531 without the onboarding-step removal (#12935) * fix: unblock speech-profile redo and STT pre-flight for already-onboarded accounts Rebased onto origin/main as a single commit. Keep both main's open_provider_selection_circuit and this PR's is_stt_available helpers, then regenerate OpenAPI clients from the rebased backend. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(app): fade transcript words in as they arrive on the speech-profile screens Add FadeInWordsText: a centered word Wrap where only the words appended since the previous render animate from transparent to opaque with a short stagger, existing words stay put, and a rewritten transcript re-reveals from the start. Both the onboarding speech-profile step and the Settings redo page adopt it in the next commit so the live transcript reads the same whether the words come from the server or the on-device fallback. Verification: flutter test test/widgets/fade_in_words_text_test.dart (4 passed); observed on an iPhone 16 Pro via hot reload while dictating. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): fall back to on-device speech recognition when server STT is unavailable The speech-profile question flow (onboarding step and Settings redo) needs a transcript only to drive the questions and progress; the voice print itself is computed server-side from the WAV uploaded at finalize(). So when the backend's streaming STT is down, transcribe on the phone instead of dead-ending: - SpeechProfileProvider gains a local-STT mode. It is entered up front when the stt-availability pre-flight fails, or mid-session after the existing three 1011 closes with no captured speech (previously STT_UNAVAILABLE). The socket becomes the existing CompositeTranscriptionSocket: an on-device polling primary (Apple speech on iOS, downloaded Whisper on Android) forwarding suggested_transcript frames to the backend listen socket in custom_stt mode, which the OnboardingHandler already consumes like server STT output. No backend change; a receiver regression test pins that seam. - iOS on-device recognition hardening (AppDelegate.swift): resolve the app's bare language code to an installed on-device locale (a recognizer built from "en" failed every request with kAFAssistantErrorDomain 1101); reply exactly once per clip on final result, error, or a 20 s timeout, keeping partial results; and expose onDeviceAvailable, which probes a silent clip so a phone with Siri and Dictation disabled (kLSRErrorDomain 201) is reported as "no local STT" instead of entering the fallback blind. - PurePollingSocket bounds each transcribe() with a 30 s timeout. A provider that never answered left the processing flag set forever and silently stopped transcription for the rest of the session; now the audio is requeued and the next tick retries. This also protects the main app's on-device mode. - When neither server nor on-device STT is available, the pre-flight dialog now says to check the connection or turn on Dictation. - Speech-profile UI: subtler mic-level glow, and the live transcript uses the new fade-in words widget. Verification: - flutter test (full suite): 1731 passed, 5 skipped; scripts/analyze_ratchet.sh passed - new tests: speech_profile_provider_test (5 fallback cases), pure_polling_test (hung-provider timeout), fade_in_words_text_test (4), backend test_onboarding_question_start (suggested_transcript reaches the transcript queue only in custom-STT mode) - live on iPhone 16 Pro against the local dev harness with the STT primary forced unavailable: session connects with custom_stt+onboarding flags, Apple on-device recognition returns the spoken answer (~180 ms per clip) and it is forwarded to the backend; with Dictation disabled the probe reports unavailable and the dialog appears. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): use SpeechAnalyzer for on-device speech on iOS 26 SFSpeechRecognizer's on-device mode fails with kLSRErrorDomain 201 whenever Siri and Dictation are turned off in Settings, which is what produced the "turn on Dictation" pre-flight dialog in the speech-profile fallback. iOS 26's SpeechAnalyzer/SpeechTranscriber has no such dependency: the language model is an asset the app installs itself through AssetInventory. - transcribe: on iOS 26 run the clip through SpeechAnalyzer (preset .transcription, analyzeSequence(from:) + finalizeAndFinish), falling back to the SFSpeechRecognizer path only if the analyzer throws. - onDeviceAvailable: report true when a supported locale's model is installed or installs within 8 s; a longer download keeps going in the background and the first transcribe() waits for it. Concurrent callers share one download. - SFSpeechRecognizer remains the path for iOS 15-18. Verified on an iPhone (iOS 26.6.1) with Dictation off: the speech-profile redo enters local-STT mode and transcribes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): speech profile talks through three topics and completes on a word target Speech-profile recording (onboarding step and Settings redo) no longer walks one question at a time with a percentage bar. Instead: - A compact white-outlined card headed "Answer with your voice:" lists three topics (where you live, what you do for work, your long-term goal), and a thin bar under it fills as the user speaks. Reaching SpeechProfileProvider.targetWordCount (60 spoken words) finalizes the recording; the backend's onboarding_complete event no longer does, so "bar full" and "done" are the same moment. Omi's own question segments are excluded from the count. - The live transcript is bottom-anchored in a box exactly three lines tall above the card, so whole lines scroll off the top and nothing overlaps. - The Play button on the Settings page plays the saved profile audio in place (just_audio) and turns into Stop, instead of opening the samples page. Redo stops playback first. - Backend ONBOARDING_QUESTIONS is the same three topics, and OnboardingHandler keeps the transcript across questions so one stretch of speech can satisfy several of them. - Removed the unused percentage progress-bar widget and the "Skip this question" button; onboarding keeps "Skip for now". Tests: speech_profile_provider_test (word target fills, finalizes once, ignores Omi segments and the backend completion event); backend test_onboarding_talk_about_flow (one transcript answers every topic; the transcript is kept when it stops answering). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(dev-harness): advertise local-storage links on OMI_DEV_HOST A phone built against OMI_DEV_HOST could reach the backend but not the files it links to: OMI_LOCAL_STORAGE_BASE_URL was always http://127.0.0.1:<port>/_local/storage, so playing the saved speech profile from a device failed. The harness now derives a dev_advertise_host from OMI_DEV_HOST (loopback stays the default) and uses it only for that base URL; every other service address still binds and talks over loopback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the last three whole transcript lines instead of a clipped scroll The speech-profile transcript was a bottom-scrolled ListView clipped to a three-line box, so a sliver of the line above always showed at the top edge and read as cut-off text. FadeInWordsText now takes visibleLines: it replays the Wrap line breaking with measured word widths and builds only the words on the last N lines, so earlier lines drop off whole, nothing is clipped or scrolled, and words keep their reveal state while on screen. Both screens use visibleLines: 3 inside a fixed three-line, bottom-anchored area, moved a little further above the topics card. Test: fade_in_words_text_test covers short text showing everything, earlier lines dropping once the text exceeds three lines, and the shown words matching the line-break replay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): measure transcript lines with the effective text style; 40-word target FadeInWordsText replayed the Wrap line breaking with the caller's raw style, but each word's Text inherits the ambient DefaultTextStyle (font family, weight) under that style, so the replay undercounted lines and the real layout could reach four lines and draw over the topics card. Measure with the same merged style, and clip the fixed three-line area on both screens as a safety net so a stray line can never overlap the card. Also lower SpeechProfileProvider.targetWordCount from 60 to 40 so the recording finishes sooner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the "<Name>'s Speech Profile" title on one line The title wrapped onto two lines for longer names; it now scales down to fit a single line instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): never clip the speech-profile transcript; raise it above the card The three-line transcript area was a fixed-height clipped box, so whenever the rendered lines ran taller than the fontSize*height estimate (text scaling, font metrics) the top line was cut off. FadeInWordsText already guarantees at most three lines, so the area now only has a three-line minimum height (scaled with the text scaler) and grows to its content instead of clipping. Both screens also keep more space between the transcript and the topics card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): finish the speech profile after three sentences; no page-load spinner The recording now completes once the user has spoken three sentences (SpeechProfileProvider.targetSentenceCount, counted on ./!/? boundaries followed by a space or the end of the text, so "3.5" is not one) instead of a word count, and the bar under the topics card fills per sentence. The progress-bar widget is renamed SpeechProgressBar to match. The Settings speech-profile page no longer swaps its Play/Redo or Get Started buttons for a spinner while the page initialises or the STT pre-flight runs; the buttons stay put and startRecording() ignores taps until the check finishes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): play the saved speech profile on the loudspeaker The app's audio session is normally configured for recording, so tapping Play on the Settings speech-profile page routed the WAV to the quiet earpiece. Before playing, configure a playback-category session (default mode, media usage on Android) and play at full volume, so the profile comes out of the main speaker like any other media. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): start every speech-profile recording with an empty transcript Tapping Redo showed the previous recording's words (and counted them toward the sentence target) because nothing cleared the provider's transcript before a new session; only close() did, on leaving the page. initialise() now calls a new resetTranscript() first, which forgets the segments, text, progress, completion and upload flags without touching the audio storage it recreates right after. resetSegments() reuses it. Test: a completed session's transcript is gone after resetTranscript and the fresh session counts sentences from zero and can finalize again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): finish the speech profile after a pause, keeping the last sentence on screen Reaching the third sentence finalized immediately, which stopped the mic mid-utterance (the recognizers punctuate each clip, so a pause can read as a sentence end) and swapped the transcript for a spinner at once. Now: - After the target is reached the provider waits completionGrace (2 s) without new speech before finalizing, restarting the wait on every new segment, and finalizes at completionCap (8 s past the target) at the latest. Once fired it does not re-arm; resetTranscript() clears it. - Both screens keep the last three transcript lines visible through the upload and the All done state, so the final sentence lingers instead of vanishing. Tests: grace/cap timing under fakeAsync, no double finalize, and the reset test now elapses the grace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(backend): give every speech-profile recording its own conversation Tapping Redo within two minutes of the previous attempt showed last time's words as soon as the user spoke again. The new listen socket attached to the still-open in-progress conversation from the previous attempt (same source, inside conversation_creation_timeout), so combine_segments() merged the first new segment into that conversation's last segment and the merged segment, old text included, was what the client received. LiveConversationController.prepare() now always creates a fresh in-progress conversation for onboarding_mode sessions (the onboarding step and the Settings redo both set it) instead of consulting the in-progress pointer. Ordinary listen sessions are unchanged. Test: test_listen_speech_profile_fresh_conversation.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): cross-fade the speech-profile recording UI into a plain All done button Keeping the transcript on screen through the upload made it pop back in on its own above the spinner and the All done button. The Settings page now cross-fades (450 ms) from the recording UI (transcript, topics card, bar) to nothing while uploading and then to the All done button, which is the same black capsule with a plain white border as the other buttons instead of the gradient box. The onboarding step likewise no longer shows the transcript in its upload and All done states. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * chore(app): remove the speech-samples page and dead progress-state code The Play button now plays the saved profile in place, so the samples page and its provider became unreachable (CI dead-code ratchet). Also drop the scroll controllers and SCROLL_DOWN signal the old clipped transcript used, and the word-count progress-message state (SpeechProfileProgressState, percentageCompleted, questionProgress) nothing reads any more. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the finished speech-profile recording on screen before All done After the third sentence the final words disappeared as soon as the upload began. Both screens now keep the finished recording (last words, topics card, full bar) on screen through the upload and for a further 1.5 s (allDoneHold) after the profile is saved, then cross-fade into the All done button. Onboarding's upload spinner row and its now-unused loading-text helper are gone; Skip for now hides once recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): fade the finished speech-profile recording out as one block Parts of the finished recording could change on their own before the cross-fade (the transcript and the mic disclaimer are built from live provider state that finalize() and its callbacks touch), so they did not disappear together. Both screens now snapshot the recording view (last words, no-device flag) the moment recording ends and build from that until a new recording starts, and the onboarding step's All done switch is now the same AnimatedSwitcher cross-fade as the Settings page, so the words, the topics card, the bar and the disclaimer fade out at the same time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the speech-profile bar full until the finished recording fades finalize() clears the provider's text once the profile is saved, and the bar derived its value from that text, so it dropped back to zero before the cross-fade. The frozen recording view now pins the bar at full from the moment recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): drop the speech-profile and memory-graph steps from first-run onboarding Onboarding now goes from Permissions straight to the completion screen. The speech profile is recorded from Settings instead, and the memory-graph preview (with its background graph prebuild) is gone. The two step widgets are deleted; their page indices stay as placeholders like the other retired steps so the existing page constants keep working. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): ease the mic glow shut as the finished speech profile fades The white glow behind the device graphic vanished the instant the upload began. It now stays through the upload and hold and eases down to nothing over the same 450 ms in which the recording view fades into All done. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the mic glow at its last size until it eases out The glow followed the live mic level, which drops to zero the instant the microphone stops after the recording ends, so it snapped down to its resting size before the ease-out. The frozen recording view now also captures the last mic level, so the glow holds still and then eases shut with the fade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * Revert "feat(app): drop the speech-profile and memory-graph steps from first-run onboarding" This reverts commit 4ebcb3d2c773747051dfeb2d519fe88aeb1faea0. * feat(backend): tune speaker verification from measured enrollments and retire the SpeechBrain matcher Speaker identification rejected most of the owner's own audio. The verification threshold (0.45 cosine distance) was copied from a clean-studio VoxCeleb figure; an offline bench over real enrollments in the speech-profiles bucket (229 users with a current profile plus an older one, 16 with extra recordings, 60 taught persons, 400 impostors; wespeaker-voxceleb-resnet34-LM, the diarizer's /v2/embedding model) puts same-user cross-session distance at a median of 0.40-0.53 and other users at 0.93. At 0.45 the owner was rejected 37-71% of the time at a 0.0% false-accept rate; the equal-error threshold is ~0.78. Same-session audio matched at either value, which is why the old constant looked fine in demos. - New utils/stt/speaker_match.py owns the policy (numpy only, shared by the live socket and the sync pipeline): threshold 0.65, plus a 0.10 margin over the runner-up so the owner is not guessed as a taught household member. - Live sessions pool up to three recent clips per diarized speaker and decide on the centroid once 5 s of clip audio has accumulated, instead of letting the first 2 s clip that lands under the threshold stick for the session. - Both surfaces log one structured speaker_id_decision line (best, runner-up, evidence, accepted) so the prod distribution can be checked against the bench from a day of logs. - The bench scripts live in backend/scripts/speaker_id_bench for reruns; user audio never leaves the machine running them. - Retire the dead SpeechBrain speaker-identification path: modal/speech_profile_modal, utils/stt/speech_profile (zero production callers), the /v1/speaker-identification route, HOSTED_SPEECH_PROFILE_API_URL in every chart/env, the speechbrain dependency, the shared-package COPY lines in the modal image, the dev-harness and e2e fakes, and the unused ListenLimits.speaker_id_target_audio field. Drop the now-unused is_same_speaker/find_best_match/bytes helpers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(backend): keep speaker_match real in sync test isolation, allowlist its import cost utils/sync/pipeline.py now imports utils.stt.speaker_match, but the hand-maintained heavy_deps mock list in test_sync_cloud_tasks.py and test_sync_v2.py didn't know about it, so `from utils.stt.speaker_match import select_speaker_match` raised ModuleNotFoundError: 'utils.stt' is not a package once utils.stt was replaced with a MagicMock. Real-import speaker_match (pure, dependency-free, like utils.stt.outcomes) instead of stubbing it, since a MagicMock decision object would also break the %.3f log formatting on decision.best_distance/runner_up_distance. Also allowlist test_speaker_match.py::test_short_clips_are_pooled_before_a_live_decision in the fast-unit duration guard: it's the first test in the file to import routers.listen.speakers, so it amortizes that module's FastAPI router-graph import cost, same structural pattern already documented for other files in the allowlist. * fix(speaker-id): preserve distinct evidence and household ambiguity Serialize live matches per speaker, subtract previously embedded audio, invalidate late session results, and keep all enrolled candidates in sync margin comparisons before enforcing unique assignment. Validation: 396 selected backend tests passed; Python typecheck has zero errors. Five live regression cases and two sync cases failed before the fixes. Changed sync expectations follow PR #12935's measured household-confusion margin. Failure-Class: new * fix(speaker-id): include owners in household benchmark cohorts Include available owner profiles even outside legacy, additional, and impostor cohorts. Distinguish offline benchmark evidence from deployed accuracy. Validation: synthetic manifest regression passes for owners outside other cohorts and people without an owner profile. No private audio or threshold retuning. Failure-Class: new * fix(speech-profile): bound native recognition and discard stale work Use one native completion owner so availability deadlines do not wait for shared model downloads and recognition cleanup precedes timeout completion. Serialize legacy recognition callbacks on the main queue. Propagate native failures to retain audio for retry; remove the polling Future timeout that allowed overlapping work. Scope fallback availability and polling results to their recording session. Validation: full Flutter suite 1836 passed, 5 skipped; analyzer ratchet passed. Native deadline behavioral tests pass and are registered in the existing manifest. Native speech code typechecks for iOS 15 deployment with Flutter boundary stubs; no full iPhone build or live enrollment claim. Preflight passed 53 selected checks. Failure-Class: new * fix(l10n): translate speech-profile flow in every supported locale Translate the eight speech-profile keys across all 48 non-English ARBs and fill two inherited missing keys exposed by generation. Use device-neutral speech recognition guidance and regenerate localization output from source catalogs. Validation: flutter gen-l10n reports zero untranslated messages; owner-name placeholders and complete catalog coverage verified. Full Flutter suite passed. Failure-Class: new * fix(speaker-id): require persisted speech profile before the redo admission bypass cubic P1: the client-supplied speech_profile_redo flag alone proved nothing; any authenticated client could send it to skip the completed-account onboarding-provenance admission gate. The runtime now confirms the redo from durable state (an actually stored speech_profile.wav) before taking the bypass, and an unprovable claim falls through to the provenance admission, failing closed when the check errors. Adds a regression test asserting a redo claim without a persisted profile is judged by the gate. * fix(listen): gate the onboarding fresh-conversation path on server admission cubic P2: onboarding=enabled is a client hint, yet prepare() took the fresh-conversation shortcut on the raw flag even when _bootstrap refused to admit the session — a client could dodge the existing-conversation lookup with a query parameter. The path now requires the runtime's onboarding_admitted (also true for the authorized Settings redo); an unadmitted claim keeps an ordinary session's behavior. Adds a regression test for the unadmitted path. * fix(onboarding): queue segments that arrive during AI answer checks cubic P2: is_checking_answer stayed set across up to three awaited LLM calls in _check_answer, and on_segments_received dropped everything spoken in that window, so answers covering later topics could be lost. Segments received while a check is in flight are now queued and replayed when it finishes, re-entering the normal accumulate-and-timer flow. Adds a regression test. * fix(speaker-id-bench): report the production threshold and true impostor rates cubic P2 x2: score.py evaluated the retired 0.45 operating point while the README and shipped policy (SPEAKER_MATCH_THRESHOLD) sit at 0.65, making its false-reject/false-accept and live-decision numbers misleading; and cohort-C impostor distances included the current user's own owner profile when that user was also sampled as an impostor, folding owner-vs-own-person confusion into the random-impostor sweep. score.py now pins T to the production 0.65 and formats every label from it; sweep.py excludes each cohort-C user's own profile from their impostor pool (the confusion keeps its dedicated diagnostic). Owner profiles for cohort C were already added to the cohort inputs by 8126713612. * fix(speech-profile): close startup, playback, and socket adoption races cubic review follow-ups still present after 1d7a2fd917: - page.dart: _isCheckingAvailability is now held until the entire startup path exits (dialogs, codec lookup, stopDeviceRecording, initialise), not just the availability round-trip, so a second tap cannot race socket and microphone init; context/mounted are rechecked after the language dialog and before initialise. - page.dart: profile playback deactivates the activated audio session on every teardown path (stop, natural completion, failure after activation, disposal) instead of leaving media routing active. - speech_profile_provider: a socket created while the session was closed or reset is discarded instead of adopted, which previously leaked a live backend session stop() never saw. - transcription_service: the speech-profile on-device fallback forwards raw audio per config.sendRawAudioToOmi, matching the conversation composite, instead of hardcoding every frame onto the Omi socket; suggested transcripts still flow and keep the backend session clock alive. Pinned by a factory test. * docs(app): keep AGENTS.md within its lean-budget ratchet after the main merge The merge combined this PR's on-device speech pointer with main's profile-build-mode and batch-contract lines, pushing app/AGENTS.md past its agents-md-lean budget (11747 > 11500 bytes). Tightens wording without dropping any fact: the batch-writer guarantee detail lives in the manifest reason and the ruby test itself; the other compressions are same-fact rewording. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Nathan Cheng <nathanjcx@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Nathan <nathan@Nathans-MacBook-Air.local> | 1 个月前 | |
fix(ios): reset transfer lease after native expiration | 16 天前 | |
feat(app): stream mobile voice reply playback | 16 天前 | |
feat(app): stream mobile voice reply playback | 16 天前 | |
feat(app): stream mobile voice reply playback | 16 天前 | |
test(ios): guard Watch chunk forwarding against holding the recording Moves the chunk-to-Flutter step into a stateless WatchAudioChunkRelay so a swiftc harness can prove every chunk is forwarded as it arrives and the empty end marker is dropped, and asserts AppDelegate keeps no chunk store (#20479). Adds the manifest check and the iOS changelog fragment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> | 3 天前 | |
initial refactor + backend added to folder | 2 年前 | |
initial refactor + backend added to folder | 2 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 年前 | ||
| 1 年前 | ||
| 6 天前 | ||
| 2 天前 | ||
| 10 天前 | ||
| 5 天前 | ||
| 16 天前 | ||
| 14 天前 | ||
| 14 天前 | ||
| 2 个月前 | ||
| 11 天前 | ||
| 2 个月前 | ||
| 2 天前 | ||
| 14 天前 | ||
| 4 个月前 | ||
| 8 天前 | ||
| 1 个月前 | ||
| 6 天前 | ||
| 6 天前 | ||
| 12 天前 | ||
| 10 天前 | ||
| 1 年前 | ||
| 16 天前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 1 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 14 天前 | ||
| 1 个月前 | ||
| 16 天前 | ||
| 16 天前 | ||
| 16 天前 | ||
| 16 天前 | ||
| 3 天前 | ||
| 2 年前 | ||
| 2 年前 |