| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(ios/widget) add accessoryCircular lock screen widget showing battery % | 4 个月前 | |
fix(app): address review on the untethered launch fix - Notice on the phone names the canonical `bash setup.sh ios` invocation from app/ instead of a repo-root path. - setup.sh warning describes the shipped behavior (engine-unavailable notice, not a crash) and only fires for the dev flavor, since its remedy builds the dev flavor. - Drop the GOOGLE_REVERSE_CLIENT_ID pin from devProfile/devRelease so Custom.xcconfig keeps owning that value. Failure-Class: none Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YU5BMEAfcFxNSWmLLDNfZ3 | 1 天前 | |
initial refactor + backend added to folder | 2 年前 | |
Speaker identification: measured threshold + margin, live clip pooling, SpeechBrain retirement; carries #12531 without the onboarding-step removal (#12935) * fix: unblock speech-profile redo and STT pre-flight for already-onboarded accounts Rebased onto origin/main as a single commit. Keep both main's open_provider_selection_circuit and this PR's is_stt_available helpers, then regenerate OpenAPI clients from the rebased backend. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(app): fade transcript words in as they arrive on the speech-profile screens Add FadeInWordsText: a centered word Wrap where only the words appended since the previous render animate from transparent to opaque with a short stagger, existing words stay put, and a rewritten transcript re-reveals from the start. Both the onboarding speech-profile step and the Settings redo page adopt it in the next commit so the live transcript reads the same whether the words come from the server or the on-device fallback. Verification: flutter test test/widgets/fade_in_words_text_test.dart (4 passed); observed on an iPhone 16 Pro via hot reload while dictating. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): fall back to on-device speech recognition when server STT is unavailable The speech-profile question flow (onboarding step and Settings redo) needs a transcript only to drive the questions and progress; the voice print itself is computed server-side from the WAV uploaded at finalize(). So when the backend's streaming STT is down, transcribe on the phone instead of dead-ending: - SpeechProfileProvider gains a local-STT mode. It is entered up front when the stt-availability pre-flight fails, or mid-session after the existing three 1011 closes with no captured speech (previously STT_UNAVAILABLE). The socket becomes the existing CompositeTranscriptionSocket: an on-device polling primary (Apple speech on iOS, downloaded Whisper on Android) forwarding suggested_transcript frames to the backend listen socket in custom_stt mode, which the OnboardingHandler already consumes like server STT output. No backend change; a receiver regression test pins that seam. - iOS on-device recognition hardening (AppDelegate.swift): resolve the app's bare language code to an installed on-device locale (a recognizer built from "en" failed every request with kAFAssistantErrorDomain 1101); reply exactly once per clip on final result, error, or a 20 s timeout, keeping partial results; and expose onDeviceAvailable, which probes a silent clip so a phone with Siri and Dictation disabled (kLSRErrorDomain 201) is reported as "no local STT" instead of entering the fallback blind. - PurePollingSocket bounds each transcribe() with a 30 s timeout. A provider that never answered left the processing flag set forever and silently stopped transcription for the rest of the session; now the audio is requeued and the next tick retries. This also protects the main app's on-device mode. - When neither server nor on-device STT is available, the pre-flight dialog now says to check the connection or turn on Dictation. - Speech-profile UI: subtler mic-level glow, and the live transcript uses the new fade-in words widget. Verification: - flutter test (full suite): 1731 passed, 5 skipped; scripts/analyze_ratchet.sh passed - new tests: speech_profile_provider_test (5 fallback cases), pure_polling_test (hung-provider timeout), fade_in_words_text_test (4), backend test_onboarding_question_start (suggested_transcript reaches the transcript queue only in custom-STT mode) - live on iPhone 16 Pro against the local dev harness with the STT primary forced unavailable: session connects with custom_stt+onboarding flags, Apple on-device recognition returns the spoken answer (~180 ms per clip) and it is forwarded to the backend; with Dictation disabled the probe reports unavailable and the dialog appears. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): use SpeechAnalyzer for on-device speech on iOS 26 SFSpeechRecognizer's on-device mode fails with kLSRErrorDomain 201 whenever Siri and Dictation are turned off in Settings, which is what produced the "turn on Dictation" pre-flight dialog in the speech-profile fallback. iOS 26's SpeechAnalyzer/SpeechTranscriber has no such dependency: the language model is an asset the app installs itself through AssetInventory. - transcribe: on iOS 26 run the clip through SpeechAnalyzer (preset .transcription, analyzeSequence(from:) + finalizeAndFinish), falling back to the SFSpeechRecognizer path only if the analyzer throws. - onDeviceAvailable: report true when a supported locale's model is installed or installs within 8 s; a longer download keeps going in the background and the first transcribe() waits for it. Concurrent callers share one download. - SFSpeechRecognizer remains the path for iOS 15-18. Verified on an iPhone (iOS 26.6.1) with Dictation off: the speech-profile redo enters local-STT mode and transcribes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): speech profile talks through three topics and completes on a word target Speech-profile recording (onboarding step and Settings redo) no longer walks one question at a time with a percentage bar. Instead: - A compact white-outlined card headed "Answer with your voice:" lists three topics (where you live, what you do for work, your long-term goal), and a thin bar under it fills as the user speaks. Reaching SpeechProfileProvider.targetWordCount (60 spoken words) finalizes the recording; the backend's onboarding_complete event no longer does, so "bar full" and "done" are the same moment. Omi's own question segments are excluded from the count. - The live transcript is bottom-anchored in a box exactly three lines tall above the card, so whole lines scroll off the top and nothing overlaps. - The Play button on the Settings page plays the saved profile audio in place (just_audio) and turns into Stop, instead of opening the samples page. Redo stops playback first. - Backend ONBOARDING_QUESTIONS is the same three topics, and OnboardingHandler keeps the transcript across questions so one stretch of speech can satisfy several of them. - Removed the unused percentage progress-bar widget and the "Skip this question" button; onboarding keeps "Skip for now". Tests: speech_profile_provider_test (word target fills, finalizes once, ignores Omi segments and the backend completion event); backend test_onboarding_talk_about_flow (one transcript answers every topic; the transcript is kept when it stops answering). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(dev-harness): advertise local-storage links on OMI_DEV_HOST A phone built against OMI_DEV_HOST could reach the backend but not the files it links to: OMI_LOCAL_STORAGE_BASE_URL was always http://127.0.0.1:<port>/_local/storage, so playing the saved speech profile from a device failed. The harness now derives a dev_advertise_host from OMI_DEV_HOST (loopback stays the default) and uses it only for that base URL; every other service address still binds and talks over loopback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the last three whole transcript lines instead of a clipped scroll The speech-profile transcript was a bottom-scrolled ListView clipped to a three-line box, so a sliver of the line above always showed at the top edge and read as cut-off text. FadeInWordsText now takes visibleLines: it replays the Wrap line breaking with measured word widths and builds only the words on the last N lines, so earlier lines drop off whole, nothing is clipped or scrolled, and words keep their reveal state while on screen. Both screens use visibleLines: 3 inside a fixed three-line, bottom-anchored area, moved a little further above the topics card. Test: fade_in_words_text_test covers short text showing everything, earlier lines dropping once the text exceeds three lines, and the shown words matching the line-break replay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): measure transcript lines with the effective text style; 40-word target FadeInWordsText replayed the Wrap line breaking with the caller's raw style, but each word's Text inherits the ambient DefaultTextStyle (font family, weight) under that style, so the replay undercounted lines and the real layout could reach four lines and draw over the topics card. Measure with the same merged style, and clip the fixed three-line area on both screens as a safety net so a stray line can never overlap the card. Also lower SpeechProfileProvider.targetWordCount from 60 to 40 so the recording finishes sooner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the "<Name>'s Speech Profile" title on one line The title wrapped onto two lines for longer names; it now scales down to fit a single line instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): never clip the speech-profile transcript; raise it above the card The three-line transcript area was a fixed-height clipped box, so whenever the rendered lines ran taller than the fontSize*height estimate (text scaling, font metrics) the top line was cut off. FadeInWordsText already guarantees at most three lines, so the area now only has a three-line minimum height (scaled with the text scaler) and grows to its content instead of clipping. Both screens also keep more space between the transcript and the topics card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): finish the speech profile after three sentences; no page-load spinner The recording now completes once the user has spoken three sentences (SpeechProfileProvider.targetSentenceCount, counted on ./!/? boundaries followed by a space or the end of the text, so "3.5" is not one) instead of a word count, and the bar under the topics card fills per sentence. The progress-bar widget is renamed SpeechProgressBar to match. The Settings speech-profile page no longer swaps its Play/Redo or Get Started buttons for a spinner while the page initialises or the STT pre-flight runs; the buttons stay put and startRecording() ignores taps until the check finishes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): play the saved speech profile on the loudspeaker The app's audio session is normally configured for recording, so tapping Play on the Settings speech-profile page routed the WAV to the quiet earpiece. Before playing, configure a playback-category session (default mode, media usage on Android) and play at full volume, so the profile comes out of the main speaker like any other media. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): start every speech-profile recording with an empty transcript Tapping Redo showed the previous recording's words (and counted them toward the sentence target) because nothing cleared the provider's transcript before a new session; only close() did, on leaving the page. initialise() now calls a new resetTranscript() first, which forgets the segments, text, progress, completion and upload flags without touching the audio storage it recreates right after. resetSegments() reuses it. Test: a completed session's transcript is gone after resetTranscript and the fresh session counts sentences from zero and can finalize again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): finish the speech profile after a pause, keeping the last sentence on screen Reaching the third sentence finalized immediately, which stopped the mic mid-utterance (the recognizers punctuate each clip, so a pause can read as a sentence end) and swapped the transcript for a spinner at once. Now: - After the target is reached the provider waits completionGrace (2 s) without new speech before finalizing, restarting the wait on every new segment, and finalizes at completionCap (8 s past the target) at the latest. Once fired it does not re-arm; resetTranscript() clears it. - Both screens keep the last three transcript lines visible through the upload and the All done state, so the final sentence lingers instead of vanishing. Tests: grace/cap timing under fakeAsync, no double finalize, and the reset test now elapses the grace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(backend): give every speech-profile recording its own conversation Tapping Redo within two minutes of the previous attempt showed last time's words as soon as the user spoke again. The new listen socket attached to the still-open in-progress conversation from the previous attempt (same source, inside conversation_creation_timeout), so combine_segments() merged the first new segment into that conversation's last segment and the merged segment, old text included, was what the client received. LiveConversationController.prepare() now always creates a fresh in-progress conversation for onboarding_mode sessions (the onboarding step and the Settings redo both set it) instead of consulting the in-progress pointer. Ordinary listen sessions are unchanged. Test: test_listen_speech_profile_fresh_conversation.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): cross-fade the speech-profile recording UI into a plain All done button Keeping the transcript on screen through the upload made it pop back in on its own above the spinner and the All done button. The Settings page now cross-fades (450 ms) from the recording UI (transcript, topics card, bar) to nothing while uploading and then to the All done button, which is the same black capsule with a plain white border as the other buttons instead of the gradient box. The onboarding step likewise no longer shows the transcript in its upload and All done states. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * chore(app): remove the speech-samples page and dead progress-state code The Play button now plays the saved profile in place, so the samples page and its provider became unreachable (CI dead-code ratchet). Also drop the scroll controllers and SCROLL_DOWN signal the old clipped transcript used, and the word-count progress-message state (SpeechProfileProgressState, percentageCompleted, questionProgress) nothing reads any more. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the finished speech-profile recording on screen before All done After the third sentence the final words disappeared as soon as the upload began. Both screens now keep the finished recording (last words, topics card, full bar) on screen through the upload and for a further 1.5 s (allDoneHold) after the profile is saved, then cross-fade into the All done button. Onboarding's upload spinner row and its now-unused loading-text helper are gone; Skip for now hides once recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): fade the finished speech-profile recording out as one block Parts of the finished recording could change on their own before the cross-fade (the transcript and the mic disclaimer are built from live provider state that finalize() and its callbacks touch), so they did not disappear together. Both screens now snapshot the recording view (last words, no-device flag) the moment recording ends and build from that until a new recording starts, and the onboarding step's All done switch is now the same AnimatedSwitcher cross-fade as the Settings page, so the words, the topics card, the bar and the disclaimer fade out at the same time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the speech-profile bar full until the finished recording fades finalize() clears the provider's text once the profile is saved, and the bar derived its value from that text, so it dropped back to zero before the cross-fade. The frozen recording view now pins the bar at full from the moment recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): drop the speech-profile and memory-graph steps from first-run onboarding Onboarding now goes from Permissions straight to the completion screen. The speech profile is recorded from Settings instead, and the memory-graph preview (with its background graph prebuild) is gone. The two step widgets are deleted; their page indices stay as placeholders like the other retired steps so the existing page constants keep working. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): ease the mic glow shut as the finished speech profile fades The white glow behind the device graphic vanished the instant the upload began. It now stays through the upload and hold and eases down to nothing over the same 450 ms in which the recording view fades into All done. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the mic glow at its last size until it eases out The glow followed the live mic level, which drops to zero the instant the microphone stops after the recording ends, so it snapped down to its resting size before the ease-out. The frozen recording view now also captures the last mic level, so the glow holds still and then eases shut with the fade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * Revert "feat(app): drop the speech-profile and memory-graph steps from first-run onboarding" This reverts commit 4ebcb3d2c773747051dfeb2d519fe88aeb1faea0. * feat(backend): tune speaker verification from measured enrollments and retire the SpeechBrain matcher Speaker identification rejected most of the owner's own audio. The verification threshold (0.45 cosine distance) was copied from a clean-studio VoxCeleb figure; an offline bench over real enrollments in the speech-profiles bucket (229 users with a current profile plus an older one, 16 with extra recordings, 60 taught persons, 400 impostors; wespeaker-voxceleb-resnet34-LM, the diarizer's /v2/embedding model) puts same-user cross-session distance at a median of 0.40-0.53 and other users at 0.93. At 0.45 the owner was rejected 37-71% of the time at a 0.0% false-accept rate; the equal-error threshold is ~0.78. Same-session audio matched at either value, which is why the old constant looked fine in demos. - New utils/stt/speaker_match.py owns the policy (numpy only, shared by the live socket and the sync pipeline): threshold 0.65, plus a 0.10 margin over the runner-up so the owner is not guessed as a taught household member. - Live sessions pool up to three recent clips per diarized speaker and decide on the centroid once 5 s of clip audio has accumulated, instead of letting the first 2 s clip that lands under the threshold stick for the session. - Both surfaces log one structured speaker_id_decision line (best, runner-up, evidence, accepted) so the prod distribution can be checked against the bench from a day of logs. - The bench scripts live in backend/scripts/speaker_id_bench for reruns; user audio never leaves the machine running them. - Retire the dead SpeechBrain speaker-identification path: modal/speech_profile_modal, utils/stt/speech_profile (zero production callers), the /v1/speaker-identification route, HOSTED_SPEECH_PROFILE_API_URL in every chart/env, the speechbrain dependency, the shared-package COPY lines in the modal image, the dev-harness and e2e fakes, and the unused ListenLimits.speaker_id_target_audio field. Drop the now-unused is_same_speaker/find_best_match/bytes helpers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(backend): keep speaker_match real in sync test isolation, allowlist its import cost utils/sync/pipeline.py now imports utils.stt.speaker_match, but the hand-maintained heavy_deps mock list in test_sync_cloud_tasks.py and test_sync_v2.py didn't know about it, so `from utils.stt.speaker_match import select_speaker_match` raised ModuleNotFoundError: 'utils.stt' is not a package once utils.stt was replaced with a MagicMock. Real-import speaker_match (pure, dependency-free, like utils.stt.outcomes) instead of stubbing it, since a MagicMock decision object would also break the %.3f log formatting on decision.best_distance/runner_up_distance. Also allowlist test_speaker_match.py::test_short_clips_are_pooled_before_a_live_decision in the fast-unit duration guard: it's the first test in the file to import routers.listen.speakers, so it amortizes that module's FastAPI router-graph import cost, same structural pattern already documented for other files in the allowlist. * fix(speaker-id): preserve distinct evidence and household ambiguity Serialize live matches per speaker, subtract previously embedded audio, invalidate late session results, and keep all enrolled candidates in sync margin comparisons before enforcing unique assignment. Validation: 396 selected backend tests passed; Python typecheck has zero errors. Five live regression cases and two sync cases failed before the fixes. Changed sync expectations follow PR #12935's measured household-confusion margin. Failure-Class: new * fix(speaker-id): include owners in household benchmark cohorts Include available owner profiles even outside legacy, additional, and impostor cohorts. Distinguish offline benchmark evidence from deployed accuracy. Validation: synthetic manifest regression passes for owners outside other cohorts and people without an owner profile. No private audio or threshold retuning. Failure-Class: new * fix(speech-profile): bound native recognition and discard stale work Use one native completion owner so availability deadlines do not wait for shared model downloads and recognition cleanup precedes timeout completion. Serialize legacy recognition callbacks on the main queue. Propagate native failures to retain audio for retry; remove the polling Future timeout that allowed overlapping work. Scope fallback availability and polling results to their recording session. Validation: full Flutter suite 1836 passed, 5 skipped; analyzer ratchet passed. Native deadline behavioral tests pass and are registered in the existing manifest. Native speech code typechecks for iOS 15 deployment with Flutter boundary stubs; no full iPhone build or live enrollment claim. Preflight passed 53 selected checks. Failure-Class: new * fix(l10n): translate speech-profile flow in every supported locale Translate the eight speech-profile keys across all 48 non-English ARBs and fill two inherited missing keys exposed by generation. Use device-neutral speech recognition guidance and regenerate localization output from source catalogs. Validation: flutter gen-l10n reports zero untranslated messages; owner-name placeholders and complete catalog coverage verified. Full Flutter suite passed. Failure-Class: new * fix(speaker-id): require persisted speech profile before the redo admission bypass cubic P1: the client-supplied speech_profile_redo flag alone proved nothing; any authenticated client could send it to skip the completed-account onboarding-provenance admission gate. The runtime now confirms the redo from durable state (an actually stored speech_profile.wav) before taking the bypass, and an unprovable claim falls through to the provenance admission, failing closed when the check errors. Adds a regression test asserting a redo claim without a persisted profile is judged by the gate. * fix(listen): gate the onboarding fresh-conversation path on server admission cubic P2: onboarding=enabled is a client hint, yet prepare() took the fresh-conversation shortcut on the raw flag even when _bootstrap refused to admit the session — a client could dodge the existing-conversation lookup with a query parameter. The path now requires the runtime's onboarding_admitted (also true for the authorized Settings redo); an unadmitted claim keeps an ordinary session's behavior. Adds a regression test for the unadmitted path. * fix(onboarding): queue segments that arrive during AI answer checks cubic P2: is_checking_answer stayed set across up to three awaited LLM calls in _check_answer, and on_segments_received dropped everything spoken in that window, so answers covering later topics could be lost. Segments received while a check is in flight are now queued and replayed when it finishes, re-entering the normal accumulate-and-timer flow. Adds a regression test. * fix(speaker-id-bench): report the production threshold and true impostor rates cubic P2 x2: score.py evaluated the retired 0.45 operating point while the README and shipped policy (SPEAKER_MATCH_THRESHOLD) sit at 0.65, making its false-reject/false-accept and live-decision numbers misleading; and cohort-C impostor distances included the current user's own owner profile when that user was also sampled as an impostor, folding owner-vs-own-person confusion into the random-impostor sweep. score.py now pins T to the production 0.65 and formats every label from it; sweep.py excludes each cohort-C user's own profile from their impostor pool (the confusion keeps its dedicated diagnostic). Owner profiles for cohort C were already added to the cohort inputs by 8126713612. * fix(speech-profile): close startup, playback, and socket adoption races cubic review follow-ups still present after 1d7a2fd917: - page.dart: _isCheckingAvailability is now held until the entire startup path exits (dialogs, codec lookup, stopDeviceRecording, initialise), not just the availability round-trip, so a second tap cannot race socket and microphone init; context/mounted are rechecked after the language dialog and before initialise. - page.dart: profile playback deactivates the activated audio session on every teardown path (stop, natural completion, failure after activation, disposal) instead of leaving media routing active. - speech_profile_provider: a socket created while the session was closed or reset is discarded instead of adopted, which previously leaked a live backend session stop() never saw. - transcription_service: the speech-profile on-device fallback forwards raw audio per config.sendRawAudioToOmi, matching the conversation composite, instead of hardcoding every frame onto the Omi socket; suggested transcripts still flow and keep the backend session clock alive. Pinned by a factory test. * docs(app): keep AGENTS.md within its lean-budget ratchet after the main merge The merge combined this PR's on-device speech pointer with main's profile-build-mode and batch-contract lines, pushing app/AGENTS.md past its agents-md-lean budget (11747 > 11500 bytes). Tightens wording without dropping any fact: the batch-writer guarantee detail lives in the manifest reason and the ruby test itself; the other compressions are same-fact rewording. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Nathan Cheng <nathanjcx@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Nathan <nathan@Nathans-MacBook-Air.local> | 1 天前 | |
Speaker identification: measured threshold + margin, live clip pooling, SpeechBrain retirement; carries #12531 without the onboarding-step removal (#12935) * fix: unblock speech-profile redo and STT pre-flight for already-onboarded accounts Rebased onto origin/main as a single commit. Keep both main's open_provider_selection_circuit and this PR's is_stt_available helpers, then regenerate OpenAPI clients from the rebased backend. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(app): fade transcript words in as they arrive on the speech-profile screens Add FadeInWordsText: a centered word Wrap where only the words appended since the previous render animate from transparent to opaque with a short stagger, existing words stay put, and a rewritten transcript re-reveals from the start. Both the onboarding speech-profile step and the Settings redo page adopt it in the next commit so the live transcript reads the same whether the words come from the server or the on-device fallback. Verification: flutter test test/widgets/fade_in_words_text_test.dart (4 passed); observed on an iPhone 16 Pro via hot reload while dictating. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): fall back to on-device speech recognition when server STT is unavailable The speech-profile question flow (onboarding step and Settings redo) needs a transcript only to drive the questions and progress; the voice print itself is computed server-side from the WAV uploaded at finalize(). So when the backend's streaming STT is down, transcribe on the phone instead of dead-ending: - SpeechProfileProvider gains a local-STT mode. It is entered up front when the stt-availability pre-flight fails, or mid-session after the existing three 1011 closes with no captured speech (previously STT_UNAVAILABLE). The socket becomes the existing CompositeTranscriptionSocket: an on-device polling primary (Apple speech on iOS, downloaded Whisper on Android) forwarding suggested_transcript frames to the backend listen socket in custom_stt mode, which the OnboardingHandler already consumes like server STT output. No backend change; a receiver regression test pins that seam. - iOS on-device recognition hardening (AppDelegate.swift): resolve the app's bare language code to an installed on-device locale (a recognizer built from "en" failed every request with kAFAssistantErrorDomain 1101); reply exactly once per clip on final result, error, or a 20 s timeout, keeping partial results; and expose onDeviceAvailable, which probes a silent clip so a phone with Siri and Dictation disabled (kLSRErrorDomain 201) is reported as "no local STT" instead of entering the fallback blind. - PurePollingSocket bounds each transcribe() with a 30 s timeout. A provider that never answered left the processing flag set forever and silently stopped transcription for the rest of the session; now the audio is requeued and the next tick retries. This also protects the main app's on-device mode. - When neither server nor on-device STT is available, the pre-flight dialog now says to check the connection or turn on Dictation. - Speech-profile UI: subtler mic-level glow, and the live transcript uses the new fade-in words widget. Verification: - flutter test (full suite): 1731 passed, 5 skipped; scripts/analyze_ratchet.sh passed - new tests: speech_profile_provider_test (5 fallback cases), pure_polling_test (hung-provider timeout), fade_in_words_text_test (4), backend test_onboarding_question_start (suggested_transcript reaches the transcript queue only in custom-STT mode) - live on iPhone 16 Pro against the local dev harness with the STT primary forced unavailable: session connects with custom_stt+onboarding flags, Apple on-device recognition returns the spoken answer (~180 ms per clip) and it is forwarded to the backend; with Dictation disabled the probe reports unavailable and the dialog appears. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): use SpeechAnalyzer for on-device speech on iOS 26 SFSpeechRecognizer's on-device mode fails with kLSRErrorDomain 201 whenever Siri and Dictation are turned off in Settings, which is what produced the "turn on Dictation" pre-flight dialog in the speech-profile fallback. iOS 26's SpeechAnalyzer/SpeechTranscriber has no such dependency: the language model is an asset the app installs itself through AssetInventory. - transcribe: on iOS 26 run the clip through SpeechAnalyzer (preset .transcription, analyzeSequence(from:) + finalizeAndFinish), falling back to the SFSpeechRecognizer path only if the analyzer throws. - onDeviceAvailable: report true when a supported locale's model is installed or installs within 8 s; a longer download keeps going in the background and the first transcribe() waits for it. Concurrent callers share one download. - SFSpeechRecognizer remains the path for iOS 15-18. Verified on an iPhone (iOS 26.6.1) with Dictation off: the speech-profile redo enters local-STT mode and transcribes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): speech profile talks through three topics and completes on a word target Speech-profile recording (onboarding step and Settings redo) no longer walks one question at a time with a percentage bar. Instead: - A compact white-outlined card headed "Answer with your voice:" lists three topics (where you live, what you do for work, your long-term goal), and a thin bar under it fills as the user speaks. Reaching SpeechProfileProvider.targetWordCount (60 spoken words) finalizes the recording; the backend's onboarding_complete event no longer does, so "bar full" and "done" are the same moment. Omi's own question segments are excluded from the count. - The live transcript is bottom-anchored in a box exactly three lines tall above the card, so whole lines scroll off the top and nothing overlaps. - The Play button on the Settings page plays the saved profile audio in place (just_audio) and turns into Stop, instead of opening the samples page. Redo stops playback first. - Backend ONBOARDING_QUESTIONS is the same three topics, and OnboardingHandler keeps the transcript across questions so one stretch of speech can satisfy several of them. - Removed the unused percentage progress-bar widget and the "Skip this question" button; onboarding keeps "Skip for now". Tests: speech_profile_provider_test (word target fills, finalizes once, ignores Omi segments and the backend completion event); backend test_onboarding_talk_about_flow (one transcript answers every topic; the transcript is kept when it stops answering). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(dev-harness): advertise local-storage links on OMI_DEV_HOST A phone built against OMI_DEV_HOST could reach the backend but not the files it links to: OMI_LOCAL_STORAGE_BASE_URL was always http://127.0.0.1:<port>/_local/storage, so playing the saved speech profile from a device failed. The harness now derives a dev_advertise_host from OMI_DEV_HOST (loopback stays the default) and uses it only for that base URL; every other service address still binds and talks over loopback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the last three whole transcript lines instead of a clipped scroll The speech-profile transcript was a bottom-scrolled ListView clipped to a three-line box, so a sliver of the line above always showed at the top edge and read as cut-off text. FadeInWordsText now takes visibleLines: it replays the Wrap line breaking with measured word widths and builds only the words on the last N lines, so earlier lines drop off whole, nothing is clipped or scrolled, and words keep their reveal state while on screen. Both screens use visibleLines: 3 inside a fixed three-line, bottom-anchored area, moved a little further above the topics card. Test: fade_in_words_text_test covers short text showing everything, earlier lines dropping once the text exceeds three lines, and the shown words matching the line-break replay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): measure transcript lines with the effective text style; 40-word target FadeInWordsText replayed the Wrap line breaking with the caller's raw style, but each word's Text inherits the ambient DefaultTextStyle (font family, weight) under that style, so the replay undercounted lines and the real layout could reach four lines and draw over the topics card. Measure with the same merged style, and clip the fixed three-line area on both screens as a safety net so a stray line can never overlap the card. Also lower SpeechProfileProvider.targetWordCount from 60 to 40 so the recording finishes sooner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the "<Name>'s Speech Profile" title on one line The title wrapped onto two lines for longer names; it now scales down to fit a single line instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): never clip the speech-profile transcript; raise it above the card The three-line transcript area was a fixed-height clipped box, so whenever the rendered lines ran taller than the fontSize*height estimate (text scaling, font metrics) the top line was cut off. FadeInWordsText already guarantees at most three lines, so the area now only has a three-line minimum height (scaled with the text scaler) and grows to its content instead of clipping. Both screens also keep more space between the transcript and the topics card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): finish the speech profile after three sentences; no page-load spinner The recording now completes once the user has spoken three sentences (SpeechProfileProvider.targetSentenceCount, counted on ./!/? boundaries followed by a space or the end of the text, so "3.5" is not one) instead of a word count, and the bar under the topics card fills per sentence. The progress-bar widget is renamed SpeechProgressBar to match. The Settings speech-profile page no longer swaps its Play/Redo or Get Started buttons for a spinner while the page initialises or the STT pre-flight runs; the buttons stay put and startRecording() ignores taps until the check finishes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): play the saved speech profile on the loudspeaker The app's audio session is normally configured for recording, so tapping Play on the Settings speech-profile page routed the WAV to the quiet earpiece. Before playing, configure a playback-category session (default mode, media usage on Android) and play at full volume, so the profile comes out of the main speaker like any other media. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): start every speech-profile recording with an empty transcript Tapping Redo showed the previous recording's words (and counted them toward the sentence target) because nothing cleared the provider's transcript before a new session; only close() did, on leaving the page. initialise() now calls a new resetTranscript() first, which forgets the segments, text, progress, completion and upload flags without touching the audio storage it recreates right after. resetSegments() reuses it. Test: a completed session's transcript is gone after resetTranscript and the fresh session counts sentences from zero and can finalize again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): finish the speech profile after a pause, keeping the last sentence on screen Reaching the third sentence finalized immediately, which stopped the mic mid-utterance (the recognizers punctuate each clip, so a pause can read as a sentence end) and swapped the transcript for a spinner at once. Now: - After the target is reached the provider waits completionGrace (2 s) without new speech before finalizing, restarting the wait on every new segment, and finalizes at completionCap (8 s past the target) at the latest. Once fired it does not re-arm; resetTranscript() clears it. - Both screens keep the last three transcript lines visible through the upload and the All done state, so the final sentence lingers instead of vanishing. Tests: grace/cap timing under fakeAsync, no double finalize, and the reset test now elapses the grace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(backend): give every speech-profile recording its own conversation Tapping Redo within two minutes of the previous attempt showed last time's words as soon as the user spoke again. The new listen socket attached to the still-open in-progress conversation from the previous attempt (same source, inside conversation_creation_timeout), so combine_segments() merged the first new segment into that conversation's last segment and the merged segment, old text included, was what the client received. LiveConversationController.prepare() now always creates a fresh in-progress conversation for onboarding_mode sessions (the onboarding step and the Settings redo both set it) instead of consulting the in-progress pointer. Ordinary listen sessions are unchanged. Test: test_listen_speech_profile_fresh_conversation.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): cross-fade the speech-profile recording UI into a plain All done button Keeping the transcript on screen through the upload made it pop back in on its own above the spinner and the All done button. The Settings page now cross-fades (450 ms) from the recording UI (transcript, topics card, bar) to nothing while uploading and then to the All done button, which is the same black capsule with a plain white border as the other buttons instead of the gradient box. The onboarding step likewise no longer shows the transcript in its upload and All done states. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * chore(app): remove the speech-samples page and dead progress-state code The Play button now plays the saved profile in place, so the samples page and its provider became unreachable (CI dead-code ratchet). Also drop the scroll controllers and SCROLL_DOWN signal the old clipped transcript used, and the word-count progress-message state (SpeechProfileProgressState, percentageCompleted, questionProgress) nothing reads any more. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the finished speech-profile recording on screen before All done After the third sentence the final words disappeared as soon as the upload began. Both screens now keep the finished recording (last words, topics card, full bar) on screen through the upload and for a further 1.5 s (allDoneHold) after the profile is saved, then cross-fade into the All done button. Onboarding's upload spinner row and its now-unused loading-text helper are gone; Skip for now hides once recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): fade the finished speech-profile recording out as one block Parts of the finished recording could change on their own before the cross-fade (the transcript and the mic disclaimer are built from live provider state that finalize() and its callbacks touch), so they did not disappear together. Both screens now snapshot the recording view (last words, no-device flag) the moment recording ends and build from that until a new recording starts, and the onboarding step's All done switch is now the same AnimatedSwitcher cross-fade as the Settings page, so the words, the topics card, the bar and the disclaimer fade out at the same time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the speech-profile bar full until the finished recording fades finalize() clears the provider's text once the profile is saved, and the bar derived its value from that text, so it dropped back to zero before the cross-fade. The frozen recording view now pins the bar at full from the moment recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): drop the speech-profile and memory-graph steps from first-run onboarding Onboarding now goes from Permissions straight to the completion screen. The speech profile is recorded from Settings instead, and the memory-graph preview (with its background graph prebuild) is gone. The two step widgets are deleted; their page indices stay as placeholders like the other retired steps so the existing page constants keep working. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): ease the mic glow shut as the finished speech profile fades The white glow behind the device graphic vanished the instant the upload began. It now stays through the upload and hold and eases down to nothing over the same 450 ms in which the recording view fades into All done. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the mic glow at its last size until it eases out The glow followed the live mic level, which drops to zero the instant the microphone stops after the recording ends, so it snapped down to its resting size before the ease-out. The frozen recording view now also captures the last mic level, so the glow holds still and then eases shut with the fade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * Revert "feat(app): drop the speech-profile and memory-graph steps from first-run onboarding" This reverts commit 4ebcb3d2c773747051dfeb2d519fe88aeb1faea0. * feat(backend): tune speaker verification from measured enrollments and retire the SpeechBrain matcher Speaker identification rejected most of the owner's own audio. The verification threshold (0.45 cosine distance) was copied from a clean-studio VoxCeleb figure; an offline bench over real enrollments in the speech-profiles bucket (229 users with a current profile plus an older one, 16 with extra recordings, 60 taught persons, 400 impostors; wespeaker-voxceleb-resnet34-LM, the diarizer's /v2/embedding model) puts same-user cross-session distance at a median of 0.40-0.53 and other users at 0.93. At 0.45 the owner was rejected 37-71% of the time at a 0.0% false-accept rate; the equal-error threshold is ~0.78. Same-session audio matched at either value, which is why the old constant looked fine in demos. - New utils/stt/speaker_match.py owns the policy (numpy only, shared by the live socket and the sync pipeline): threshold 0.65, plus a 0.10 margin over the runner-up so the owner is not guessed as a taught household member. - Live sessions pool up to three recent clips per diarized speaker and decide on the centroid once 5 s of clip audio has accumulated, instead of letting the first 2 s clip that lands under the threshold stick for the session. - Both surfaces log one structured speaker_id_decision line (best, runner-up, evidence, accepted) so the prod distribution can be checked against the bench from a day of logs. - The bench scripts live in backend/scripts/speaker_id_bench for reruns; user audio never leaves the machine running them. - Retire the dead SpeechBrain speaker-identification path: modal/speech_profile_modal, utils/stt/speech_profile (zero production callers), the /v1/speaker-identification route, HOSTED_SPEECH_PROFILE_API_URL in every chart/env, the speechbrain dependency, the shared-package COPY lines in the modal image, the dev-harness and e2e fakes, and the unused ListenLimits.speaker_id_target_audio field. Drop the now-unused is_same_speaker/find_best_match/bytes helpers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(backend): keep speaker_match real in sync test isolation, allowlist its import cost utils/sync/pipeline.py now imports utils.stt.speaker_match, but the hand-maintained heavy_deps mock list in test_sync_cloud_tasks.py and test_sync_v2.py didn't know about it, so `from utils.stt.speaker_match import select_speaker_match` raised ModuleNotFoundError: 'utils.stt' is not a package once utils.stt was replaced with a MagicMock. Real-import speaker_match (pure, dependency-free, like utils.stt.outcomes) instead of stubbing it, since a MagicMock decision object would also break the %.3f log formatting on decision.best_distance/runner_up_distance. Also allowlist test_speaker_match.py::test_short_clips_are_pooled_before_a_live_decision in the fast-unit duration guard: it's the first test in the file to import routers.listen.speakers, so it amortizes that module's FastAPI router-graph import cost, same structural pattern already documented for other files in the allowlist. * fix(speaker-id): preserve distinct evidence and household ambiguity Serialize live matches per speaker, subtract previously embedded audio, invalidate late session results, and keep all enrolled candidates in sync margin comparisons before enforcing unique assignment. Validation: 396 selected backend tests passed; Python typecheck has zero errors. Five live regression cases and two sync cases failed before the fixes. Changed sync expectations follow PR #12935's measured household-confusion margin. Failure-Class: new * fix(speaker-id): include owners in household benchmark cohorts Include available owner profiles even outside legacy, additional, and impostor cohorts. Distinguish offline benchmark evidence from deployed accuracy. Validation: synthetic manifest regression passes for owners outside other cohorts and people without an owner profile. No private audio or threshold retuning. Failure-Class: new * fix(speech-profile): bound native recognition and discard stale work Use one native completion owner so availability deadlines do not wait for shared model downloads and recognition cleanup precedes timeout completion. Serialize legacy recognition callbacks on the main queue. Propagate native failures to retain audio for retry; remove the polling Future timeout that allowed overlapping work. Scope fallback availability and polling results to their recording session. Validation: full Flutter suite 1836 passed, 5 skipped; analyzer ratchet passed. Native deadline behavioral tests pass and are registered in the existing manifest. Native speech code typechecks for iOS 15 deployment with Flutter boundary stubs; no full iPhone build or live enrollment claim. Preflight passed 53 selected checks. Failure-Class: new * fix(l10n): translate speech-profile flow in every supported locale Translate the eight speech-profile keys across all 48 non-English ARBs and fill two inherited missing keys exposed by generation. Use device-neutral speech recognition guidance and regenerate localization output from source catalogs. Validation: flutter gen-l10n reports zero untranslated messages; owner-name placeholders and complete catalog coverage verified. Full Flutter suite passed. Failure-Class: new * fix(speaker-id): require persisted speech profile before the redo admission bypass cubic P1: the client-supplied speech_profile_redo flag alone proved nothing; any authenticated client could send it to skip the completed-account onboarding-provenance admission gate. The runtime now confirms the redo from durable state (an actually stored speech_profile.wav) before taking the bypass, and an unprovable claim falls through to the provenance admission, failing closed when the check errors. Adds a regression test asserting a redo claim without a persisted profile is judged by the gate. * fix(listen): gate the onboarding fresh-conversation path on server admission cubic P2: onboarding=enabled is a client hint, yet prepare() took the fresh-conversation shortcut on the raw flag even when _bootstrap refused to admit the session — a client could dodge the existing-conversation lookup with a query parameter. The path now requires the runtime's onboarding_admitted (also true for the authorized Settings redo); an unadmitted claim keeps an ordinary session's behavior. Adds a regression test for the unadmitted path. * fix(onboarding): queue segments that arrive during AI answer checks cubic P2: is_checking_answer stayed set across up to three awaited LLM calls in _check_answer, and on_segments_received dropped everything spoken in that window, so answers covering later topics could be lost. Segments received while a check is in flight are now queued and replayed when it finishes, re-entering the normal accumulate-and-timer flow. Adds a regression test. * fix(speaker-id-bench): report the production threshold and true impostor rates cubic P2 x2: score.py evaluated the retired 0.45 operating point while the README and shipped policy (SPEAKER_MATCH_THRESHOLD) sit at 0.65, making its false-reject/false-accept and live-decision numbers misleading; and cohort-C impostor distances included the current user's own owner profile when that user was also sampled as an impostor, folding owner-vs-own-person confusion into the random-impostor sweep. score.py now pins T to the production 0.65 and formats every label from it; sweep.py excludes each cohort-C user's own profile from their impostor pool (the confusion keeps its dedicated diagnostic). Owner profiles for cohort C were already added to the cohort inputs by 8126713612. * fix(speech-profile): close startup, playback, and socket adoption races cubic review follow-ups still present after 1d7a2fd917: - page.dart: _isCheckingAvailability is now held until the entire startup path exits (dialogs, codec lookup, stopDeviceRecording, initialise), not just the availability round-trip, so a second tap cannot race socket and microphone init; context/mounted are rechecked after the language dialog and before initialise. - page.dart: profile playback deactivates the activated audio session on every teardown path (stop, natural completion, failure after activation, disposal) instead of leaving media routing active. - speech_profile_provider: a socket created while the session was closed or reset is discarded instead of adopted, which previously leaked a live backend session stop() never saw. - transcription_service: the speech-profile on-device fallback forwards raw audio per config.sendRawAudioToOmi, matching the conversation composite, instead of hardcoding every frame onto the Omi socket; suggested transcripts still flow and keep the backend session clock alive. Pinned by a factory test. * docs(app): keep AGENTS.md within its lean-budget ratchet after the main merge The merge combined this PR's on-device speech pointer with main's profile-build-mode and batch-contract lines, pushing app/AGENTS.md past its agents-md-lean budget (11747 > 11500 bytes). Tightens wording without dropping any fact: the batch-writer guarantee detail lives in the manifest reason and the ruby test itself; the other compressions are same-fact rewording. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Nathan Cheng <nathanjcx@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Nathan <nathan@Nathans-MacBook-Air.local> | 1 天前 | |
fix(watch): harden recording presentation review guards Failure-Class: none Co-authored-by: multica-agent <github@multica.ai> | 1 个月前 | |
Speaker identification: measured threshold + margin, live clip pooling, SpeechBrain retirement; carries #12531 without the onboarding-step removal (#12935) * fix: unblock speech-profile redo and STT pre-flight for already-onboarded accounts Rebased onto origin/main as a single commit. Keep both main's open_provider_selection_circuit and this PR's is_stt_available helpers, then regenerate OpenAPI clients from the rebased backend. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(app): fade transcript words in as they arrive on the speech-profile screens Add FadeInWordsText: a centered word Wrap where only the words appended since the previous render animate from transparent to opaque with a short stagger, existing words stay put, and a rewritten transcript re-reveals from the start. Both the onboarding speech-profile step and the Settings redo page adopt it in the next commit so the live transcript reads the same whether the words come from the server or the on-device fallback. Verification: flutter test test/widgets/fade_in_words_text_test.dart (4 passed); observed on an iPhone 16 Pro via hot reload while dictating. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): fall back to on-device speech recognition when server STT is unavailable The speech-profile question flow (onboarding step and Settings redo) needs a transcript only to drive the questions and progress; the voice print itself is computed server-side from the WAV uploaded at finalize(). So when the backend's streaming STT is down, transcribe on the phone instead of dead-ending: - SpeechProfileProvider gains a local-STT mode. It is entered up front when the stt-availability pre-flight fails, or mid-session after the existing three 1011 closes with no captured speech (previously STT_UNAVAILABLE). The socket becomes the existing CompositeTranscriptionSocket: an on-device polling primary (Apple speech on iOS, downloaded Whisper on Android) forwarding suggested_transcript frames to the backend listen socket in custom_stt mode, which the OnboardingHandler already consumes like server STT output. No backend change; a receiver regression test pins that seam. - iOS on-device recognition hardening (AppDelegate.swift): resolve the app's bare language code to an installed on-device locale (a recognizer built from "en" failed every request with kAFAssistantErrorDomain 1101); reply exactly once per clip on final result, error, or a 20 s timeout, keeping partial results; and expose onDeviceAvailable, which probes a silent clip so a phone with Siri and Dictation disabled (kLSRErrorDomain 201) is reported as "no local STT" instead of entering the fallback blind. - PurePollingSocket bounds each transcribe() with a 30 s timeout. A provider that never answered left the processing flag set forever and silently stopped transcription for the rest of the session; now the audio is requeued and the next tick retries. This also protects the main app's on-device mode. - When neither server nor on-device STT is available, the pre-flight dialog now says to check the connection or turn on Dictation. - Speech-profile UI: subtler mic-level glow, and the live transcript uses the new fade-in words widget. Verification: - flutter test (full suite): 1731 passed, 5 skipped; scripts/analyze_ratchet.sh passed - new tests: speech_profile_provider_test (5 fallback cases), pure_polling_test (hung-provider timeout), fade_in_words_text_test (4), backend test_onboarding_question_start (suggested_transcript reaches the transcript queue only in custom-STT mode) - live on iPhone 16 Pro against the local dev harness with the STT primary forced unavailable: session connects with custom_stt+onboarding flags, Apple on-device recognition returns the spoken answer (~180 ms per clip) and it is forwarded to the backend; with Dictation disabled the probe reports unavailable and the dialog appears. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1zmRTRh3DV8NsM3QUrrAB * feat(app): use SpeechAnalyzer for on-device speech on iOS 26 SFSpeechRecognizer's on-device mode fails with kLSRErrorDomain 201 whenever Siri and Dictation are turned off in Settings, which is what produced the "turn on Dictation" pre-flight dialog in the speech-profile fallback. iOS 26's SpeechAnalyzer/SpeechTranscriber has no such dependency: the language model is an asset the app installs itself through AssetInventory. - transcribe: on iOS 26 run the clip through SpeechAnalyzer (preset .transcription, analyzeSequence(from:) + finalizeAndFinish), falling back to the SFSpeechRecognizer path only if the analyzer throws. - onDeviceAvailable: report true when a supported locale's model is installed or installs within 8 s; a longer download keeps going in the background and the first transcribe() waits for it. Concurrent callers share one download. - SFSpeechRecognizer remains the path for iOS 15-18. Verified on an iPhone (iOS 26.6.1) with Dictation off: the speech-profile redo enters local-STT mode and transcribes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): speech profile talks through three topics and completes on a word target Speech-profile recording (onboarding step and Settings redo) no longer walks one question at a time with a percentage bar. Instead: - A compact white-outlined card headed "Answer with your voice:" lists three topics (where you live, what you do for work, your long-term goal), and a thin bar under it fills as the user speaks. Reaching SpeechProfileProvider.targetWordCount (60 spoken words) finalizes the recording; the backend's onboarding_complete event no longer does, so "bar full" and "done" are the same moment. Omi's own question segments are excluded from the count. - The live transcript is bottom-anchored in a box exactly three lines tall above the card, so whole lines scroll off the top and nothing overlaps. - The Play button on the Settings page plays the saved profile audio in place (just_audio) and turns into Stop, instead of opening the samples page. Redo stops playback first. - Backend ONBOARDING_QUESTIONS is the same three topics, and OnboardingHandler keeps the transcript across questions so one stretch of speech can satisfy several of them. - Removed the unused percentage progress-bar widget and the "Skip this question" button; onboarding keeps "Skip for now". Tests: speech_profile_provider_test (word target fills, finalizes once, ignores Omi segments and the backend completion event); backend test_onboarding_talk_about_flow (one transcript answers every topic; the transcript is kept when it stops answering). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(dev-harness): advertise local-storage links on OMI_DEV_HOST A phone built against OMI_DEV_HOST could reach the backend but not the files it links to: OMI_LOCAL_STORAGE_BASE_URL was always http://127.0.0.1:<port>/_local/storage, so playing the saved speech profile from a device failed. The harness now derives a dev_advertise_host from OMI_DEV_HOST (loopback stays the default) and uses it only for that base URL; every other service address still binds and talks over loopback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the last three whole transcript lines instead of a clipped scroll The speech-profile transcript was a bottom-scrolled ListView clipped to a three-line box, so a sliver of the line above always showed at the top edge and read as cut-off text. FadeInWordsText now takes visibleLines: it replays the Wrap line breaking with measured word widths and builds only the words on the last N lines, so earlier lines drop off whole, nothing is clipped or scrolled, and words keep their reveal state while on screen. Both screens use visibleLines: 3 inside a fixed three-line, bottom-anchored area, moved a little further above the topics card. Test: fade_in_words_text_test covers short text showing everything, earlier lines dropping once the text exceeds three lines, and the shown words matching the line-break replay. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): measure transcript lines with the effective text style; 40-word target FadeInWordsText replayed the Wrap line breaking with the caller's raw style, but each word's Text inherits the ambient DefaultTextStyle (font family, weight) under that style, so the replay undercounted lines and the real layout could reach four lines and draw over the topics card. Measure with the same merged style, and clip the fixed three-line area on both screens as a safety net so a stray line can never overlap the card. Also lower SpeechProfileProvider.targetWordCount from 60 to 40 so the recording finishes sooner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the "<Name>'s Speech Profile" title on one line The title wrapped onto two lines for longer names; it now scales down to fit a single line instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): never clip the speech-profile transcript; raise it above the card The three-line transcript area was a fixed-height clipped box, so whenever the rendered lines ran taller than the fontSize*height estimate (text scaling, font metrics) the top line was cut off. FadeInWordsText already guarantees at most three lines, so the area now only has a three-line minimum height (scaled with the text scaler) and grows to its content instead of clipping. Both screens also keep more space between the transcript and the topics card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): finish the speech profile after three sentences; no page-load spinner The recording now completes once the user has spoken three sentences (SpeechProfileProvider.targetSentenceCount, counted on ./!/? boundaries followed by a space or the end of the text, so "3.5" is not one) instead of a word count, and the bar under the topics card fills per sentence. The progress-bar widget is renamed SpeechProgressBar to match. The Settings speech-profile page no longer swaps its Play/Redo or Get Started buttons for a spinner while the page initialises or the STT pre-flight runs; the buttons stay put and startRecording() ignores taps until the check finishes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): play the saved speech profile on the loudspeaker The app's audio session is normally configured for recording, so tapping Play on the Settings speech-profile page routed the WAV to the quiet earpiece. Before playing, configure a playback-category session (default mode, media usage on Android) and play at full volume, so the profile comes out of the main speaker like any other media. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): start every speech-profile recording with an empty transcript Tapping Redo showed the previous recording's words (and counted them toward the sentence target) because nothing cleared the provider's transcript before a new session; only close() did, on leaving the page. initialise() now calls a new resetTranscript() first, which forgets the segments, text, progress, completion and upload flags without touching the audio storage it recreates right after. resetSegments() reuses it. Test: a completed session's transcript is gone after resetTranscript and the fresh session counts sentences from zero and can finalize again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): finish the speech profile after a pause, keeping the last sentence on screen Reaching the third sentence finalized immediately, which stopped the mic mid-utterance (the recognizers punctuate each clip, so a pause can read as a sentence end) and swapped the transcript for a spinner at once. Now: - After the target is reached the provider waits completionGrace (2 s) without new speech before finalizing, restarting the wait on every new segment, and finalizes at completionCap (8 s past the target) at the latest. Once fired it does not re-arm; resetTranscript() clears it. - Both screens keep the last three transcript lines visible through the upload and the All done state, so the final sentence lingers instead of vanishing. Tests: grace/cap timing under fakeAsync, no double finalize, and the reset test now elapses the grace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(backend): give every speech-profile recording its own conversation Tapping Redo within two minutes of the previous attempt showed last time's words as soon as the user spoke again. The new listen socket attached to the still-open in-progress conversation from the previous attempt (same source, inside conversation_creation_timeout), so combine_segments() merged the first new segment into that conversation's last segment and the merged segment, old text included, was what the client received. LiveConversationController.prepare() now always creates a fresh in-progress conversation for onboarding_mode sessions (the onboarding step and the Settings redo both set it) instead of consulting the in-progress pointer. Ordinary listen sessions are unchanged. Test: test_listen_speech_profile_fresh_conversation.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): cross-fade the speech-profile recording UI into a plain All done button Keeping the transcript on screen through the upload made it pop back in on its own above the spinner and the All done button. The Settings page now cross-fades (450 ms) from the recording UI (transcript, topics card, bar) to nothing while uploading and then to the All done button, which is the same black capsule with a plain white border as the other buttons instead of the gradient box. The onboarding step likewise no longer shows the transcript in its upload and All done states. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * chore(app): remove the speech-samples page and dead progress-state code The Play button now plays the saved profile in place, so the samples page and its provider became unreachable (CI dead-code ratchet). Also drop the scroll controllers and SCROLL_DOWN signal the old clipped transcript used, and the word-count progress-message state (SpeechProfileProgressState, percentageCompleted, questionProgress) nothing reads any more. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the finished speech-profile recording on screen before All done After the third sentence the final words disappeared as soon as the upload began. Both screens now keep the finished recording (last words, topics card, full bar) on screen through the upload and for a further 1.5 s (allDoneHold) after the profile is saved, then cross-fade into the All done button. Onboarding's upload spinner row and its now-unused loading-text helper are gone; Skip for now hides once recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): fade the finished speech-profile recording out as one block Parts of the finished recording could change on their own before the cross-fade (the transcript and the mic disclaimer are built from live provider state that finalize() and its callbacks touch), so they did not disappear together. Both screens now snapshot the recording view (last words, no-device flag) the moment recording ends and build from that until a new recording starts, and the onboarding step's All done switch is now the same AnimatedSwitcher cross-fade as the Settings page, so the words, the topics card, the bar and the disclaimer fade out at the same time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): keep the speech-profile bar full until the finished recording fades finalize() clears the provider's text once the profile is saved, and the bar derived its value from that text, so it dropped back to zero before the cross-fade. The frozen recording view now pins the bar at full from the moment recording ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * feat(app): drop the speech-profile and memory-graph steps from first-run onboarding Onboarding now goes from Permissions straight to the completion screen. The speech profile is recorded from Settings instead, and the memory-graph preview (with its background graph prebuild) is gone. The two step widgets are deleted; their page indices stay as placeholders like the other retired steps so the existing page constants keep working. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): ease the mic glow shut as the finished speech profile fades The white glow behind the device graphic vanished the instant the upload began. It now stays through the upload and hold and eases down to nothing over the same 450 ms in which the recording view fades into All done. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * fix(app): hold the mic glow at its last size until it eases out The glow followed the live mic level, which drops to zero the instant the microphone stops after the recording ends, so it snapped down to its resting size before the ease-out. The frozen recording view now also captures the last mic level, so the glow holds still and then eases shut with the fade. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017iFNr6rZDcD1FkCS3dJ3Cm * Revert "feat(app): drop the speech-profile and memory-graph steps from first-run onboarding" This reverts commit 4ebcb3d2c773747051dfeb2d519fe88aeb1faea0. * feat(backend): tune speaker verification from measured enrollments and retire the SpeechBrain matcher Speaker identification rejected most of the owner's own audio. The verification threshold (0.45 cosine distance) was copied from a clean-studio VoxCeleb figure; an offline bench over real enrollments in the speech-profiles bucket (229 users with a current profile plus an older one, 16 with extra recordings, 60 taught persons, 400 impostors; wespeaker-voxceleb-resnet34-LM, the diarizer's /v2/embedding model) puts same-user cross-session distance at a median of 0.40-0.53 and other users at 0.93. At 0.45 the owner was rejected 37-71% of the time at a 0.0% false-accept rate; the equal-error threshold is ~0.78. Same-session audio matched at either value, which is why the old constant looked fine in demos. - New utils/stt/speaker_match.py owns the policy (numpy only, shared by the live socket and the sync pipeline): threshold 0.65, plus a 0.10 margin over the runner-up so the owner is not guessed as a taught household member. - Live sessions pool up to three recent clips per diarized speaker and decide on the centroid once 5 s of clip audio has accumulated, instead of letting the first 2 s clip that lands under the threshold stick for the session. - Both surfaces log one structured speaker_id_decision line (best, runner-up, evidence, accepted) so the prod distribution can be checked against the bench from a day of logs. - The bench scripts live in backend/scripts/speaker_id_bench for reruns; user audio never leaves the machine running them. - Retire the dead SpeechBrain speaker-identification path: modal/speech_profile_modal, utils/stt/speech_profile (zero production callers), the /v1/speaker-identification route, HOSTED_SPEECH_PROFILE_API_URL in every chart/env, the speechbrain dependency, the shared-package COPY lines in the modal image, the dev-harness and e2e fakes, and the unused ListenLimits.speaker_id_target_audio field. Drop the now-unused is_same_speaker/find_best_match/bytes helpers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(backend): keep speaker_match real in sync test isolation, allowlist its import cost utils/sync/pipeline.py now imports utils.stt.speaker_match, but the hand-maintained heavy_deps mock list in test_sync_cloud_tasks.py and test_sync_v2.py didn't know about it, so `from utils.stt.speaker_match import select_speaker_match` raised ModuleNotFoundError: 'utils.stt' is not a package once utils.stt was replaced with a MagicMock. Real-import speaker_match (pure, dependency-free, like utils.stt.outcomes) instead of stubbing it, since a MagicMock decision object would also break the %.3f log formatting on decision.best_distance/runner_up_distance. Also allowlist test_speaker_match.py::test_short_clips_are_pooled_before_a_live_decision in the fast-unit duration guard: it's the first test in the file to import routers.listen.speakers, so it amortizes that module's FastAPI router-graph import cost, same structural pattern already documented for other files in the allowlist. * fix(speaker-id): preserve distinct evidence and household ambiguity Serialize live matches per speaker, subtract previously embedded audio, invalidate late session results, and keep all enrolled candidates in sync margin comparisons before enforcing unique assignment. Validation: 396 selected backend tests passed; Python typecheck has zero errors. Five live regression cases and two sync cases failed before the fixes. Changed sync expectations follow PR #12935's measured household-confusion margin. Failure-Class: new * fix(speaker-id): include owners in household benchmark cohorts Include available owner profiles even outside legacy, additional, and impostor cohorts. Distinguish offline benchmark evidence from deployed accuracy. Validation: synthetic manifest regression passes for owners outside other cohorts and people without an owner profile. No private audio or threshold retuning. Failure-Class: new * fix(speech-profile): bound native recognition and discard stale work Use one native completion owner so availability deadlines do not wait for shared model downloads and recognition cleanup precedes timeout completion. Serialize legacy recognition callbacks on the main queue. Propagate native failures to retain audio for retry; remove the polling Future timeout that allowed overlapping work. Scope fallback availability and polling results to their recording session. Validation: full Flutter suite 1836 passed, 5 skipped; analyzer ratchet passed. Native deadline behavioral tests pass and are registered in the existing manifest. Native speech code typechecks for iOS 15 deployment with Flutter boundary stubs; no full iPhone build or live enrollment claim. Preflight passed 53 selected checks. Failure-Class: new * fix(l10n): translate speech-profile flow in every supported locale Translate the eight speech-profile keys across all 48 non-English ARBs and fill two inherited missing keys exposed by generation. Use device-neutral speech recognition guidance and regenerate localization output from source catalogs. Validation: flutter gen-l10n reports zero untranslated messages; owner-name placeholders and complete catalog coverage verified. Full Flutter suite passed. Failure-Class: new * fix(speaker-id): require persisted speech profile before the redo admission bypass cubic P1: the client-supplied speech_profile_redo flag alone proved nothing; any authenticated client could send it to skip the completed-account onboarding-provenance admission gate. The runtime now confirms the redo from durable state (an actually stored speech_profile.wav) before taking the bypass, and an unprovable claim falls through to the provenance admission, failing closed when the check errors. Adds a regression test asserting a redo claim without a persisted profile is judged by the gate. * fix(listen): gate the onboarding fresh-conversation path on server admission cubic P2: onboarding=enabled is a client hint, yet prepare() took the fresh-conversation shortcut on the raw flag even when _bootstrap refused to admit the session — a client could dodge the existing-conversation lookup with a query parameter. The path now requires the runtime's onboarding_admitted (also true for the authorized Settings redo); an unadmitted claim keeps an ordinary session's behavior. Adds a regression test for the unadmitted path. * fix(onboarding): queue segments that arrive during AI answer checks cubic P2: is_checking_answer stayed set across up to three awaited LLM calls in _check_answer, and on_segments_received dropped everything spoken in that window, so answers covering later topics could be lost. Segments received while a check is in flight are now queued and replayed when it finishes, re-entering the normal accumulate-and-timer flow. Adds a regression test. * fix(speaker-id-bench): report the production threshold and true impostor rates cubic P2 x2: score.py evaluated the retired 0.45 operating point while the README and shipped policy (SPEAKER_MATCH_THRESHOLD) sit at 0.65, making its false-reject/false-accept and live-decision numbers misleading; and cohort-C impostor distances included the current user's own owner profile when that user was also sampled as an impostor, folding owner-vs-own-person confusion into the random-impostor sweep. score.py now pins T to the production 0.65 and formats every label from it; sweep.py excludes each cohort-C user's own profile from their impostor pool (the confusion keeps its dedicated diagnostic). Owner profiles for cohort C were already added to the cohort inputs by 8126713612. * fix(speech-profile): close startup, playback, and socket adoption races cubic review follow-ups still present after 1d7a2fd917: - page.dart: _isCheckingAvailability is now held until the entire startup path exits (dialogs, codec lookup, stopDeviceRecording, initialise), not just the availability round-trip, so a second tap cannot race socket and microphone init; context/mounted are rechecked after the language dialog and before initialise. - page.dart: profile playback deactivates the activated audio session on every teardown path (stop, natural completion, failure after activation, disposal) instead of leaving media routing active. - speech_profile_provider: a socket created while the session was closed or reset is discarded instead of adopted, which previously leaked a live backend session stop() never saw. - transcription_service: the speech-profile on-device fallback forwards raw audio per config.sendRawAudioToOmi, matching the conversation composite, instead of hardcoding every frame onto the Omi socket; suggested transcripts still flow and keep the backend session clock alive. Pinned by a factory test. * docs(app): keep AGENTS.md within its lean-budget ratchet after the main merge The merge combined this PR's on-device speech pointer with main's profile-build-mode and batch-contract lines, pushing app/AGENTS.md past its agents-md-lean budget (11747 > 11500 bytes). Tightens wording without dropping any fact: the batch-writer guarantee detail lives in the manifest reason and the ruby test itself; the other compressions are same-fact rewording. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Nathan Cheng <nathanjcx@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Nathan <nathan@Nathans-MacBook-Air.local> | 1 天前 | |
feat: add iOS lock screen widget Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 6 个月前 | |
fix(app): isolate Ray-Ban DAT SwiftProtobuf graph Add a dedicated raybanDat target that links Meta DAT 0.8.0 while excluding mcumgr_flutter and its CocoaPods SwiftProtobuf copy only for the DAT transaction. Guard Omi pendant DFU in DAT builds and restore the default plugin, pod, lock, and Flutter flavor state exactly after every run. Verified with the full Flutter test suite (918 passed), Ray-Ban backend tests (7 passed), DAT boundary/graph/wrapper contracts (30 runs before final wrapper hardening; final wrapper suite 12 runs/129 assertions), unsigned DAT/default device builds, and a DAT simulator launch reporting availabilityMode=full without duplicate-class or crash signatures. Physical iPhone/glasses transcript and photo acceptance remains pending Apple membership renewal. Failure-Class: none | 1 个月前 | |
chore(app): keep google_sign_in on 6.x, pin flutter_foreground_task to 10.0.0 Both changes come from building the prod flavor on a physical iPhone, which is the only place they surface: CI has no iOS job, so Dart Analyze and Android Compile Smoke pass on versions that cannot build or run on iOS. google_sign_in stays on 6.2.2 (google_sign_in_ios 5.7.5, native GoogleSignIn pod 7.0.0) and auth_service.dart returns to the v6 API. The v7 migration analyzed and compiled cleanly, but sign-in failed on device with "Authentication failed"; the cause was not established, so the bump is withdrawn rather than shipped unproven. flutter_foreground_task pins to 10.0.0 instead of 11.0.1. v11.0.0 moved the @objc FlutterForegroundTaskPlugin class behind `#if SWIFT_PACKAGE`, so under CocoaPods the class the generated registrant calls is never compiled: Semantic Issue (Xcode): Unknown receiver 'FlutterForegroundTaskPlugin'; did you mean 'SwiftFlutterForegroundTaskPlugin'? ios/Runner/GeneratedPluginRegistrant.m:296 Removing the bridging-header import and adding the module import, as the plugin's migration doc prescribes, is necessary but not sufficient — 11.0.1 additionally requires project-wide Swift Package Manager adoption, which this project does not use and which would have to hold for Codemagic too. 10.0.0 keeps the Obj-C class, needs only Flutter >=3.38, and still carries v10's iOS UIScene lifecycle support. Podfile.lock is regenerated for the surviving bumps: Intercom 19.4.1 -> 19.7.2, flutter_sound_core 9.25.1 -> 9.30.0, GoogleSignIn back to 7.0.0. Verification: flutter analyze (3.44.5) reports 0 error-severity issues, 164 pre-existing info lints; flutter pub get reproduces the committed lockfile; the prod flavor builds and installs on an iPhone XR (iOS 18.7.9) with this tree. | 17 天前 | |
Omi Apple Watch Integration + Comm arch changes (#3012) * init watch app and audio streaming * watch pairing check + build fixes * apple watch integration and cleanup * missing import * rename watch folder * add app icon * improve connection ui stuff and cleanup * cleanup and prepend 3 dummy bytes * misc * move file from src to gen folder * introduce discoverers and communicators * introduce transporters * use transporter for communication in connection and device service * remove communicators and cleanup * run discoverers parallely | 11 个月前 | |
fix(app): isolate Ray-Ban DAT SwiftProtobuf graph Add a dedicated raybanDat target that links Meta DAT 0.8.0 while excluding mcumgr_flutter and its CocoaPods SwiftProtobuf copy only for the DAT transaction. Guard Omi pendant DFU in DAT builds and restore the default plugin, pod, lock, and Flutter flavor state exactly after every run. Verified with the full Flutter test suite (918 passed), Ray-Ban backend tests (7 passed), DAT boundary/graph/wrapper contracts (30 runs before final wrapper hardening; final wrapper suite 12 runs/129 assertions), unsigned DAT/default device builds, and a DAT simulator launch reporting availabilityMode=full without duplicate-class or crash signatures. Physical iPhone/glasses transcript and photo acceptance remains pending Apple membership renewal. Failure-Class: none | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 4 个月前 | ||
| 1 天前 | ||
| 2 年前 | ||
| 1 天前 | ||
| 1 天前 | ||
| 1 个月前 | ||
| 1 天前 | ||
| 6 个月前 | ||
| 1 个月前 | ||
| 17 天前 | ||
| 11 个月前 | ||
| 1 个月前 |