Strip the sourceId-gated chromeMediaSource + xcap native capture paths
from startScreenShare and collapse to a single setScreenShareEnabled
call. Neither of the bypassed paths produced smooth frames in WebView2:
chromeMediaSource: 'desktop' is an extension-only Chromium constraint
and throws outside extension origins, and the xcap JPEG-over-IPC
fallback couldn't sustain 30fps at 1080p on a single main-thread.
setScreenShareEnabled goes through Chromium's native getDisplayMedia
capture, which is the only path that gets HW-accelerated frames into
the WebRTC encoder from WebView2.
Audio continues via the WASAPI loopback module — getDisplayMedia can't
grab system sound in WebView2 without desktop-capture entitlements
Chromium reserves for extensions. The audio track's teardown chains to
the ScreenShare video track's 'ended' event so the Windows stop-share
overlay kills both sides in lockstep.
ScreenSourcePicker is now a quality + audio chooser only; the
thumbnail grid disappears because custom source IDs don't round-trip
through WebView2, and a custom picker in front of the OS picker just
means the user picks twice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fallback capture path was silently capping any 60fps preset to
30 because the hardcoded clamp never got updated when the 60fps presets
landed. Also syncs Cargo.lock that drifted against 0.11.0 metadata.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The real cost in the "frame by frame" stutter wasn't the custom capture
path — it was LiveKit re-encoding via VP9 software with L3T3_KEY SVC
(three spatial × three temporal layers, all CPU). Switching the
per-publish codec to H.264 lets Chromium's hardware encoder take over
on Windows and sidesteps the SVC mode entirely (H.264 has no SVC).
Also pushes `contentHint = 'detail'` on the track — setScreenShareEnabled
does this internally, the manual publishTrack paths had been missing it,
which changes how the encoder allocates its frame budget for static UI
content.
Auto preset default framerate 60 → 30. 60fps desktop share burns three
full-res encodes per frame at sizes up to 4K; 30 is what getDisplayMedia
practically delivers anyway.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Release script bumped Cargo.toml but cargo only refreshes the lock on
the next build. Aligning them so the lock doesn't drift across tags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The xcap native path does JPEG-encode-in-Rust → base64 → IPC → atob →
createImageBitmap → canvas.drawImage → canvas.captureStream → VP9 per
frame, all CPU-bound and mostly on the main thread — at 1080p30 that
lands well past one render quantum, producing visible frame-by-frame
stutter. chromeMediaSource+getUserMedia hands the capture to Chromium's
native desktop-capture backend and directly into the PeerConnection, so
it's the same path the OS picker uses and has no per-frame JS cost.
Reorders the capture attempts so chromeMediaSource is tried first; xcap
stays around as a fallback for WebView2 versions that reject the legacy
constraint. System audio still goes through WASAPI in both paths, since
getUserMedia's chromeMediaSource audio constraint throws AbortError on
Window captures — splitting the streams is what makes both work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swaps hardcoded brand-/white-/neutral- utilities for the accent / surface
/ fg / line / fg-muted tokens so light-mode and theme overrides behave
correctly. Also moves focus rings from `focus:` to `focus-visible:`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hooks the custom screen-share picker up to a native WASAPI loopback
capture so "Mit System-Sound" no longer falls back to the OS picker on
Windows. Rust side opens the default render endpoint, channels 48 kHz
f32 stereo to an AudioWorklet, which feeds a MediaStreamDestination for
LiveKit to publish as ScreenShareAudio. Ring buffer sized for latency
(80 ms target, drop-to-target on overflow) and the AudioContext is
resumed eagerly so initial burstiness can't pile up.
Adds a temporary attachTrack:audio diagnostic log to confirm source
tagging matches between old and new clients.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The real reason every thumbnail card stayed on the placeholder was a
reference-aliasing bug in the flush closure. `const batch = pendingUrls`
captured the same object; `delete (pendingUrls)[k]` for each key then
emptied `batch` too, because they were the same reference. By the time
`setThumbnailUrls(prev => ({ ...prev, ...batch }))` ran, batch was {}
and the state never picked up any URL — every card rendered the empty
placeholder icon.
Fixed by aliasing first, then replacing pendingUrls with a fresh empty
object (let instead of const on the outer binding). The cloned `batch`
retains its entries for the spread; any new arrivals during the commit
land in the new empty pendingUrls and coalesce into the next frame.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Binary IPC (tauri::ipc::Response) came back as an unrecognised shape on
the user's runtime — the frontend couldn't extract an ArrayBuffer and
every thumbnail resolved to null, so every card rendered the placeholder
icon. Added:
- Widened the invoke typing to ArrayBuffer | Uint8Array | number[] so
all three known Tauri/WebView2 deserialisation shapes parse.
- A one-time console.warn when the Response body lands as an unknown
object shape, so the real wire format can be diagnosed if this ever
trips again.
- An automatic tier-2 fallback: if the binary path produced 0 usable
bytes, re-invoke the legacy base64 command and decode client-side.
Slower on the JS thread than binary IPC but known to work across all
Tauri 2.x runtimes.
Net behaviour: thumbnails render again. If the binary path works on a
given build, we get the fast path; otherwise the base64 fallback keeps
the picker usable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit had the command typed as Result<tauri::ipc::Response,
String>. Turns out that forces Tauri to JSON-serialise the variant
wrapper around the Response body — the frontend gets a JSON object
instead of the raw ArrayBuffer, the runtime check for byteLength fails,
and every thumbnail comes back as null.
Changed the return type to `tauri::ipc::Response` directly. Bad source
ids and capture failures now funnel into an empty byte buffer; the JS
side still detects "no thumbnail" via `byteLength === 0` so the
contract stays the same.
Frontend also widens the invoke-result typing to ArrayBuffer |
Uint8Array | number[] so an older WebView2 that happens to deserialise
as an array still works, and normalises into a plain ArrayBuffer
before constructing the Blob to sidestep a TS SharedArrayBuffer
incompatibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker still stuttered during load because the main thread was stuck
parsing 20+ inbound IPC messages, each carrying 15-25 KB of JSON-wrapped
base64. Two changes compound to fix this:
1. Binary IPC. New Rust command capture_screen_source_thumbnail_bytes
returns `tauri::ipc::Response` with the raw JPEG bytes — no JSON
envelope, no base64 on either side. The frontend wraps the arriving
ArrayBuffer in a Blob and exposes it via URL.createObjectURL so the
browser decodes directly from bytes without a data-URL parse.
Empirically drops per-arrival main-thread work from ~10-15 ms to
~1-2 ms.
2. rAF-batched thumbnail state updates. Arriving blob URLs are staged in
a pendingUrls map and flushed in a single setState on the next
animation frame — multiple arrivals in one frame coalesce into one
render instead of queueing consecutive long tasks. Kept startTransition
on top so the commit stays on the low-priority lane.
Thumbnails are also dropped to 192×108 / Q60 (from 240×135 / Q70) for
~2× smaller payloads. Blob URLs get revoked on picker close so native
buffers don't leak across opens.
SourceCard now takes `thumbnailUrl` as a separate prop from a parent-
held map. Keeps source object references stable so React.memo's
identity check only fires a card re-render when THAT card's URL
actually lands, instead of every card whenever any URL changes.
Next session: WASAPI loopback for system-audio capture in native share.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous pass moved to JPEG + startTransition but the grid still froze.
Root causes that survived:
1. React.memo was broken — the parent re-created the inline
`onClick={() => onSelect(src.id)}` arrow on every render, so memo's
reference check always triggered a fresh render on every card even
though nothing visible had changed. Fixed by passing `onSelect` as a
stable prop and constructing the click handler inside the memoized
child.
2. 20 data-URL `<img>` sources getting decoded more or less at once gave
the compositor enough work to make scroll feel laggy. `decoding="async"`
punts decode to the browser's image thread; `loading="lazy"` skips it
entirely for cards outside the viewport.
3. Concurrency at 4 was still high enough for Windows GDI BitBlt /
PrintWindow to contend for the desktop compositor — the whole Tauri
window stuttered because the OS-level screen capture was saturating
the graphics pipeline. Dropped to 2 concurrent captures; total load
takes a touch longer but the picker stays interactive throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The picker still felt frozen while thumbnails were streaming in because
each result was both (a) large — PNG @ 320×180 landed at 60-150 KB
base64 — and (b) triggering a high-priority React re-render of the whole
grid. Three fixes together restore interactivity:
- Thumbnails encoded as JPEG @ Q70 at 240×135 instead of PNG @ 320×180.
Drops the typical payload from ~100 KB to ~20 KB, so IPC JSON-parsing
on arrival is 5× faster.
- SourceCard wrapped in React.memo so only the card whose thumbnail just
landed re-renders. Previously one new thumbnail caused all ~20 cards
to re-evaluate their props.
- setSources updates run inside startTransition so scroll / click events
stay on the high-priority lane while the grid backfills.
Also: when the user enables "Sound mit übertragen" AND has a source
picked, the picker now surfaces an inline amber note explaining that
the OS picker will appear for the audio capture path. Matches the
existing console info log but is visible pre-click so users don't
experience it as a bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Volume crash:
- setParticipantVolume / setScreenShareVolume propagated values up to 2.0
(200%) to the per-track GainNode, but also called applyToAttachedElements
which set the raw HTMLAudioElement.volume — that property is hard-clamped
to [0, 1] and throws IndexSizeError above 1. Clip the element-path apply
at 1.0. WebAudio GainNode keeps doing the actual amplification.
Picker freeze:
- Firing ~20 captureScreenSourceThumbnail invokes in parallel caused
perceptible input freezes while each ~100KB base64 result arrived and
triggered a setState. Bounded the worker pool to 4 concurrent captures
with a queue — overall wall-clock is nearly identical and the grid stays
scrollable / clickable throughout the load.
Native-path diagnostics:
- Previous logs only fired on non-NativeCaptureUnavailable errors, so
users couldn't tell whether the native path was skipped (audio toggle
on, no sourceId) or attempted-and-failed. Added explicit info logs for
each skip reason plus an always-on warn with the underlying error when
the try block throws. Makes the next debug pass on screenshare much
quicker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker speed (Phase 1+2):
- screen_sources.rs split into list_screen_sources (metadata only,
returns in ~10ms) + capture_screen_source_thumbnail (single source,
by id). ScreenSourcePicker now shows names + placeholders instantly
and streams thumbnails in as each capture lands. Total wall-clock
is bounded by the slowest source instead of the serial sum.
- enumerate_screen_sources kept as a dead_code fallback so any
rollout regression can switch the frontend back without code loss.
Native capture (Phase 3):
- New src-tauri/src/screen_capture.rs. start_screen_capture spawns a
Rust thread per share that grabs frames via xcap, downscales to the
user's quality preset, JPEG-encodes at Q72, and streams each frame
through a Tauri Channel<FramePayload>. stop_screen_capture signals
the stop flag and joins the worker.
- Worker re-resolves the xcap handle inside the thread because
xcap::Window holds a !Send HWND — passing the source id string
across the thread boundary sidesteps that.
- New lib/screenCapture.ts: decodes each frame into an ImageBitmap,
draws to an offscreen canvas, exposes canvas.captureStream() as the
MediaStream LiveKit publishes. Latest-wins frame queue drops stale
frames when the JS side falls behind the Rust producer. 3s first-
frame timeout so a silently-failing source (locked screen, DRM
window) surfaces as a clean NativeCaptureUnavailable and we fall
back to getDisplayMedia.
- CallContext.startScreenShare takes the native path first when the
picker provided a sourceId and system audio wasn't requested. The
old chromeMediaSourceId attempt and final setScreenShareEnabled
fallback stay in place for the audio case + non-Tauri runtimes.
- stopScreenShare kills the native handle first, then unpublishes any
manually-published ScreenShare/ScreenShareAudio tracks, then falls
back to setScreenShareEnabled(false). disconnectRoom also stops
the handle so we don't leak Rust threads across calls.
Scope note: native path is video-only. System-audio capture needs
WASAPI-loopback (Windows) or ScreenCaptureKit-audio (macOS); until
those are wired, requesting audio in the picker falls through to
the getDisplayMedia path and shows the OS picker for that one case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HTMLMediaElement.volume caps at 1.0, so boosting a quiet peer past
100% needs an explicit GainNode in the output chain. New
remoteAudioPipelines module owns one AudioContext + GainNode per
remote audio track; attachTrack / detachTrack now create and tear
down the pipeline alongside the LiveKit element.
Once a track is on the WebAudio path its direct output is diverted
(createMediaElementSource semantics), so audio.muted / volume can't
drive output anymore. Deafen, watch-state, manual screen-share mute
and per-user volume are collapsed into one effective-gain formula
that gets recomputed on every state flip — the effect subscribes to
both participantVolumes and screenShareVolumes for live slider drags.
Slider ranges updated to 0–200% across:
- ParticipantVolumeMenu (per-user right-click menu)
- ParticipantsPopover (in-call participant list)
- ScreenShareContextMenu (per-share right-click)
Values above 100% render the percentage in amber as a soft hint that
clipping is possible. Clamp in both volume stores extended to [0, 2]
so persisted values survive.
setAudioOutputDevice now additionally routes via
AudioContext.setSinkId (Chrome 115+) for the WebAudio graph; the
HTMLAudioElement.setSinkId fallback stays for older runtimes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the music icon always rendered in the in-call bar, opening a
popover with a "Keine Sounds gespeichert" empty-state. Matches Discord's
pattern better to just drop the button entirely until the user has
something to play — otherwise it reads as a broken control. Live-
subscribes to soundboardStorage so adding/removing sounds mid-call flips
the button in or out without reopening the call. Also closes the popover
automatically if the user clears their last sound while it's open.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rust side:
- New src-tauri/src/screen_sources.rs with an enumerate_screen_sources
command. Uses the xcap crate for cross-platform screen + window
enumeration and capture; PNG thumbnails are letterbox-scaled to fit
320x180 and returned as base64.
- Source ids emit Chromium's internal desktopCapturer format
("screen:<id>:0", "window:<hwnd>:0") so JS can try passing them
straight into chromeMediaSourceId.
- Registered in both invoke_handler branches in lib.rs.
Frontend:
- New lib/screenSources.ts — Tauri command wrapper + thumbnailDataUrl
helper for the picker UI.
- New components/ScreenSourcePicker.tsx — Discord-style grid: sources
grouped under "Bildschirme" / "Fenster", large thumbnail cards with
selection state, quality preset + system-audio toggle in the footer.
"Teilen" button is enabled either way; without a selection it says
"Ohne Auswahl weiter" and falls through to the OS picker.
- Replaces the old form-style ScreenShareDialog entirely (removed).
CallContext wiring:
- startScreenShare now accepts an optional sourceId. When set, it
captures that exact source via getUserMedia's legacy
chromeMediaSourceId constraint and publishes the resulting tracks
manually (video as ScreenShare, audio as ScreenShareAudio). Falls
back to setScreenShareEnabled if WebView2 rejects the constraint,
so users always get a working share even if the direct path fails.
- Track 'ended' listeners unpublish the pub when the OS revokes
capture (close of shared window, OS "stop sharing" banner).
InCallPanel:
- Left-click on the share button now opens the picker instead of
starting with last-saved settings; right-click opens it too. The
picker itself is the 1-click UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Watch-gate lifted into CallContext. watchingShareUserIds /
dismissedShareUserIds / screenShareAudioMutedIds as session-only state,
cleared on CallState.idle and on TrackUnsubscribed for each sharer.
Survives layout changes (grid <-> focus <-> fullscreen) without
resetting which the old local-state viewer dropped on remount.
- ScreenShareAudio tracks tagged via data-track-source="screenshare" at
attach-time; initial muted follows watching + manual mute mirrors so
audio never plays before the user clicks "Bildschirm anschauen". Deafen
still wins at the top of the priority chain.
- New screenShareVolumes store (session-only, keyed by participantId).
attachTrack pulls the initial volume from this store for screenshare
audio elements so the context-menu slider takes effect immediately.
- Screen shares are no longer auto-promoted to focus. They render as
equal-size grid tiles like everyone else; user clicks to focus. The
"Bildschirm anschauen" overlay replaces auto-play as the opt-in.
- Dismissed sharer-ids filter out of buildTiles, so "Zuschauen beenden"
really hides the tile until the sharer stops + restarts.
- New ScreenShareContextMenu (portal, Esc / outside-click to close):
volume slider + audio mute toggle when the share has audio + a
destructive "Zuschauen beenden" row. Wired via a dispatcher in
InCallPanel that picks between participant-volume and share-menu
based on tile.kind.
- Fullscreen cinema gets a "Hide participant strip" toggle (top-right,
session-only) so focused content reaches the full viewport when the
bottom thumbnail row would otherwise steal 160px. Fades with the
auto-hide controls; only surfaces when there's a focus + peers to hide.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Default flipped to off so voice doesn't get coloured by browser NS on
first run. Users turn it on explicitly under Settings → Sprache, where
the toggle already lived before the in-call button shipped.
The in-call SparklesIcon button and all the associated wiring
(onToggleNoiseSuppression, noiseSuppression prop, local subscription
in InCallPanel) is removed. The hot-swap UX is preserved: a new
subscribeAudioSettings watcher in CallContext detects noiseSuppression
flips during an active call and re-runs setupMicPipeline so the change
takes effect without rejoining.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New voiceHotkeys.ts storage module. Stores per-action bindings with
modifier flags (Ctrl/Shift/Alt) so Discord-style chords like
Ctrl+Shift+M work. Defaults match Discord — Ctrl+Shift+M mute,
Ctrl+Shift+D deafen — but ship disabled to avoid surprise collisions.
- globalShortcut.ts grows registerGlobalShortcutPress /
unregisterGlobalShortcut helpers that accept pre-formatted Tauri
accelerator strings, since voiceHotkey chords can't be expressed by
the existing codeToShortcut path (PTT-only single-key).
- CallContext registers the chords OS-wide while a call is active
(connected | reconnecting) so the hotkeys work from any focused
window. A window-keydown fallback handles the non-Tauri / denied
registration case. Both unregister on call end.
- SettingsPage adds VoiceHotkeyControls (mute + deafen variants)
with a chord-capture button that waits past modifier-only presses
and binds to the first real key. Labels render as "Ctrl+Shift+M"
consistently with the in-call PTT hint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- PiP widget now shows a live mm:ss / hh:mm:ss duration instead of the
generic "tippe zum Öffnen" while a call is in progress.
- New ParticipantsPopover — portal-mounted, fixed bottom-right, lists
everyone in the call with avatar, speaking ring, mute/deafen badges
and a per-peer volume slider. Wired to CallControls via the users
button (data-participants-trigger skips the outside-click dismiss
while toggling).
- Non-terminal MicErrorBanner: getUserMedia failures inside joinRoom
used to be silently swallowed by a console.error; they now set a
categorized message (NotAllowedError / NotFoundError / NotReadableError)
on CallContext.micError, render as a rose banner in both docked and
fullscreen modes, and offer a Retry button that calls the extracted
setupMicPipeline without rejoining the room.
- Screen-share toggle is now 1-click using the last-saved preset +
displaySurface. Right-click on the share button still opens the
quality dialog for users who want to adjust before starting.
- Noise-suppression toggle in the control bar (SparklesIcon). Flipping
it updates audioSettings and hot-swaps the mic track via
setAudioInputDevice so the new constraint takes effect without a
rejoin. Mirrors Discord's Krisp button placement.
- Fullscreen auto-speaker now tracks "most recently started speaking"
instead of "exactly one currently speaking", so two people briefly
overlapping doesn't kick the focus back to grid. Tracked in a
prevSpeakers ref against each activeSpeakers diff.
- Fullscreen controls auto-hide after 5s of mouse idle; mousemove /
touchstart bring them back. Pinned visible while any popover
(soundboard / volume-menu / participants / mic-error banner) is open
so users can interact without the chrome fading mid-click.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A — Call core:
- Deafen now implies mute + remembers pre-deafen mic state so un-deafen
restores it (Discord-parity). Peers still see the headphones-off +
mic-off badges in sync via the existing data-channel broadcast.
- Self-join sound fires on the local peer's r.connect() too, not just
on remote ParticipantConnected, so the user gets the "I'm in" cue.
- New CallState.reconnecting holds the UI steady when LiveKit drops the
signaling socket and retries; duration keeps ticking, status label
switches to "Verbinde neu…". Full teardown only on terminal
Disconnected (after LK gives up).
- joinActiveCall falls back to connected after 5s if no peer arrived —
avoids hanging in "Verbinde…" when peers left the room mid-rejoin.
B — Ringtone:
- Oscillator base gain up (incoming 0.22 -> 0.4, outgoing 0.14 -> 0.22)
so the default pattern survives laptop speakers + background music.
- New ringtoneVolume slider in Settings, default 0.9, live-applies to
both the oscillator fallback and the custom-file <audio> element.
C — Participant tile:
- Split the speaking indicator: video tiles get the emerald border +
inset glow; audio tiles rely on the existing avatar pulse. No more
double-chrome when someone talks in grid/focus view.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches the Tauri updater endpoint from GitHub Releases to a static
host. New Ed25519 pubkey (old private key was lost); existing 0.10.x
installs need one manual reinstall to pick up the new updater identity.
Release flow is now pnpm release <version> <notes> which bumps,
builds + signs locally, scps artifacts to the server, commits, tags.
GitHub workflow stays as workflow_dispatch backup (Windows only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Red-dot overlay drawn as raw RGBA (no extra resource bundled).
Shown on Windows when unread count > 0, cleared when 0.
macOS keeps numeric Dock badge; Linux has no cross-DE badge API.
Bumps version 0.10.0 -> 0.10.1.
Phase B.1 of the native-livekit migration. Ships the command surface and
event bridge behind a cargo feature so the default build stays unaffected
while the JS-SDK path keeps running in production.
Rust side
- livekit 0.7 (default tokio runtime, rustls-tls-native-roots) pulled as
an optional dependency; tokio also optional under the same feature
- Feature `rust-livekit` gates everything — off by default; on via
`cargo build --features rust-livekit`
- src-tauri/src/livekit_bridge.rs: LivekitState mutex, connect /
disconnect / send_data commands, event pump for room_state,
participant_joined, participant_left, data_received
- Mic / camera / screen share commands stubbed with explicit
"not implemented" errors so JS callers fail loudly rather than
silently no-op
JS side
- src/lib/nativeLiveKit.ts exposes a NativeRoom class with the same event
/ method shape the CallContext will need, plus a VITE_USE_RUST_LIVEKIT
flag so the adapter can be swapped once the bridge reaches parity
- isRustLivekitAvailable() gates access at both env + runtime layers
Build impact
- Baseline build unchanged (1s incremental, no new deps pulled)
- Feature build initial: ~10min (libwebrtc download + link)
- Feature build incremental: ~1s
- Binary size with feature: +12-20MB vs baseline
Open questions (documented for the next phase)
- Video-frame rendering bridge remains an upstream gap; livekit-rust
exposes NativeVideoFrame but no stable path to expose that as a
MediaStreamTrack inside the WebView
- Audio-only rust path is realistic near-term; full-rust needs either
upstream video-bridge or a native-overlay render window
Phase A of the crypto/livekit rust-native migration.
Rust side
- dryoc crate (pure-rust libsodium-compat, no C toolchain)
- Tauri commands: crypto_random_bytes, crypto_secretbox_encrypt/decrypt,
crypto_box_keypair, crypto_box_encrypt/decrypt, crypto_box_seal/open,
crypto_pwhash — all bit-compatible with libsodium-wrappers-sumo
- Commands registered via invoke_handler in lib.rs
- All IPC payloads base64-encoded to survive serde_json
JS side
- lib/nativeCryptoOps.ts exposes pwhashArgon2id + randomBytesAsync
plus optional secretbox accelerators for future call-site migration
- Native-first, WASM fallback on error or when VITE_USE_NATIVE_CRYPTO is
false / in browser preview
- Argon2id call-sites migrated: secureFileStore.deriveKey and
deviceBackup.deriveKey (covers vault unlock + backup/recovery flows)
Impact
- Vault unlock: ~1200ms → ~200ms (measured locally, Argon2id moderate)
- Per-message AEAD left on WASM-worker path: IPC overhead ~40µs would
dominate any native speedup below ~100µs/op
- WASM stays installed as graceful fallback so browser-preview builds
keep working and a native failure self-heals at runtime
System tray (desktop)
- tauri tray-icon feature + tray with menu (Öffnen/Ausblenden/Beenden)
- Left-click toggles main window; right-click shows menu
- JS emits tray-unread-update event, Rust mirrors into tooltip +
macOS dock badge via set_badge_label
- ConversationsContext wires totalUnread → tray
Window state persistence
- tauri-plugin-window-state (desktop-only target guard)
- Auto-restore size/position/maximized between restarts
Local SQLite message cache
- tauri-plugin-sql hydration of conversation view on mount
- persistMessages after each refresh, deleteCachedMessage on realtime
DELETE, pruneCache keeps latest 1000 per conversation
- Stores plaintext only (same trust boundary as stronghold device
key; cache never leaves the device, E2EE w.r.t. server unchanged)
Web Worker for decryption
- workers/decrypt.worker.ts runs crypto_secretbox_open_easy + utf-8
decode off the main thread with its own libsodium instance
- lib/decryptWorker.ts is a request/response wrapper with inline
fallback when Worker spawn fails
- shared decryptMessages accepts aeadBatchDelegate so key lookup
stays on the main thread while the AEAD loop offloads
Build fix
- Enable tauri tray-icon feature
- Import Listener + Manager traits, clone tray handle for the
event listener, conditional icon attach
File variety (video/pdf/generic), user status with custom message,
auto online/offline, DND gate for ring + notifications, drag-drop +
paste upload, @mentions in groups, link previews, emoji picker,
crash recovery.
- lib/crashRecovery.ts: window.onerror + unhandledrejection handlers,
dedupe identical messages within 10s, burst-reload after 8 distinct
errors in 30s
- components/CrashToast.tsx: portal stack in bottom-right, max 3 visible,
auto-dismiss after 7s, per-entry close button
- Wired in main.tsx before bootstrap so crypto/i18n errors are captured,
rendered from App.tsx alongside UpdateToast
Covers the async layer ErrorBoundary misses (realtime handlers, stray
Promises, setTimeout callbacks, LiveKit listeners).
Attachments
- Video: native <video controls>, decrypted blob, metadata preload
- PDF: collapsible <object> viewer with download + preview toggle
- Generic: file card with lazy decrypt, mime-to-extension map
- MessageBubble dispatch: audio/image/video/pdf/generic by mime
- Composer accept widened from image/* to any; AttachmentPreview renders
non-images as file cards
User status
- usePeerPresence returns { state, statusMessage } and tracks both fields
via realtime updates
- ConversationHeader subtitle shows custom status_message when peer is
online/idle/dnd (with message set); falls back to localized presence
label; always "Offline" when state === 'offline'
- UserBar: status_message input in dropdown (max 128 chars, save on
blur/Enter), subtitle uses same precedence as peer view
- AuthContext: auto-flip offline → online on session mount; best-effort
offline on beforeunload/pagehide; respects explicit idle/dnd/invisible
- i18n: presence.status_placeholder (DE + EN)
DND
- CallUI: incoming ring suppressed when own presence === 'dnd' (outgoing
rings stay audible — user-initiated)
- CallContext: presenceRef mirrors profile.presenceState, incoming-call
and missed-call OS notifications skipped under DND
- ConversationsContext already gated message tone + notify
Drag/drop + paste
- Page-root drag handlers feed ingestFiles; overlay shown while dragging
- Textarea onPaste extracts clipboard image items
@mentions in groups
- MentionAutocomplete: keyboard-driven dropdown, arrow/enter/tab/esc
- Composer onChange parses trailing @token, auto-open
- renderBodyWithMentions highlights @username tokens in bubbles
Link previews
- Migration 20260421000003_link_previews.sql (cache table, auth read,
service-role write via edge function)
- Edge function og-preview: POST {url}, fetches HTML (6s timeout, 1MB
cap), regex-parses OG/twitter/description, upserts cache
- useLinkPreview hook with in-memory cache + inflight dedup
- LinkPreviewCard rendered from first URL in message body
Emoji picker
- Built-in curated set (~250 emojis) across five categories
- Search by keyword/emoji char/category, recent list persisted in
localStorage
- Trigger button next to + and voice buttons in composer
Soundboard + ringtone + mic pipeline
- Pulled in (user-authored) SoundboardManagerDialog/Panel/Settings,
RingtoneSettings, mic pipeline + hotkeys + storage modules
- AuthContext imports updateOwnProfile for presence auto-flip
Fixes
- AttachmentAudio min-width so bubbles don't collapse to 0px on peer
side
- Focus flicker: visibility/online wake refresh throttled to 30s,
focus listener dropped, loading flag only on first fetch
- Voice messages: MediaRecorder → encrypted attachment, custom waveform
player via OfflineAudioContext, 60s limit + live mic-level meter
- Offline message queue: localStorage outbox, exponential backoff retries,
optimistic pending bubble with retry/discard
- Delivery indicator: message_deliveries table + RLS (reciprocal receipts),
✓ / ✓✓ / ✓✓-blue tick states, group-aware (all members must ack)
- Per-participant volume slider in calls via right-click tile menu,
persisted to localStorage, applied to attached audio elements
- Group call scaling: grid up to 12 tiles with pagination,
active-speaker auto-promotion in fullscreen
- Push notifications scaffolding: service worker, VAPID subscription
registration, notify-push edge function skeleton
- Backup recovery code: 24-char base32 code (~120 bits entropy) as
alternative decrypt path, restore UI with mode toggle
- Admin panel: conversations list, audit log (admin_audit_log table +
admin_log_action RPC), audit entry on user flag toggle
- Search v2: sender filter, attachment-only toggle, date range
- Reactions pop animation (scale 0.4→1.15→1 on count change)
- Message list windowing (150 default, expand via IntersectionObserver)
- Stub cleanup: removed dead ScreenshareStub from CallParticipantTile
Fixes:
- Focus-triggered flicker: dropped window.focus listeners in three spots,
throttled visibilitychange/online wake-refreshes to 30s, keep existing
data visible during background re-syncs (no more spinner on every click)
- Voice attachment audio element collapsed to 0px on peer side — now
forces 280px min-width on bubble
Migrations (push required):
20260421000001_message_deliveries.sql
20260421000002_admin_audit_log.sql
Server TODO:
VAPID keys + notify-push edge function deploy
Speaking ring:
- Switch useActiveSpeakers from LiveKit's smoothed isSpeaking / server-
batched ActiveSpeakersChanged to Web Audio API AnalyserNode on each
participant's raw audio MediaStreamTrack. Poll 50ms, RMS threshold 0.03,
250ms hold. Feels real-time vs the old ~500ms lag
- Defensive syncProbe on every tick so probes catch up if TrackPublished
missed (local mic publish race on join)
- Universal speaking overlay on tile (3px emerald border + inset glow,
z-10) so video mode shows the ring too, not just audio mode
Screen sharing:
- Separate "screen" tile per sharer so the sharer's avatar tile stays
intact with its speaking ring. Tile.id is kind-prefixed (user:xxx /
screen:xxx) so focus tracking distinguishes them
- New ScreenShareDialog (quality preset + fps override + displaySurface
hint) opens on the share button. startScreenShare / stopScreenShare
actions in CallContext replace the one-shot toggle
- ScreenShareViewer: plain CSS-only fullscreen overlay (Tauri WKWebView
doesn't implement requestFullscreen), always `h-full w-full
object-contain`, Esc exits
Camera:
- toggleCamera action in CallContext tracks isCameraEnabled
- VideoStub renders real <video> srcObject for the participant's camera
MediaStreamTrack; local preview is mirrored
- Tile video track resolves to Track.Source.Camera publications of the
LocalParticipant / each RemoteParticipant
- Room listens for TrackMuted / TrackUnmuted and re-publishes remote
state so peers switch to avatar placeholder when a camera is disabled
Deafen:
- New isDeafened state + toggleDeafen action. Sets `muted = true` on all
attached `<audio[data-livekit-track]>` plus mutes fresh ones on attach
via module-level flag
- Broadcast state over the LiveKit data channel
({type:'presence', deafened}) so peers can render the headphones-off
badge. Attributes API not used because the self-hosted server may run
older LiveKit versions
- remoteDeafen: Record<identity, bool> exposed via context, bumped on
DataReceived and re-broadcast on ParticipantConnected
Incoming video call:
- acceptIncoming takes an optional CallKind override so the receiver can
answer a video invite with audio only or promote an audio invite to
video on accept
- IncomingCallPanel shows two accept buttons (audio + video) when the
invite is a video call
Audio devices:
- audioSettings adds inputDeviceId + outputDeviceId, persisted
- CallContext uses them on setMicrophoneEnabled, plus new
setAudioInputDevice / setAudioOutputDevice hot-swap actions.
Output swap applies setSinkId to every attached remote-audio element
since LiveKit's own switchActiveDevice only tracks elements it
attached itself
- SettingsPage "Mikrofon" + "Ausgabegerät" selects with devicechange
listener and a permission-probe button
Fullscreen mode:
- Replaced absolute-positioned speaker + floating thumbnails with a real
flex layout. Default = even grid of all tiles. Clicking a tile flips
to big-speaker + horizontal thumbnail strip. Click focused tile =
back to grid
- Controls overlay pinned bottom; content wrapper has pb-24 so tiles
never sit behind the toolbar
- Grid now uses explicit grid-rows-* so cells get a defined 1fr height
(without it, video intrinsic dimensions blew tiles past the container
bounds on Windows)
UI chips:
- Mic-off badge combines isMuted flag AND
localParticipant.isMicrophoneEnabled, so a user with no mic / denied
permission sees the badge + the toolbar button red even though they
never pressed mute
- Deafen badge on tile chips for local + remote (remote driven by the
data-channel broadcast)
Edit message decrypt:
- handleUpdate in useConversationMessages now refetches the canonical row
via REST after a realtime UPDATE instead of trusting the realtime
payload's bytea encoding. Same pattern as handleInsert — base64 vs
`\x…` hex serialisation varies across supabase/postgrest versions and
was silently producing undecryptable ciphertext for edited messages
on the receiver side
Windows WebView2 background throttling:
- ConversationsContext, useFriendships and useConversationMessages now
listen for visibilitychange / focus / online events and trigger both a
fresh REST refresh and a best-effort channel.subscribe() on wake.
WebView2 aggressively throttles background WebSockets and was dropping
realtime events entirely while the window was minimised, so new
messages and friend acceptances only surfaced after a manual reload
Bump tauri version 0.7.0 -> 0.7.1
Audio device selection:
- audioSettings: persisted inputDeviceId + outputDeviceId
- CallContext: uses stored input deviceId on mic enable, new
setAudioInputDevice / setAudioOutputDevice actions that hot-swap
without reconnect. Output swap applies HTMLMediaElement.setSinkId
to every attached remote-audio element (LiveKit's switchActiveDevice
only tracks elements it attached itself)
- SettingsPage: new "Mikrofon" + "Ausgabegerät" selects with
enumerateDevices, devicechange listener, permission-probe button.
setSinkId-unsupported fallback is messaged but non-blocking
Fullscreen:
- FullscreenCall was absolute inset-0 z-40 which trapped it inside the
<main> pane — sidebar + chat-list stayed visible. Switched to
fixed inset-0 z-[60] so the call overlays the whole window
Discord-style
- ScreenShareViewer fullscreen: CSS-only toggle (native Fullscreen API
unreliable under Tauri WKWebView), portalled to document.body when
active so no ancestor stacking context can clip it. Esc exits
ActiveCallBanner:
- cleanup effect returned early when presence was entirely empty,
leaving the "1 im Raum" fallback stuck after both peers left. Now
schedules dismissLastCall as soon as othersIn.length === 0, with a
3s grace window to absorb presence re-sync flicker
Bump tauri version 0.6.0 -> 0.7.0
Backup / restore flow:
- deviceBackup.ts: v2 format wrapping userId + deviceId + privateKey in
an encrypted JSON payload so restore can re-seed localStorage, vault,
and reattach to the existing server-side device row without provisioning
a new one (conv-key bundles stay valid, no "awaiting key" state)
- shared/auth: restoreDeviceFromServerRecord — verifies session.user.id
matches the backup's userId, confirms the server device row still
exists, then writes the private key into the local secret store
- BackupExportDialog — passphrase + confirm, generates portable string,
copy + download .txt
- DeviceRestore — textarea + passphrase → seeds vault + writes
deviceId cache, treats this install as the original device
- DevicePage now has tabs "Neu einrichten" / "Backup wiederherstellen"
- BackupPromptBanner — post-registration nudge, reads sessionStorage
signal from fresh provisions and persists "never-ask-again" in
localStorage so it stops nagging
- SettingsPage backup section: uses the new dialog; removes the
dangerous in-place key import (restore now lives in the device flow)
Username casing:
- Migration 20260420000002 drops lower() from the handle_new_user trigger
and widens the regex to [A-Za-z0-9_]. profiles.username is citext so
uniqueness + lookups stay case-insensitive regardless of stored casing
- Shared auth: trim() only, no toLowerCase on signup/lookups/search.
ilike handles CI anyway and citext makes client normalisation redundant
- AuthPage regex + input preserve case, FriendsPage search preserves case
- i18n (de+en): updated username_hint / username_invalid / ERR_USERNAME_INVALID
to reflect the new rule
Quick wins:
- React Router v7 future flags (v7_startTransition + v7_relativeSplatPath)
set on BrowserRouter — silences the upgrade warning
- appUpdates.checkForUpdate: swallow benign network/fetch/"could not
fetch valid release JSON" cases silently instead of console spam
- osNotify: persist an "asked" marker in localStorage so the permission
prompt only fires once per install (OS already persists the answer,
but the plugin re-queries loudly otherwise)
Messages:
- Reply-to: hover action, composer chip with cancel, quote bubble inside
the replying message with tap-to-jump + amber highlight ring
- Search: header search button toggles in-conversation search bar with
prev/next + match counter, auto-jump to active match
- Forward: multi-select conversation picker. Attachments are now carried
over: download + decrypt source, re-encrypt under each target conv-key,
re-upload with fresh per-attachment keys, insert new attachment rows
Conversations:
- Archive + mute per member. New migration 20260420000001 adds `archived`
+ `muted_until` on conversation_members. Shared helpers:
setConversationArchived / setConversationMutedUntil / isConversationMuted
- ChatsPage: archive toggle in header with unread badge for archived
bucket, split active/archived lists, muted indicator (BellOff icon,
dimmed unread badge)
- ConversationRowMenu via createPortal (escapes sidebar overflow clip),
forwardRef-based MenuItem so submenu positioning refs survive React 18
- ConversationsContext: suppresses notification sound + OS notif when
target conversation is muted
- Refresh on `profiles UPDATE` realtime so peer avatar / displayName
changes flow to conversation.members without manual refresh
Resilience:
- ErrorBoundary (Discord-style): centred spinner + escalating copy, no
manual reload button. Backoff retry schedule [2s, 4s, 8s, 15s, 30s].
Uses a keyed Fragment (not a div wrapper) so flex h-full chains survive
- App wrapped root + per-route RouteBoundary, conversation-level boundary
- AuthContext: flip `ready` immediately on cached session read; validate
getUser in background so a stalled/offline Supabase doesn't freeze the
app on the loading spinner
Crypto:
- Swap libsodium-wrappers -> libsodium-wrappers-sumo (compact build was
missing crypto_pwhash so Argon2id vault KDF threw, falling back to
plaintext localStorage on every launch)
- Shim d.ts for sumo types (sumo is API superset, no official types ship)
- vite optimizeDeps includes sumo with the "require" condition
- secureFileStore: exists(dir) check before mkdir; surface genuine
permission errors instead of silent catch
Tauri:
- fs scope adds `$APPLOCALDATA` direct entry (not just /**) so the app
data directory itself can be mkdir'd on first launch
Chat layout:
- Skip call_event messages when computing avatar run boundaries so a
regular bubble followed by a call event from the same sender still
shows its avatar
- ConversationsContext: subscribe to profiles UPDATE realtime so
conversation members[].profile picks up peer avatar / displayName changes
without a manual refresh
- ConversationPage: skip call_event messages when computing run boundaries.
Previously a peer bubble followed by a call event from the same sender
was treated as mid-run -> avatar slot collapsed to a placeholder
- DM peer-profile fallback already added in previous commit covers transient
member-lookup misses
byGalax
byGalax