Hooks the custom screen-share picker up to a native WASAPI loopback
capture so "Mit System-Sound" no longer falls back to the OS picker on
Windows. Rust side opens the default render endpoint, channels 48 kHz
f32 stereo to an AudioWorklet, which feeds a MediaStreamDestination for
LiveKit to publish as ScreenShareAudio. Ring buffer sized for latency
(80 ms target, drop-to-target on overflow) and the AudioContext is
resumed eagerly so initial burstiness can't pile up.
Adds a temporary attachTrack:audio diagnostic log to confirm source
tagging matches between old and new clients.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit had the command typed as Result<tauri::ipc::Response,
String>. Turns out that forces Tauri to JSON-serialise the variant
wrapper around the Response body — the frontend gets a JSON object
instead of the raw ArrayBuffer, the runtime check for byteLength fails,
and every thumbnail comes back as null.
Changed the return type to `tauri::ipc::Response` directly. Bad source
ids and capture failures now funnel into an empty byte buffer; the JS
side still detects "no thumbnail" via `byteLength === 0` so the
contract stays the same.
Frontend also widens the invoke-result typing to ArrayBuffer |
Uint8Array | number[] so an older WebView2 that happens to deserialise
as an array still works, and normalises into a plain ArrayBuffer
before constructing the Blob to sidestep a TS SharedArrayBuffer
incompatibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker still stuttered during load because the main thread was stuck
parsing 20+ inbound IPC messages, each carrying 15-25 KB of JSON-wrapped
base64. Two changes compound to fix this:
1. Binary IPC. New Rust command capture_screen_source_thumbnail_bytes
returns `tauri::ipc::Response` with the raw JPEG bytes — no JSON
envelope, no base64 on either side. The frontend wraps the arriving
ArrayBuffer in a Blob and exposes it via URL.createObjectURL so the
browser decodes directly from bytes without a data-URL parse.
Empirically drops per-arrival main-thread work from ~10-15 ms to
~1-2 ms.
2. rAF-batched thumbnail state updates. Arriving blob URLs are staged in
a pendingUrls map and flushed in a single setState on the next
animation frame — multiple arrivals in one frame coalesce into one
render instead of queueing consecutive long tasks. Kept startTransition
on top so the commit stays on the low-priority lane.
Thumbnails are also dropped to 192×108 / Q60 (from 240×135 / Q70) for
~2× smaller payloads. Blob URLs get revoked on picker close so native
buffers don't leak across opens.
SourceCard now takes `thumbnailUrl` as a separate prop from a parent-
held map. Keeps source object references stable so React.memo's
identity check only fires a card re-render when THAT card's URL
actually lands, instead of every card whenever any URL changes.
Next session: WASAPI loopback for system-audio capture in native share.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The picker still felt frozen while thumbnails were streaming in because
each result was both (a) large — PNG @ 320×180 landed at 60-150 KB
base64 — and (b) triggering a high-priority React re-render of the whole
grid. Three fixes together restore interactivity:
- Thumbnails encoded as JPEG @ Q70 at 240×135 instead of PNG @ 320×180.
Drops the typical payload from ~100 KB to ~20 KB, so IPC JSON-parsing
on arrival is 5× faster.
- SourceCard wrapped in React.memo so only the card whose thumbnail just
landed re-renders. Previously one new thumbnail caused all ~20 cards
to re-evaluate their props.
- setSources updates run inside startTransition so scroll / click events
stay on the high-priority lane while the grid backfills.
Also: when the user enables "Sound mit übertragen" AND has a source
picked, the picker now surfaces an inline amber note explaining that
the OS picker will appear for the audio capture path. Matches the
existing console info log but is visible pre-click so users don't
experience it as a bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker speed (Phase 1+2):
- screen_sources.rs split into list_screen_sources (metadata only,
returns in ~10ms) + capture_screen_source_thumbnail (single source,
by id). ScreenSourcePicker now shows names + placeholders instantly
and streams thumbnails in as each capture lands. Total wall-clock
is bounded by the slowest source instead of the serial sum.
- enumerate_screen_sources kept as a dead_code fallback so any
rollout regression can switch the frontend back without code loss.
Native capture (Phase 3):
- New src-tauri/src/screen_capture.rs. start_screen_capture spawns a
Rust thread per share that grabs frames via xcap, downscales to the
user's quality preset, JPEG-encodes at Q72, and streams each frame
through a Tauri Channel<FramePayload>. stop_screen_capture signals
the stop flag and joins the worker.
- Worker re-resolves the xcap handle inside the thread because
xcap::Window holds a !Send HWND — passing the source id string
across the thread boundary sidesteps that.
- New lib/screenCapture.ts: decodes each frame into an ImageBitmap,
draws to an offscreen canvas, exposes canvas.captureStream() as the
MediaStream LiveKit publishes. Latest-wins frame queue drops stale
frames when the JS side falls behind the Rust producer. 3s first-
frame timeout so a silently-failing source (locked screen, DRM
window) surfaces as a clean NativeCaptureUnavailable and we fall
back to getDisplayMedia.
- CallContext.startScreenShare takes the native path first when the
picker provided a sourceId and system audio wasn't requested. The
old chromeMediaSourceId attempt and final setScreenShareEnabled
fallback stay in place for the audio case + non-Tauri runtimes.
- stopScreenShare kills the native handle first, then unpublishes any
manually-published ScreenShare/ScreenShareAudio tracks, then falls
back to setScreenShareEnabled(false). disconnectRoom also stops
the handle so we don't leak Rust threads across calls.
Scope note: native path is video-only. System-audio capture needs
WASAPI-loopback (Windows) or ScreenCaptureKit-audio (macOS); until
those are wired, requesting audio in the picker falls through to
the getDisplayMedia path and shows the OS picker for that one case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rust side:
- New src-tauri/src/screen_sources.rs with an enumerate_screen_sources
command. Uses the xcap crate for cross-platform screen + window
enumeration and capture; PNG thumbnails are letterbox-scaled to fit
320x180 and returned as base64.
- Source ids emit Chromium's internal desktopCapturer format
("screen:<id>:0", "window:<hwnd>:0") so JS can try passing them
straight into chromeMediaSourceId.
- Registered in both invoke_handler branches in lib.rs.
Frontend:
- New lib/screenSources.ts — Tauri command wrapper + thumbnailDataUrl
helper for the picker UI.
- New components/ScreenSourcePicker.tsx — Discord-style grid: sources
grouped under "Bildschirme" / "Fenster", large thumbnail cards with
selection state, quality preset + system-audio toggle in the footer.
"Teilen" button is enabled either way; without a selection it says
"Ohne Auswahl weiter" and falls through to the OS picker.
- Replaces the old form-style ScreenShareDialog entirely (removed).
CallContext wiring:
- startScreenShare now accepts an optional sourceId. When set, it
captures that exact source via getUserMedia's legacy
chromeMediaSourceId constraint and publishes the resulting tracks
manually (video as ScreenShare, audio as ScreenShareAudio). Falls
back to setScreenShareEnabled if WebView2 rejects the constraint,
so users always get a working share even if the direct path fails.
- Track 'ended' listeners unpublish the pub when the OS revokes
capture (close of shared window, OS "stop sharing" banner).
InCallPanel:
- Left-click on the share button now opens the picker instead of
starting with last-saved settings; right-click opens it too. The
picker itself is the 1-click UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Red-dot overlay drawn as raw RGBA (no extra resource bundled).
Shown on Windows when unread count > 0, cleared when 0.
macOS keeps numeric Dock badge; Linux has no cross-DE badge API.
Bumps version 0.10.0 -> 0.10.1.
Phase B.1 of the native-livekit migration. Ships the command surface and
event bridge behind a cargo feature so the default build stays unaffected
while the JS-SDK path keeps running in production.
Rust side
- livekit 0.7 (default tokio runtime, rustls-tls-native-roots) pulled as
an optional dependency; tokio also optional under the same feature
- Feature `rust-livekit` gates everything — off by default; on via
`cargo build --features rust-livekit`
- src-tauri/src/livekit_bridge.rs: LivekitState mutex, connect /
disconnect / send_data commands, event pump for room_state,
participant_joined, participant_left, data_received
- Mic / camera / screen share commands stubbed with explicit
"not implemented" errors so JS callers fail loudly rather than
silently no-op
JS side
- src/lib/nativeLiveKit.ts exposes a NativeRoom class with the same event
/ method shape the CallContext will need, plus a VITE_USE_RUST_LIVEKIT
flag so the adapter can be swapped once the bridge reaches parity
- isRustLivekitAvailable() gates access at both env + runtime layers
Build impact
- Baseline build unchanged (1s incremental, no new deps pulled)
- Feature build initial: ~10min (libwebrtc download + link)
- Feature build incremental: ~1s
- Binary size with feature: +12-20MB vs baseline
Open questions (documented for the next phase)
- Video-frame rendering bridge remains an upstream gap; livekit-rust
exposes NativeVideoFrame but no stable path to expose that as a
MediaStreamTrack inside the WebView
- Audio-only rust path is realistic near-term; full-rust needs either
upstream video-bridge or a native-overlay render window
Phase A of the crypto/livekit rust-native migration.
Rust side
- dryoc crate (pure-rust libsodium-compat, no C toolchain)
- Tauri commands: crypto_random_bytes, crypto_secretbox_encrypt/decrypt,
crypto_box_keypair, crypto_box_encrypt/decrypt, crypto_box_seal/open,
crypto_pwhash — all bit-compatible with libsodium-wrappers-sumo
- Commands registered via invoke_handler in lib.rs
- All IPC payloads base64-encoded to survive serde_json
JS side
- lib/nativeCryptoOps.ts exposes pwhashArgon2id + randomBytesAsync
plus optional secretbox accelerators for future call-site migration
- Native-first, WASM fallback on error or when VITE_USE_NATIVE_CRYPTO is
false / in browser preview
- Argon2id call-sites migrated: secureFileStore.deriveKey and
deviceBackup.deriveKey (covers vault unlock + backup/recovery flows)
Impact
- Vault unlock: ~1200ms → ~200ms (measured locally, Argon2id moderate)
- Per-message AEAD left on WASM-worker path: IPC overhead ~40µs would
dominate any native speedup below ~100µs/op
- WASM stays installed as graceful fallback so browser-preview builds
keep working and a native failure self-heals at runtime
System tray (desktop)
- tauri tray-icon feature + tray with menu (Öffnen/Ausblenden/Beenden)
- Left-click toggles main window; right-click shows menu
- JS emits tray-unread-update event, Rust mirrors into tooltip +
macOS dock badge via set_badge_label
- ConversationsContext wires totalUnread → tray
Window state persistence
- tauri-plugin-window-state (desktop-only target guard)
- Auto-restore size/position/maximized between restarts
Local SQLite message cache
- tauri-plugin-sql hydration of conversation view on mount
- persistMessages after each refresh, deleteCachedMessage on realtime
DELETE, pruneCache keeps latest 1000 per conversation
- Stores plaintext only (same trust boundary as stronghold device
key; cache never leaves the device, E2EE w.r.t. server unchanged)
Web Worker for decryption
- workers/decrypt.worker.ts runs crypto_secretbox_open_easy + utf-8
decode off the main thread with its own libsodium instance
- lib/decryptWorker.ts is a request/response wrapper with inline
fallback when Worker spawn fails
- shared decryptMessages accepts aeadBatchDelegate so key lookup
stays on the main thread while the AEAD loop offloads
Build fix
- Enable tauri tray-icon feature
- Import Listener + Manager traits, clone tray handle for the
event listener, conditional icon attach
byGalax
byGalax