Hooks the custom screen-share picker up to a native WASAPI loopback
capture so "Mit System-Sound" no longer falls back to the OS picker on
Windows. Rust side opens the default render endpoint, channels 48 kHz
f32 stereo to an AudioWorklet, which feeds a MediaStreamDestination for
LiveKit to publish as ScreenShareAudio. Ring buffer sized for latency
(80 ms target, drop-to-target on overflow) and the AudioContext is
resumed eagerly so initial burstiness can't pile up.
Adds a temporary attachTrack:audio diagnostic log to confirm source
tagging matches between old and new clients.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The real reason every thumbnail card stayed on the placeholder was a
reference-aliasing bug in the flush closure. `const batch = pendingUrls`
captured the same object; `delete (pendingUrls)[k]` for each key then
emptied `batch` too, because they were the same reference. By the time
`setThumbnailUrls(prev => ({ ...prev, ...batch }))` ran, batch was {}
and the state never picked up any URL — every card rendered the empty
placeholder icon.
Fixed by aliasing first, then replacing pendingUrls with a fresh empty
object (let instead of const on the outer binding). The cloned `batch`
retains its entries for the spread; any new arrivals during the commit
land in the new empty pendingUrls and coalesce into the next frame.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker still stuttered during load because the main thread was stuck
parsing 20+ inbound IPC messages, each carrying 15-25 KB of JSON-wrapped
base64. Two changes compound to fix this:
1. Binary IPC. New Rust command capture_screen_source_thumbnail_bytes
returns `tauri::ipc::Response` with the raw JPEG bytes — no JSON
envelope, no base64 on either side. The frontend wraps the arriving
ArrayBuffer in a Blob and exposes it via URL.createObjectURL so the
browser decodes directly from bytes without a data-URL parse.
Empirically drops per-arrival main-thread work from ~10-15 ms to
~1-2 ms.
2. rAF-batched thumbnail state updates. Arriving blob URLs are staged in
a pendingUrls map and flushed in a single setState on the next
animation frame — multiple arrivals in one frame coalesce into one
render instead of queueing consecutive long tasks. Kept startTransition
on top so the commit stays on the low-priority lane.
Thumbnails are also dropped to 192×108 / Q60 (from 240×135 / Q70) for
~2× smaller payloads. Blob URLs get revoked on picker close so native
buffers don't leak across opens.
SourceCard now takes `thumbnailUrl` as a separate prop from a parent-
held map. Keeps source object references stable so React.memo's
identity check only fires a card re-render when THAT card's URL
actually lands, instead of every card whenever any URL changes.
Next session: WASAPI loopback for system-audio capture in native share.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous pass moved to JPEG + startTransition but the grid still froze.
Root causes that survived:
1. React.memo was broken — the parent re-created the inline
`onClick={() => onSelect(src.id)}` arrow on every render, so memo's
reference check always triggered a fresh render on every card even
though nothing visible had changed. Fixed by passing `onSelect` as a
stable prop and constructing the click handler inside the memoized
child.
2. 20 data-URL `<img>` sources getting decoded more or less at once gave
the compositor enough work to make scroll feel laggy. `decoding="async"`
punts decode to the browser's image thread; `loading="lazy"` skips it
entirely for cards outside the viewport.
3. Concurrency at 4 was still high enough for Windows GDI BitBlt /
PrintWindow to contend for the desktop compositor — the whole Tauri
window stuttered because the OS-level screen capture was saturating
the graphics pipeline. Dropped to 2 concurrent captures; total load
takes a touch longer but the picker stays interactive throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The picker still felt frozen while thumbnails were streaming in because
each result was both (a) large — PNG @ 320×180 landed at 60-150 KB
base64 — and (b) triggering a high-priority React re-render of the whole
grid. Three fixes together restore interactivity:
- Thumbnails encoded as JPEG @ Q70 at 240×135 instead of PNG @ 320×180.
Drops the typical payload from ~100 KB to ~20 KB, so IPC JSON-parsing
on arrival is 5× faster.
- SourceCard wrapped in React.memo so only the card whose thumbnail just
landed re-renders. Previously one new thumbnail caused all ~20 cards
to re-evaluate their props.
- setSources updates run inside startTransition so scroll / click events
stay on the high-priority lane while the grid backfills.
Also: when the user enables "Sound mit übertragen" AND has a source
picked, the picker now surfaces an inline amber note explaining that
the OS picker will appear for the audio capture path. Matches the
existing console info log but is visible pre-click so users don't
experience it as a bug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Volume crash:
- setParticipantVolume / setScreenShareVolume propagated values up to 2.0
(200%) to the per-track GainNode, but also called applyToAttachedElements
which set the raw HTMLAudioElement.volume — that property is hard-clamped
to [0, 1] and throws IndexSizeError above 1. Clip the element-path apply
at 1.0. WebAudio GainNode keeps doing the actual amplification.
Picker freeze:
- Firing ~20 captureScreenSourceThumbnail invokes in parallel caused
perceptible input freezes while each ~100KB base64 result arrived and
triggered a setState. Bounded the worker pool to 4 concurrent captures
with a queue — overall wall-clock is nearly identical and the grid stays
scrollable / clickable throughout the load.
Native-path diagnostics:
- Previous logs only fired on non-NativeCaptureUnavailable errors, so
users couldn't tell whether the native path was skipped (audio toggle
on, no sourceId) or attempted-and-failed. Added explicit info logs for
each skip reason plus an always-on warn with the underlying error when
the try block throws. Makes the next debug pass on screenshare much
quicker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picker speed (Phase 1+2):
- screen_sources.rs split into list_screen_sources (metadata only,
returns in ~10ms) + capture_screen_source_thumbnail (single source,
by id). ScreenSourcePicker now shows names + placeholders instantly
and streams thumbnails in as each capture lands. Total wall-clock
is bounded by the slowest source instead of the serial sum.
- enumerate_screen_sources kept as a dead_code fallback so any
rollout regression can switch the frontend back without code loss.
Native capture (Phase 3):
- New src-tauri/src/screen_capture.rs. start_screen_capture spawns a
Rust thread per share that grabs frames via xcap, downscales to the
user's quality preset, JPEG-encodes at Q72, and streams each frame
through a Tauri Channel<FramePayload>. stop_screen_capture signals
the stop flag and joins the worker.
- Worker re-resolves the xcap handle inside the thread because
xcap::Window holds a !Send HWND — passing the source id string
across the thread boundary sidesteps that.
- New lib/screenCapture.ts: decodes each frame into an ImageBitmap,
draws to an offscreen canvas, exposes canvas.captureStream() as the
MediaStream LiveKit publishes. Latest-wins frame queue drops stale
frames when the JS side falls behind the Rust producer. 3s first-
frame timeout so a silently-failing source (locked screen, DRM
window) surfaces as a clean NativeCaptureUnavailable and we fall
back to getDisplayMedia.
- CallContext.startScreenShare takes the native path first when the
picker provided a sourceId and system audio wasn't requested. The
old chromeMediaSourceId attempt and final setScreenShareEnabled
fallback stay in place for the audio case + non-Tauri runtimes.
- stopScreenShare kills the native handle first, then unpublishes any
manually-published ScreenShare/ScreenShareAudio tracks, then falls
back to setScreenShareEnabled(false). disconnectRoom also stops
the handle so we don't leak Rust threads across calls.
Scope note: native path is video-only. System-audio capture needs
WASAPI-loopback (Windows) or ScreenCaptureKit-audio (macOS); until
those are wired, requesting audio in the picker falls through to
the getDisplayMedia path and shows the OS picker for that one case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rust side:
- New src-tauri/src/screen_sources.rs with an enumerate_screen_sources
command. Uses the xcap crate for cross-platform screen + window
enumeration and capture; PNG thumbnails are letterbox-scaled to fit
320x180 and returned as base64.
- Source ids emit Chromium's internal desktopCapturer format
("screen:<id>:0", "window:<hwnd>:0") so JS can try passing them
straight into chromeMediaSourceId.
- Registered in both invoke_handler branches in lib.rs.
Frontend:
- New lib/screenSources.ts — Tauri command wrapper + thumbnailDataUrl
helper for the picker UI.
- New components/ScreenSourcePicker.tsx — Discord-style grid: sources
grouped under "Bildschirme" / "Fenster", large thumbnail cards with
selection state, quality preset + system-audio toggle in the footer.
"Teilen" button is enabled either way; without a selection it says
"Ohne Auswahl weiter" and falls through to the OS picker.
- Replaces the old form-style ScreenShareDialog entirely (removed).
CallContext wiring:
- startScreenShare now accepts an optional sourceId. When set, it
captures that exact source via getUserMedia's legacy
chromeMediaSourceId constraint and publishes the resulting tracks
manually (video as ScreenShare, audio as ScreenShareAudio). Falls
back to setScreenShareEnabled if WebView2 rejects the constraint,
so users always get a working share even if the direct path fails.
- Track 'ended' listeners unpublish the pub when the OS revokes
capture (close of shared window, OS "stop sharing" banner).
InCallPanel:
- Left-click on the share button now opens the picker instead of
starting with last-saved settings; right-click opens it too. The
picker itself is the 1-click UX.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
byGalax