feat(call): native screen-capture pipeline + faster picker
Picker speed (Phase 1+2): - screen_sources.rs split into list_screen_sources (metadata only, returns in ~10ms) + capture_screen_source_thumbnail (single source, by id). ScreenSourcePicker now shows names + placeholders instantly and streams thumbnails in as each capture lands. Total wall-clock is bounded by the slowest source instead of the serial sum. - enumerate_screen_sources kept as a dead_code fallback so any rollout regression can switch the frontend back without code loss. Native capture (Phase 3): - New src-tauri/src/screen_capture.rs. start_screen_capture spawns a Rust thread per share that grabs frames via xcap, downscales to the user's quality preset, JPEG-encodes at Q72, and streams each frame through a Tauri Channel<FramePayload>. stop_screen_capture signals the stop flag and joins the worker. - Worker re-resolves the xcap handle inside the thread because xcap::Window holds a !Send HWND — passing the source id string across the thread boundary sidesteps that. - New lib/screenCapture.ts: decodes each frame into an ImageBitmap, draws to an offscreen canvas, exposes canvas.captureStream() as the MediaStream LiveKit publishes. Latest-wins frame queue drops stale frames when the JS side falls behind the Rust producer. 3s first- frame timeout so a silently-failing source (locked screen, DRM window) surfaces as a clean NativeCaptureUnavailable and we fall back to getDisplayMedia. - CallContext.startScreenShare takes the native path first when the picker provided a sourceId and system audio wasn't requested. The old chromeMediaSourceId attempt and final setScreenShareEnabled fallback stay in place for the audio case + non-Tauri runtimes. - stopScreenShare kills the native handle first, then unpublishes any manually-published ScreenShare/ScreenShareAudio tracks, then falls back to setScreenShareEnabled(false). disconnectRoom also stops the handle so we don't leak Rust threads across calls. Scope note: native path is video-only. System-audio capture needs WASAPI-loopback (Windows) or ScreenCaptureKit-audio (macOS); until those are wired, requesting audio in the picker falls through to the getDisplayMedia path and shows the OS picker for that one case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,8 @@ import {
|
||||
updateScreenShareSettings,
|
||||
} from '../lib/screenShareSettings';
|
||||
import {
|
||||
enumerateScreenSources,
|
||||
captureScreenSourceThumbnail,
|
||||
listScreenSources,
|
||||
type ScreenSource,
|
||||
thumbnailDataUrl,
|
||||
} from '../lib/screenSources';
|
||||
@@ -47,9 +48,11 @@ export function ScreenSourcePicker({ open, onClose, onStart }: Props) {
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
// Re-enumerate every time the picker opens so closed windows + new ones
|
||||
// stay accurate. A previous stale list would surface sources the user
|
||||
// can't actually share anymore.
|
||||
// Two-phase load: (1) fast list returns names + placeholders so the grid
|
||||
// paints instantly, (2) fire one thumbnail capture per source in
|
||||
// parallel. Thumbnails fill in as each capture completes — Tauri's
|
||||
// command thread pool runs them concurrently so wall-clock is bounded
|
||||
// by the slowest source, not the sum.
|
||||
useEffect(() => {
|
||||
if (!open) {
|
||||
setSources(null);
|
||||
@@ -59,9 +62,27 @@ export function ScreenSourcePicker({ open, onClose, onStart }: Props) {
|
||||
}
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
const list = await enumerateScreenSources();
|
||||
const list = await listScreenSources();
|
||||
if (cancelled) return;
|
||||
setSources(list);
|
||||
// Fan out thumbnail captures. No Promise.all — we want each result
|
||||
// to render as it lands, not wait for the full batch. The id-based
|
||||
// setState patch means the slowest source can still be in flight
|
||||
// while the user already picked one of the fast ones.
|
||||
for (const src of list) {
|
||||
void (async () => {
|
||||
const png = await captureScreenSourceThumbnail(src.id);
|
||||
if (cancelled || png === null) return;
|
||||
setSources((prev) => {
|
||||
if (!prev) return prev;
|
||||
const idx = prev.findIndex((s) => s.id === src.id);
|
||||
if (idx === -1) return prev;
|
||||
const next = prev.slice();
|
||||
next[idx] = { ...prev[idx]!, thumbnailPng: png };
|
||||
return next;
|
||||
});
|
||||
})();
|
||||
}
|
||||
})();
|
||||
return () => {
|
||||
cancelled = true;
|
||||
|
||||
Reference in New Issue
Block a user