feat(call): native screen-capture pipeline + faster picker

Picker speed (Phase 1+2):
- screen_sources.rs split into list_screen_sources (metadata only,
  returns in ~10ms) + capture_screen_source_thumbnail (single source,
  by id). ScreenSourcePicker now shows names + placeholders instantly
  and streams thumbnails in as each capture lands. Total wall-clock
  is bounded by the slowest source instead of the serial sum.
- enumerate_screen_sources kept as a dead_code fallback so any
  rollout regression can switch the frontend back without code loss.

Native capture (Phase 3):
- New src-tauri/src/screen_capture.rs. start_screen_capture spawns a
  Rust thread per share that grabs frames via xcap, downscales to the
  user's quality preset, JPEG-encodes at Q72, and streams each frame
  through a Tauri Channel<FramePayload>. stop_screen_capture signals
  the stop flag and joins the worker.
- Worker re-resolves the xcap handle inside the thread because
  xcap::Window holds a !Send HWND — passing the source id string
  across the thread boundary sidesteps that.
- New lib/screenCapture.ts: decodes each frame into an ImageBitmap,
  draws to an offscreen canvas, exposes canvas.captureStream() as the
  MediaStream LiveKit publishes. Latest-wins frame queue drops stale
  frames when the JS side falls behind the Rust producer. 3s first-
  frame timeout so a silently-failing source (locked screen, DRM
  window) surfaces as a clean NativeCaptureUnavailable and we fall
  back to getDisplayMedia.
- CallContext.startScreenShare takes the native path first when the
  picker provided a sourceId and system audio wasn't requested. The
  old chromeMediaSourceId attempt and final setScreenShareEnabled
  fallback stay in place for the audio case + non-Tauri runtimes.
- stopScreenShare kills the native handle first, then unpublishes any
  manually-published ScreenShare/ScreenShareAudio tracks, then falls
  back to setScreenShareEnabled(false). disconnectRoom also stops
  the handle so we don't leak Rust threads across calls.

Scope note: native path is video-only. System-audio capture needs
WASAPI-loopback (Windows) or ScreenCaptureKit-audio (macOS); until
those are wired, requesting audio in the picker falls through to
the getDisplayMedia path and shows the OS picker for that one case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
byGalax
2026-04-22 21:42:52 +02:00
parent 8f9b823d69
commit c3e0c47d32
7 changed files with 752 additions and 40 deletions
+38
View File
@@ -18,6 +18,44 @@ export interface ScreenSource {
height: number;
}
// Fast, metadata-only list. The picker uses this first so names show up
// immediately; thumbnails stream in via captureScreenSourceThumbnail below.
export async function listScreenSources(): Promise<ScreenSource[]> {
if (!isTauriRuntime()) return [];
try {
const { invoke } = await import('@tauri-apps/api/core');
const raw = await invoke<ScreenSource[]>('list_screen_sources');
return raw ?? [];
} catch (err: unknown) {
console.warn('list_screen_sources failed', err);
return [];
}
}
// Single-source thumbnail capture. Called N times in parallel from the
// picker so Tauri's command thread pool runs captures concurrently — total
// wall-clock time becomes bounded by the slowest source, not the sum.
// Returns the base64 PNG or null when the source disappeared / capture
// permission was denied.
export async function captureScreenSourceThumbnail(
sourceId: string,
): Promise<string | null> {
if (!isTauriRuntime()) return null;
try {
const { invoke } = await import('@tauri-apps/api/core');
const result = await invoke<string | null>('capture_screen_source_thumbnail', {
sourceId,
});
return result ?? null;
} catch (err: unknown) {
console.warn('capture_screen_source_thumbnail failed', { sourceId, err });
return null;
}
}
// Legacy single-shot variant. Captures everything serially on the Rust side
// before returning. Prefer listScreenSources + captureScreenSourceThumbnail
// for user-facing flows — they feel 510× more responsive in practice.
export async function enumerateScreenSources(): Promise<ScreenSource[]> {
if (!isTauriRuntime()) return [];
try {