byGalax/ChatApp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: call UX overhaul — deafen sync, share dialog, fullscreen redesign

Browse Source 
Speaking ring:
- Switch useActiveSpeakers from LiveKit's smoothed isSpeaking / server-
  batched ActiveSpeakersChanged to Web Audio API AnalyserNode on each
  participant's raw audio MediaStreamTrack. Poll 50ms, RMS threshold 0.03,
  250ms hold. Feels real-time vs the old ~500ms lag
- Defensive syncProbe on every tick so probes catch up if TrackPublished
  missed (local mic publish race on join)
- Universal speaking overlay on tile (3px emerald border + inset glow,
  z-10) so video mode shows the ring too, not just audio mode

Screen sharing:
- Separate "screen" tile per sharer so the sharer's avatar tile stays
  intact with its speaking ring. Tile.id is kind-prefixed (user:xxx /
  screen:xxx) so focus tracking distinguishes them
- New ScreenShareDialog (quality preset + fps override + displaySurface
  hint) opens on the share button. startScreenShare / stopScreenShare
  actions in CallContext replace the one-shot toggle
- ScreenShareViewer: plain CSS-only fullscreen overlay (Tauri WKWebView
  doesn't implement requestFullscreen), always `h-full w-full
  object-contain`, Esc exits

Camera:
- toggleCamera action in CallContext tracks isCameraEnabled
- VideoStub renders real <video> srcObject for the participant's camera
  MediaStreamTrack; local preview is mirrored
- Tile video track resolves to Track.Source.Camera publications of the
  LocalParticipant / each RemoteParticipant
- Room listens for TrackMuted / TrackUnmuted and re-publishes remote
  state so peers switch to avatar placeholder when a camera is disabled

Deafen:
- New isDeafened state + toggleDeafen action. Sets `muted = true` on all
  attached `<audio[data-livekit-track]>` plus mutes fresh ones on attach
  via module-level flag
- Broadcast state over the LiveKit data channel
  ({type:'presence', deafened}) so peers can render the headphones-off
  badge. Attributes API not used because the self-hosted server may run
  older LiveKit versions
- remoteDeafen: Record<identity, bool> exposed via context, bumped on
  DataReceived and re-broadcast on ParticipantConnected

Incoming video call:
- acceptIncoming takes an optional CallKind override so the receiver can
  answer a video invite with audio only or promote an audio invite to
  video on accept
- IncomingCallPanel shows two accept buttons (audio + video) when the
  invite is a video call

Audio devices:
- audioSettings adds inputDeviceId + outputDeviceId, persisted
- CallContext uses them on setMicrophoneEnabled, plus new
  setAudioInputDevice / setAudioOutputDevice hot-swap actions.
  Output swap applies setSinkId to every attached remote-audio element
  since LiveKit's own switchActiveDevice only tracks elements it
  attached itself
- SettingsPage "Mikrofon" + "Ausgabegerät" selects with devicechange
  listener and a permission-probe button

Fullscreen mode:
- Replaced absolute-positioned speaker + floating thumbnails with a real
  flex layout. Default = even grid of all tiles. Clicking a tile flips
  to big-speaker + horizontal thumbnail strip. Click focused tile =
  back to grid
- Controls overlay pinned bottom; content wrapper has pb-24 so tiles
  never sit behind the toolbar
- Grid now uses explicit grid-rows-* so cells get a defined 1fr height
  (without it, video intrinsic dimensions blew tiles past the container
  bounds on Windows)

UI chips:
- Mic-off badge combines isMuted flag AND
  localParticipant.isMicrophoneEnabled, so a user with no mic / denied
  permission sees the badge + the toolbar button red even though they
  never pressed mute
- Deafen badge on tile chips for local + remote (remote driven by the
  data-channel broadcast)
This commit is contained in:
Dennis Landmann
2026-04-21 00:02:43 +02:00
parent a4c9b959a9
commit da85f0ba54
10 changed files with 1129 additions and 241 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/desktop/src/context/CallContext.tsx
+242 -40
View File
@@ -48,8 +48,11 @@ import {
isE2EESupported,
} from '../lib/callE2EE';
import {
type DisplaySurfaceHint,
getPresetParams,
getScreenShareSettings,
type ScreenSharePreset,
updateScreenShareSettings,
} from '../lib/screenShareSettings';
import { devLocalSecretStore } from '../lib/secretStore';
import { supabase } from '../lib/supabase';
@@ -102,6 +105,10 @@ interface CallContextValue {
isMuted: boolean;
isE2EEActive: boolean;
isScreenSharing: boolean;
isCameraEnabled: boolean;
isDeafened: boolean;
/** identity -> their deafen state, received via data channel. */
remoteDeafen: Record<string, boolean>;
// Zero or more remote screen shares (LiveKit supports multiple simultaneous).
remoteScreenShares: RemoteScreenShare[];
// Remembers the conversation of the last call we left so a sidebar widget
@@ -113,11 +120,21 @@ interface CallContextValue {
// Actions:
startCall: (conversationId: string, mediaKind?: CallKind) => Promise<void>;
joinActiveCall: (conversationId: string, mediaKind?: CallKind) => Promise<void>;
acceptIncoming: () => Promise<void>;
acceptIncoming: (override?: CallKind) => Promise<void>;
rejectIncoming: () => void;
hangup: () => Promise<void>;
toggleMute: () => void;
toggleScreenShare: () => Promise<void>;
startScreenShare: (
overrides?: Partial<{
preset: ScreenSharePreset;
displaySurface: DisplaySurfaceHint;
framerate: number | null;
}>,
) => Promise<void>;
stopScreenShare: () => Promise<void>;
toggleCamera: () => Promise<void>;
toggleDeafen: () => void;
dismissLastCall: () => void;
setCallMode: (mode: CallMode) => void;
setFocusedId: (id: string | null) => void;
@@ -149,6 +166,8 @@ export function CallProvider({ children }: { children: ReactNode }) {
const [isMuted, setIsMuted] = useState(false);
const [isE2EEActive, setIsE2EEActive] = useState(false);
const [isScreenSharing, setIsScreenSharing] = useState(false);
const [isCameraEnabled, setIsCameraEnabled] = useState(false);
const [isDeafened, setIsDeafened] = useState(false);
const [remoteScreenShares, setRemoteScreenShares] = useState<RemoteScreenShare[]>([]);
const [lastCallConversationId, setLastCallConversationId] = useState<string | null>(null);
const [callMode, setCallModeState] = useState<CallMode>('grid');
@@ -167,6 +186,10 @@ export function CallProvider({ children }: { children: ReactNode }) {
// For 1:1 calls the set has one entry so behaviour is unchanged; for
// groups, a single reject doesn't terminate the call while others ring.
const pendingPeersRef = useRef<Set<string>>(new Set());
// identity -> their current deafen state. Populated via LiveKit data
// channel messages ({ type: 'presence', deafened: bool }). Exposed as
// state so consumer components re-render on change.
const [remoteDeafen, setRemoteDeafen] = useState<Record<string, boolean>>({});
const stateRef = useRef<CallState>(state);
stateRef.current = state;
// Keep latest conversations accessible from signal-channel closures without
@@ -243,6 +266,10 @@ export function CallProvider({ children }: { children: ReactNode }) {
setRemoteParticipants([]);
setRemoteScreenShares([]);
setIsScreenSharing(false);
setIsCameraEnabled(false);
setIsDeafened(false);
deafenedActive = false;
setRemoteDeafen({});
setIsE2EEActive(false);
const pres = presenceChannelRef.current;
@@ -463,6 +490,45 @@ export function CallProvider({ children }: { children: ReactNode }) {
}
});
// Mute/unmute a camera doesn't publish or unpublish — the publication
// stays, just its `muted` flag flips. Without this listener, remote
// participants who toggle video mid-call appear as a frozen last frame
// or (worse) a black tile on every other client. Bumping the
// remoteParticipants state reference forces buildTiles to re-read
// `isCameraEnabled` and swap to the avatar placeholder.
const bumpParticipants = () => {
setRemoteParticipants(Array.from(r.remoteParticipants.values()));
};
r.on(RoomEvent.TrackMuted, bumpParticipants);
r.on(RoomEvent.TrackUnmuted, bumpParticipants);
// Remote deafen state is broadcast via the LiveKit data channel. We
// store incoming states in `remoteDeafenMapRef` and bump participants
// so buildTiles re-reads it.
r.on(
RoomEvent.DataReceived,
(payload: Uint8Array, participant?: RemoteParticipant | undefined) => {
if (!participant?.identity) return;
try {
const text = new TextDecoder().decode(payload);
const msg = JSON.parse(text) as { type?: string; deafened?: boolean };
if (msg.type === 'presence' && typeof msg.deafened === 'boolean') {
const id: string = participant.identity;
const deafened: boolean = msg.deafened;
setRemoteDeafen((prev) => {
if (prev[id] === deafened) return prev;
return { ...prev, [id]: deafened };
});
}
} catch {
/* ignore malformed */
}
},
);
// When someone joins, re-send our current deafen state so they know.
r.on(RoomEvent.ParticipantConnected, () => {
void broadcastPresence(r, deafenedActive);
});
// Track my own screen-share state via LocalTrack events so the toggle
// stays in sync if the user stops sharing via the browser's native UI.
r.on(RoomEvent.LocalTrackPublished, (publication) => {
@@ -506,6 +572,7 @@ export function CallProvider({ children }: { children: ReactNode }) {
if (mediaKind === 'video') {
try {
await r.localParticipant.setCameraEnabled(true);
setIsCameraEnabled(true);
} catch (camErr: unknown) {
console.error('setCameraEnabled failed', camErr);
}
@@ -650,23 +717,31 @@ export function CallProvider({ children }: { children: ReactNode }) {
[myId, joinRoom, disconnectRoom],
);
const acceptIncoming = useCallback(async () => {
const s = stateRef.current;
if (s.kind !== 'incoming' || !myId) return;
const { callId, conversationId, mediaKind } = s;
everConnectedRef.current = false;
setLastCallConversationId(null);
setState({ kind: 'connecting', callId, conversationId, mediaKind });
try {
await joinRoom(conversationId, mediaKind, callId);
} catch (err: unknown) {
setState({
kind: 'error',
message: err instanceof Error ? err.message : 'join failed',
});
await disconnectRoom();
}
}, [myId, joinRoom, disconnectRoom]);
const acceptIncoming = useCallback(
async (override?: CallKind) => {
const s = stateRef.current;
if (s.kind !== 'incoming' || !myId) return;
const { callId, conversationId } = s;
// Caller's `mediaKind` is the INVITE kind (what they started with). The
// receiver can accept with audio even if the caller rang as video, or
// upgrade an audio invite to video on accept. `override` picks the
// receiver's choice.
const mediaKind: CallKind = override ?? s.mediaKind;
everConnectedRef.current = false;
setLastCallConversationId(null);
setState({ kind: 'connecting', callId, conversationId, mediaKind });
try {
await joinRoom(conversationId, mediaKind, callId);
} catch (err: unknown) {
setState({
kind: 'error',
message: err instanceof Error ? err.message : 'join failed',
});
await disconnectRoom();
}
},
[myId, joinRoom, disconnectRoom],
);
const rejectIncoming = useCallback(() => {
const s = stateRef.current;
@@ -740,33 +815,129 @@ export function CallProvider({ children }: { children: ReactNode }) {
});
}, []);
const toggleScreenShare = useCallback(async () => {
const startScreenShare = useCallback(
async (
overrides?: Partial<{
preset: ScreenSharePreset;
displaySurface: DisplaySurfaceHint;
framerate: number | null;
}>,
) => {
const r = roomRef.current;
if (!r) return;
const lp = r.localParticipant;
if (lp.isScreenShareEnabled) return;
// Persist the user's choice so subsequent shares use the same config
// without re-opening the picker unless they want to change something.
const settings = getScreenShareSettings();
const preset = overrides?.preset ?? settings.preset;
const displaySurface =
overrides?.displaySurface !== undefined
? overrides.displaySurface
: settings.displaySurface;
const framerateOverride =
overrides?.framerate !== undefined
? overrides.framerate
: settings.framerateOverride;
updateScreenShareSettings({ preset, displaySurface, framerateOverride });
const ssParams = getPresetParams(preset);
const fps = framerateOverride ?? ssParams.framerate;
try {
await lp.setScreenShareEnabled(true, {
audio: false,
...(ssParams.dims
? {
resolution: {
width: ssParams.dims.width,
height: ssParams.dims.height,
frameRate: fps,
},
}
: {
resolution: {
width: 3840,
height: 2160,
frameRate: fps,
},
}),
// Hints the OS picker to pre-filter by source kind. `null` = no
// filter (show both). Cast because TS lib.dom doesn't know the
// field yet on all branches.
...(displaySurface
? ({ displaySurface } as { displaySurface: DisplaySurfaceHint })
: {}),
contentHint: 'detail',
});
setIsScreenSharing(true);
} catch (err: unknown) {
console.error('setScreenShareEnabled failed', err);
}
},
[],
);
const stopScreenShare = useCallback(async () => {
const r = roomRef.current;
if (!r) return;
const lp = r.localParticipant;
const nextOn = !lp.isScreenShareEnabled;
if (!lp.isScreenShareEnabled) return;
try {
const ssParams = getPresetParams(getScreenShareSettings().preset);
await lp.setScreenShareEnabled(nextOn, {
audio: false,
// Omitting `resolution` lets the browser return native source size —
// best possible input quality. Fixed presets pass explicit dims so
// the encoder has a predictable target.
...(ssParams.dims
? {
resolution: {
width: ssParams.dims.width,
height: ssParams.dims.height,
frameRate: ssParams.framerate,
},
}
: {}),
contentHint: 'detail',
});
setIsScreenSharing(nextOn);
await lp.setScreenShareEnabled(false);
setIsScreenSharing(false);
} catch (err: unknown) {
console.error('setScreenShareEnabled failed', err);
// User cancelled or permission denied — leave state as-is.
console.error('stopScreenShare failed', err);
}
}, []);
// Legacy toggle kept for convenience elsewhere — opens/closes with the
// last-persisted settings and no picker UI.
const toggleScreenShare = useCallback(async () => {
const r = roomRef.current;
if (!r) return;
if (r.localParticipant.isScreenShareEnabled) {
await stopScreenShare();
} else {
await startScreenShare();
}
}, [startScreenShare, stopScreenShare]);
const toggleDeafen = useCallback(() => {
setIsDeafened((prev) => {
const next = !prev;
deafenedActive = next;
// Apply to every currently-attached remote-audio element. Fresh tracks
// that attach during a deafened session are muted in attachTrack above.
const els = document.querySelectorAll<HTMLAudioElement>(
'audio[data-livekit-track]',
);
els.forEach((el) => {
el.muted = next;
});
// Broadcast via LiveKit data channel so peers' UIs can show the
// headphones-off badge. Data channel works on any LiveKit server
// version, unlike `setAttributes` which requires a newer server.
const r = roomRef.current;
if (r) void broadcastPresence(r, next);
return next;
});
}, []);
const toggleCamera = useCallback(async () => {
const r = roomRef.current;
if (!r) return;
const lp = r.localParticipant;
const nextOn = !lp.isCameraEnabled;
try {
await lp.setCameraEnabled(nextOn);
setIsCameraEnabled(nextOn);
} catch (err: unknown) {
console.error('setCameraEnabled failed', err);
// Permission denied / no camera — keep state in sync with actual
// publication state so the button doesn't lie.
setIsCameraEnabled(lp.isCameraEnabled);
}
}, []);
@@ -1054,6 +1225,9 @@ export function CallProvider({ children }: { children: ReactNode }) {
isMuted,
isE2EEActive,
isScreenSharing,
isCameraEnabled,
isDeafened,
remoteDeafen,
remoteScreenShares,
lastCallConversationId,
callMode,
@@ -1065,6 +1239,10 @@ export function CallProvider({ children }: { children: ReactNode }) {
hangup,
toggleMute,
toggleScreenShare,
startScreenShare,
stopScreenShare,
toggleCamera,
toggleDeafen,
dismissLastCall,
setCallMode,
setFocusedId,
@@ -1078,6 +1256,9 @@ export function CallProvider({ children }: { children: ReactNode }) {
isMuted,
isE2EEActive,
isScreenSharing,
isCameraEnabled,
isDeafened,
remoteDeafen,
remoteScreenShares,
lastCallConversationId,
callMode,
@@ -1089,6 +1270,10 @@ export function CallProvider({ children }: { children: ReactNode }) {
hangup,
toggleMute,
toggleScreenShare,
startScreenShare,
stopScreenShare,
toggleCamera,
toggleDeafen,
dismissLastCall,
setCallMode,
setFocusedId,
@@ -1106,6 +1291,22 @@ export function useCall(): CallContextValue {
return ctx;
}
// Shared flag so attachTrack (called from LiveKit event listeners, outside the
// React component) can apply the current deafen state to freshly-attached
// audio elements. Toggled by toggleDeafen in sync with the React state.
let deafenedActive = false;
async function broadcastPresence(room: Room, deafened: boolean): Promise<void> {
try {
const payload = new TextEncoder().encode(
JSON.stringify({ type: 'presence', deafened }),
);
await room.localParticipant.publishData(payload, { reliable: true });
} catch (err: unknown) {
console.warn('broadcastPresence failed', err);
}
}
function attachTrack(
track: RemoteTrack,
_publication: RemoteTrackPublication,
@@ -1117,6 +1318,7 @@ function attachTrack(
audio.autoplay = true;
audio.setAttribute('playsinline', 'true');
audio.setAttribute('data-livekit-track', track.sid ?? '');
if (deafenedActive) audio.muted = true;
document.body.appendChild(audio);
// Apply persisted sinkId so the element routes to the user's chosen
// speaker/headphone from the start (LiveKit's own `switchActiveDevice`