A camera that can be turned on in a call already running

The missing half of #189. A call placed as a video call already opened a camera; a call placed as a voice call could never grow one, and the button was hidden unless the call was on a media server - so on the two shapes of Matrix call that do not use one, there was no camera at all. Added and removed rather than enabled and disabled. A disabled track keeps sending - a black rectangle at whatever frame rate the encoder feels like - so the far end sees a dark square where a face was and cannot tell that from a camera that has frozen. Removing the track ends it properly, and every other client then draws the person's name. The loopback harness below measures exactly this difference: with the old enabled=false the far end still counts one live video track after the camera is switched off, and with the track removed it counts none. One gesture for all three shapes of call - a media server, a mesh between several people, a single connection between two - because the button is one button and the person pressing it is asking one thing. Which shape this is belongs to the call, not to the control. On a mesh the camera is still opened once and lent to every leg, since five connections asking the machine for the same camera get one capture and four refusals. Renegotiations are now serialised. Turning a camera on and sharing a screen a second later are two changes to one connection, and starting the second offer while the first is unanswered leaves a peer connection neither end can recover from. A remote track that is taken away is dropped from the stream it arrived on, so a tile stops drawing the last frame it received - Chromium reports that as a mute before it reports it as an end, and on a removed track the mute is often all there is. cameraOn is asked of the call rather than assumed, in both branches: that block runs on every phase change, and a fixed answer put the button back to "off" the moment a call connected. The call UI is unchanged in shape and deliberately not Matrix-shaped - Discord's camera (#180) and screen share (#14) get the same row of buttons when they arrive. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 18:35:29 -04:00
parent 3c5fbfc71d
commit f965642a55
9 changed files with 391 additions and 46 deletions
@@ -169,7 +169,9 @@ called `subscribe` for that buffer.
  polls, stickers, knocking, reporting, room moderation, ignore lists, and a room directory
  search that asks every homeserver this account knows at once. Calls both ways: one-to-one
  signalling, and the group calls Element holds on a LiveKit media server, with the media keys
-  the room passes round.
+  the room passes round. Camera and screen share in all three shapes of call - a single
+  connection, a mesh, or a media server - and either can be turned on part-way through, which
+  is a renegotiation rather than a new call.
 - **Kick** - the streaming site's chat, over its Pusher socket. Joins by streamer handle, imports
  the account's follows, and carries the three emote tiers with subscriber gating, redemptions,
  subscriptions, gifted subs and raids, moderation, and polls and predictions that can be
@@ -22,8 +22,9 @@ again and everything is where you left it.
 - **It stays connected.** The daemon keeps IRC registered, Matrix syncing and Tor circuits
  open while the window is shut. Quitting says a proper goodbye rather than dropping the
  socket, so you don't come back to a ghosted nick.
- **Voice calls.** Real audio on Discord and on Matrix, including the group calls Element
-  holds on a media server. No video or screen share yet.
+- **Calls.** Real audio on Discord and on Matrix, including the group calls Element holds
+  on a media server. On Matrix the camera and the screen both work, and either can be turned
+  on part-way through a call. Discord is voice only for now.
 - **Matrix, properly encrypted.** End-to-end encryption with device verification by emoji or
  QR, cross-signing, key backup, and encrypted attachments.
 - **Sneedchat over Tor.** An embedded Tor client, the site's proof-of-work gate and its login
@@ -0,0 +1,129 @@
+// Two ends of a real call in one page, wired to each other.
+//
+// The claim under test is the one that cannot be checked by reading: that a
+// camera turned on in a call already running reaches the other end, and that
+// turning it off again takes it away rather than freezing it. That is
+// offer/answer, a renegotiation each way and a track appearing on a stream
+// somebody else owns - four things no unit test touches.
+//
+// Real RTCPeerConnections over loopback, and a real captured video track from
+// a canvas rather than a webcam, so this runs anywhere.
+import { Call } from '../../src/renderer/src/lib/webrtc'
+
+const out: string[] = []
+const say = (line: string): void => {
+  out.push(line)
+  const el = document.getElementById('log')!
+  el.textContent = out.join('\n')
+}
+
+/** A moving picture with no camera in it. */
+function fakeCamera(): MediaStream {
+  const canvas = document.createElement('canvas')
+  canvas.width = 320
+  canvas.height = 180
+  const ctx = canvas.getContext('2d')!
+  let frame = 0
+  setInterval(() => {
+    frame++
+    ctx.fillStyle = `hsl(${frame % 360}, 70%, 50%)`
+    ctx.fillRect(0, 0, canvas.width, canvas.height)
+  }, 40)
+  return (canvas as HTMLCanvasElement & { captureStream(fps: number): MediaStream }).captureStream(15)
+}
+
+/** A microphone with no microphone in it. */
+function fakeMic(): MediaStream {
+  const ctx = new AudioContext()
+  const dest = ctx.createMediaStreamDestination()
+  const osc = ctx.createOscillator()
+  osc.connect(dest)
+  osc.start()
+  return dest.stream
+}
+
+const wait = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms))
+
+async function run(): Promise<void> {
+  const ice: RTCIceServer[] = []
+  let a!: Call
+  let b!: Call
+  let bRemote: MediaStream | null = null
+  let aRemote: MediaStream | null = null
+
+  a = new Call({
+    id: 'test',
+    outgoing: true,
+    iceServers: ice,
+    local: fakeMic(),
+    handlers: {
+      send: (s) => void deliver('a', s),
+      onRemoteStream: (st) => (aRemote = st),
+      onPhase: (p) => say(`  a: ${p}`),
+      onError: (m) => say(`  a error: ${m}`)
+    }
+  })
+  b = new Call({
+    id: 'test',
+    outgoing: false,
+    iceServers: ice,
+    local: fakeMic(),
+    handlers: {
+      send: (s) => void deliver('b', s),
+      onRemoteStream: (st) => (bRemote = st),
+      onPhase: (p) => say(`  b: ${p}`),
+      onError: (m) => say(`  b error: ${m}`)
+    }
+  })
+
+  async function deliver(from: string, signal: { kind: string; content: Record<string, unknown> }): Promise<void> {
+    const to = from === 'a' ? b : a
+    const c = signal.content as Record<string, any>
+    if (signal.kind === 'invite') await to.accept(c.offer.sdp, false)
+    else if (signal.kind === 'answer') await to.takeAnswer(c.answer.sdp)
+    else if (signal.kind === 'candidates') await to.addCandidates(c.candidates)
+    else if (signal.kind === 'negotiate') await to.takeNegotiation(c.description)
+  }
+
+  const videoTracks = (s: MediaStream | null): number =>
+    (s?.getVideoTracks() ?? []).filter((t) => t.readyState === 'live' && !t.muted).length
+
+  say('placing a voice call')
+  await a.invite(false)
+  await wait(1500)
+  say(`connected: a sees ${videoTracks(aRemote)} live video, b sees ${videoTracks(bRemote)}`)
+  const startedVoiceOnly = videoTracks(aRemote) === 0 && videoTracks(bRemote) === 0
+
+  say('a turns its camera on mid-call')
+  const on = await a.setCamera(true, fakeCamera())
+  await wait(2000)
+  const bSeesCamera = videoTracks(bRemote)
+  say(`  a.cameraOn=${on}; b now sees ${bSeesCamera} live video track(s)`)
+
+  say('a turns it off again')
+  const off = await a.setCamera(false)
+  await wait(2000)
+  const bSeesAfter = videoTracks(bRemote)
+  say(`  a.cameraOn=${a.cameraOn} (returned ${off}); b now sees ${bSeesAfter}`)
+
+  say('b turns its camera on, the other direction')
+  await b.setCamera(true, fakeCamera())
+  await wait(2000)
+  const aSeesCamera = videoTracks(aRemote)
+  say(`  a now sees ${aSeesCamera} live video track(s)`)
+
+  const verdict =
+    startedVoiceOnly && bSeesCamera === 1 && bSeesAfter === 0 && aSeesCamera === 1 ? 'PASS' : 'FAIL'
+  say(`\nVERDICT ${verdict}`)
+  document.title = `callcheck ${verdict}`
+  ;(window as unknown as { verdict: string }).verdict = verdict
+
+  await a.hangUp(undefined, false)
+  await b.hangUp(undefined, false)
+}
+
+void run().catch((e) => {
+  say(`threw: ${(e as Error).message}`)
+  document.title = 'callcheck FAIL'
+  ;(window as unknown as { verdict: string }).verdict = 'FAIL'
+})
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Two ends of a real call in one page, wired to each other.
+#
+# The claim this checks is the one that cannot be checked by reading: that a
+# camera turned on in a call already running reaches the other end, and that
+# turning it off takes it away rather than freezing it. That is an offer, an
+# answer, a renegotiation each way and a track arriving on a stream somebody
+# else owns - four things no unit test touches, and the renderer has no test
+# runner of its own (see #212).
+#
+# Real RTCPeerConnections over loopback, with a canvas standing in for the
+# camera and an oscillator for the microphone, so it needs no hardware and no
+# second person. Open the printed URL and read the verdict; the page title
+# carries it too.
+set -euo pipefail
+here="$(cd "$(dirname "$0")" && pwd)"
+root="$(cd "$here/../.." && pwd)"
+out="${TMPDIR:-/tmp}/moho-callcheck"
+mkdir -p "$out"
+
+"$root/node_modules/.bin/esbuild" "$here/loopback.ts" \
+  --bundle --format=iife --outfile="$out/harness.js" --log-level=warning
+
+cat > "$out/index.html" <<'HTML'
+<!doctype html>
+<meta charset="utf-8">
+<title>callcheck</title>
+<body style="background:#111;color:#ddd;font:13px ui-monospace,monospace;padding:12px">
+<pre id="log">starting…</pre>
+<script src="harness.js"></script>
+HTML
+
+echo "serving $out on http://localhost:8791 - ctrl-c when done"
+cd "$out" && exec python3 -m http.server 8791
@@ -216,17 +216,14 @@ export function CallStage({ mode = 'inline' }: { mode?: 'inline' | 'pip' }): JSX
          />
          {/* Between the microphone and the screen, which is the order every
              client puts them in - and the order of how often they are
-              pressed. Offered only where it can work: a one-to-one Matrix
-              call is voice here, and a camera button that cannot turn a
-              camera on is worse than no button. */}
-          {call.onServer && (
-            <IconButton
-              name={call.cameraOn ? 'videocam' : 'videocam_off'}
-              title={call.cameraOn ? 'Turn the camera off' : 'Turn the camera on'}
-              className={call.cameraOn ? 'active' : undefined}
-              onClick={() => void store.toggleCamera()}
-            />
-          )}
+              pressed. On every call now, whatever shape it is: a media
+              server, a mesh, or the one connection between two people. */}
+          <IconButton
+            name={call.cameraOn ? 'videocam' : 'videocam_off'}
+            title={call.cameraOn ? 'Turn the camera off' : 'Turn the camera on'}
+            className={call.cameraOn ? 'active' : undefined}
+            onClick={() => void store.toggleCamera()}
+          />
          <IconButton
            name={call.sharingScreen ? 'stop_screen_share' : 'screen_share'}
            title={call.sharingScreen ? 'Stop sharing' : 'Share a screen or window'}
@@ -250,6 +250,67 @@ export class MatrixCalls {
    return this.group
  }

+  /**
+   * Whether this end is sending a camera, in whatever kind of call it is in.
+   *
+   * One question with one answer, because the button asking it is one button
+   * - the caller has no business knowing whether this call is a mesh, a media
+   * server or a single connection.
+   */
+  get cameraOn(): boolean {
+    if (this.group?.sfu) return this.group.sfu.cameraOn
+    if (this.group) return this.group.peers.values().next().value?.call.cameraOn ?? !!this.group.local?.getVideoTracks().some((t) => t.readyState === 'live')
+    return this.active?.call.cameraOn ?? false
+  }
+
+  /**
+   * Turns the camera on or off, wherever this end is.
+   *
+   * Three shapes of call and one gesture. On a media server the server's own
+   * client does it; on a mesh the camera is opened once here and lent to
+   * every leg, because five connections asking the machine for the same
+   * camera get one capture and four refusals; in a call between two people
+   * the one connection opens its own.
+   *
+   * Returns whether the camera is now on.
+   */
+  async setCamera(on: boolean): Promise<boolean> {
+    if (this.group?.sfu) {
+      return await this.group.sfu.toggleCamera()
+    }
+
+    if (this.group) {
+      const group = this.group
+      if (on) {
+        // Added to the capture the whole call shares, so this end's own tile
+        // shows it too - `local` is what the self tile is pointed at.
+        let track = group.local?.getVideoTracks().find((t) => t.readyState === 'live')
+        if (!track) {
+          const opened = await navigator.mediaDevices.getUserMedia({ video: true })
+          track = opened.getVideoTracks()[0]
+          if (!track) return false
+          if (!group.local) group.local = new MediaStream()
+          group.local.addTrack(track)
+        }
+        const lent = new MediaStream([track])
+        for (const peer of group.peers.values()) await peer.call.setCamera(true, lent)
+        return true
+      }
+      for (const peer of group.peers.values()) await peer.call.setCamera(false)
+      // Stopped once, here, because this is where it was opened - each leg
+      // spares a lent track deliberately.
+      for (const track of group.local?.getVideoTracks() ?? []) {
+        track.stop()
+        group.local?.removeTrack(track)
+      }
+      return false
+    }
+
+    const call = this.active
+    if (!call) return false
+    return await call.call.setCamera(on)
+  }
+
  /**
   * Joins the room's call, starting one if nobody is in it yet.
   *
@@ -69,6 +69,15 @@ export class Call {
  private answered = false
  /** The screen track while one is being shared, so it can be stopped. */
  private screen: MediaStreamTrack | null = null
+  /**
+   * The camera track this leg is sending, while it is sending one.
+   *
+   * Held separately from `local` because a leg can be sending a camera, a
+   * screen, both or neither, and "the video track" is ambiguous the moment
+   * there are two of them - the screen-share code below would otherwise
+   * switch off somebody's face when they stopped sharing.
+   */
+  private camera: MediaStreamTrack | null = null
  /** Media opened by somebody else and lent to this leg - see the
   *  constructor. Never stopped here, because it is not this leg's to stop. */
  private shared: MediaStream | null = null
@@ -114,7 +123,25 @@ export class Call {

    this.pc.ontrack = (e) => {
      for (const track of e.streams[0]?.getTracks() ?? [e.track]) {
-        if (!this.remote.getTracks().some((t) => t.id === track.id)) this.remote.addTrack(track)
+        if (this.remote.getTracks().some((t) => t.id === track.id)) continue
+        this.remote.addTrack(track)
+        // A track the other end takes away - a camera switched off, a screen
+        // share stopped - has to leave this stream, or the tile keeps drawing
+        // the last frame it received for ever. Chromium reports it as a mute
+        // before it reports it as an end, and on a removed track the mute is
+        // often all there is.
+        const drop = (): void => {
+          this.remote.removeTrack(track)
+          this.handlers.onRemoteStream(this.remote)
+        }
+        track.onended = drop
+        track.onmute = drop
+        // And back again when they turn it on: the same transceiver is
+        // reused, so the track unmutes rather than a new one arriving.
+        track.onunmute = () => {
+          if (!this.remote.getTracks().some((t) => t.id === track.id)) this.remote.addTrack(track)
+          this.handlers.onRemoteStream(this.remote)
+        }
      }
      this.handlers.onRemoteStream(this.remote)
    }
@@ -163,6 +190,10 @@ export class Call {
    return !!this.screen
  }

+  get cameraOn(): boolean {
+    return !!this.camera && this.camera.readyState === 'live'
+  }
+
  private setPhase(phase: CallPhase): void {
    if (this.phase === phase || this.phase === 'ended') return
    this.phase = phase
@@ -181,6 +212,7 @@ export class Call {
    // Already open, and open for several legs at once: take it as it is.
    if (this.shared) {
      this.local = this.shared
+      this.camera = this.shared.getVideoTracks()[0] ?? null
      for (const track of this.shared.getTracks()) this.pc.addTrack(track, this.shared)
      return
    }
@@ -194,6 +226,7 @@ export class Call {
      this.handlers.onError('No camera here - calling with audio only')
      this.local = await navigator.mediaDevices.getUserMedia({ audio, video: false })
    }
+    this.camera = this.local.getVideoTracks()[0] ?? null
    for (const track of this.local.getTracks()) this.pc.addTrack(track, this.local)
  }

@@ -268,12 +301,62 @@ export class Call {
    return !track.enabled
  }

-  /** Turns the camera off or on. Returns whether it is now on. */
-  toggleCamera(): boolean {
-    const track = this.local?.getVideoTracks()[0]
+  /**
+   * Turns the camera on or off. Returns whether it is now on.
+   *
+   * Added and removed rather than enabled and disabled. A disabled track
+   * keeps sending - a black rectangle at whatever frame rate the encoder
+   * feels like - so the other end sees a dark square where a face was and
+   * cannot tell that from a camera that has frozen. Removing the track ends
+   * it properly: every other client then draws the person's name or avatar,
+   * which is what "camera off" is supposed to look like.
+   *
+   * The cost is a renegotiation each way, which is the same cost sharing a
+   * screen already pays and is unnoticeable beside the second the camera
+   * takes to open.
+   *
+   * `source` is a capture somebody else opened - the group call opens one
+   * camera and lends it to every leg, because asking the machine five times
+   * for the same camera gets one capture and four refusals. A track lent
+   * this way is never stopped here.
+   */
+  async setCamera(on: boolean, source?: MediaStream | null): Promise<boolean> {
+    if (on === this.cameraOn) return this.cameraOn
+
+    if (!on) {
+      const track = this.camera
+      this.camera = null
+      if (!track) return false
+      const sender = this.pc.getSenders().find((s) => s.track === track)
+      if (sender) this.pc.removeTrack(sender)
+      // Not a lent one: the other legs of this call are still sending it.
+      if (!this.shared) {
+        track.stop()
+        this.local?.removeTrack(track)
+      }
+      await this.renegotiate()
+      return false
+    }
+
+    let stream = source ?? null
+    if (!stream) {
+      // Its own request rather than reopening the microphone with it: the
+      // microphone is already open and already being listened to, and asking
+      // for both again would interrupt the audio to add a picture.
+      stream = await navigator.mediaDevices.getUserMedia({ video: true })
+    }
+    const track = stream.getVideoTracks()[0]
    if (!track) return false
-    track.enabled = !track.enabled
-    return track.enabled
+    this.camera = track
+    // The browser's own "stop" - a camera unplugged, a privacy shutter, the
+    // device taken by something else - has to reach the call, or the far end
+    // keeps a frozen last frame for ever.
+    track.onended = () => void this.setCamera(false)
+    if (!this.local) this.local = new MediaStream()
+    if (!this.shared && !this.local.getTracks().includes(track)) this.local.addTrack(track)
+    this.pc.addTrack(track, this.local)
+    await this.renegotiate()
+    return true
  }

  /**
@@ -304,19 +387,46 @@ export class Call {
    return true
  }

-  private async renegotiate(): Promise<void> {
-    const offer = await this.pc.createOffer()
-    await this.pc.setLocalDescription(offer)
-    this.handlers.send({ kind: 'negotiate', content: { description: { type: 'offer', sdp: offer.sdp } } })
+  /**
+   * One renegotiation at a time.
+   *
+   * Turning a camera on and sharing a screen a second later are two changes
+   * to the same connection, and starting the second offer while the first is
+   * still unanswered puts the peer connection in a state neither end can
+   * recover from - the browser throws and the call is left half-described.
+   * Chained rather than dropped, because both changes were asked for and both
+   * have to happen.
+   */
+  private pendingNegotiation: Promise<void> = Promise.resolve()
+
+  private renegotiate(): Promise<void> {
+    this.pendingNegotiation = this.pendingNegotiation.then(async () => {
+      if (this.phase === 'ended') return
+      const offer = await this.pc.createOffer()
+      await this.pc.setLocalDescription(offer)
+      this.handlers.send({ kind: 'negotiate', content: { description: { type: 'offer', sdp: offer.sdp } } })
+    })
+    return this.pendingNegotiation
  }

-  /** A renegotiation from the other end - somebody starting to share. */
-  async takeNegotiation(description: RTCSessionDescriptionInit): Promise<void> {
-    await this.pc.setRemoteDescription(description)
-    if (description.type !== 'offer') return
-    const answer = await this.pc.createAnswer()
-    await this.pc.setLocalDescription(answer)
-    this.handlers.send({ kind: 'negotiate', content: { description: { type: 'answer', sdp: answer.sdp } } })
+  /**
+   * A renegotiation from the other end - somebody turning a camera on, or
+   * starting to share.
+   *
+   * Queued behind this end's own, for the reason above: two descriptions
+   * being applied to one connection at once is how a call ends up describing
+   * something neither end is sending.
+   */
+  takeNegotiation(description: RTCSessionDescriptionInit): Promise<void> {
+    this.pendingNegotiation = this.pendingNegotiation.then(async () => {
+      if (this.phase === 'ended') return
+      await this.pc.setRemoteDescription(description)
+      if (description.type !== 'offer') return
+      const answer = await this.pc.createAnswer()
+      await this.pc.setLocalDescription(answer)
+      this.handlers.send({ kind: 'negotiate', content: { description: { type: 'answer', sdp: answer.sdp } } })
+    })
+    return this.pendingNegotiation
  }

  /** Ends the call, telling the other end unless it was them who ended it. */
@@ -330,6 +440,10 @@ export class Call {
    }
    this.screen?.stop()
    this.screen = null
+    // Not the lent one, for the same reason the shared capture above is
+    // spared: it belongs to the call, not to this leg of it.
+    if (!this.shared) this.camera?.stop()
+    this.camera = null
    this.pc.close()
    this.setPhase('ended')
  }
@@ -908,9 +908,10 @@ export class ChatStore {
            video: group.video,
            muted: this.state.activeCall?.muted ?? false,
            sharingScreen: false,
-            // A call joined with video starts sending one; otherwise the
-            // camera is off until somebody turns it on.
-            cameraOn: group.video,
+            // Asked of the call, for the same reason the one-to-one branch
+            // below does: this runs on every phase change, and a fixed
+            // answer would undo a camera somebody had just turned on.
+            cameraOn: this.matrixCalls.cameraOn,
            onServer: true
          }
        })
@@ -929,7 +930,12 @@ export class ChatStore {
          video: call.video,
          muted: this.state.activeCall?.muted ?? false,
          sharingScreen: call.call.sharingScreen,
-          cameraOn: false,
+          // Asked of the call rather than assumed. A call placed with video
+          // is already sending a camera, one placed without is not, and one
+          // where somebody has pressed the button since is neither - this
+          // block runs on every phase change and would otherwise put the
+          // button back to "off" the moment the call connected.
+          cameraOn: call.call.cameraOn,
          onServer: false
        }
      })
@@ -2275,19 +2281,20 @@ export class ChatStore {
  /**
   * Turns the camera on, or off, in a call already running.
   *
-   * Only on a media server. A one-to-one Matrix call is voice here - see #189 -
-   * and offering a camera button that cannot do anything would be worse than
-   * not offering one.
+   * One gesture for all three shapes of call - a media server, a mesh between
+   * several people, a single connection between two - because the button
+   * asking for it is one button and the person pressing it is asking one
+   * thing. Which of the three this is belongs to the call, not to the
+   * control.
   */
  async toggleCamera(): Promise<void> {
    if (!this.state.activeCall) return
-    const group = this.matrixCalls.currentGroup
-    if (!group?.sfu) {
-      this.toast('info', 'A camera needs a call held on a media server.')
-      return
-    }
-    const on = await group.sfu.toggleCamera().catch((e: Error) => {
-      this.toast('error', e.message)
+    const want = !this.matrixCalls.cameraOn
+    const on = await this.matrixCalls.setCamera(want).catch((e: Error) => {
+      // The usual failure is a machine with no camera, or one already taken
+      // by something else. Both are worth a sentence: a button that appears
+      // to do nothing is the worse outcome.
+      this.toast('error', `Couldn't turn the camera on: ${e.message}`)
      return false
    })
    this.set({ activeCall: { ...this.state.activeCall, cameraOn: on } })