Matrix: join Element's calls, which run through a media server #132

Closed
opened 2026-09-06 14:23:07 -04:00 by Salastil · 2 comments
Owner

moho now publishes and reads call membership in Element's own shape (org.matrix.msc3401.call.member, per-device state keys, created_ts/expires) — so both clients can see the same call and who is in it. What they still cannot do is hear each other: Element's media goes through a LiveKit SFU and moho's goes peer to peer.

Joining an Element call means becoming a LiveKit client. The pieces, in order:

1. Find the focus. .well-known/matrix/client → org.matrix.msc4143.rtc_foci: [{type: "livekit", livekit_service_url}], or the authenticated /_matrix/client/unstable/org.matrix.msc4143/rtc/transports. A call in progress also names one: focus_active / foci_preferred in the oldest membership, which is what focus_selection: "oldest_membership" means.

2. Get a token. An OpenID token from the homeserver (POST /_matrix/client/v3/user/{userId}/openid/request_token), handed to the JWT service beside the SFU. Two generations, both in element-hq/lk-jwt-service:

  • legacy /sfu/get — {room, openid_token, device_id} → {url, jwt}
  • current /get_token — {room_id, slot_id, openid_token, member: {id, claimed_user_id, claimed_device_id}} → {url, jwt}

3. Speak LiveKit. livekit-client in the renderer: its websocket signalling, its tracks, its participant model. This is the large piece and a real dependency.

4. Encrypt like Element does. Element Call E2EEs the media per sender and shares the keys over Matrix; a client that joins and sends plaintext is a client nobody can hear. RTCEncryptionManager / ToDeviceKeyTransport in matrix-js-sdk is the reference.

5. Publish a livekit transport in our own membership once we can actually serve one, instead of the moho.mesh marker we publish today.

Worth knowing before starting: neither homeserver on this machine advertises a focus — poa.st serves no JSON well-known at all, matrix.salastil.com has no rtc_foci — while matrix.org advertises https://livekit-jwt.call.matrix.org. So testing needs either an account on a server that has one, or lk-jwt-service + LiveKit deployed on matrix.salastil.com (which would also give that server working Element calls).

Keep the mesh. It is what works with no infrastructure at all, which is most self-hosted homeservers, and MSC4143's whole shape is pluggable transports. The SFU path is a second transport, not a replacement.

moho now publishes and reads call membership in Element's own shape (`org.matrix.msc3401.call.member`, per-device state keys, `created_ts`/`expires`) — so both clients can see the same call and who is in it. What they still cannot do is *hear* each other: Element's media goes through a LiveKit SFU and moho's goes peer to peer. Joining an Element call means becoming a LiveKit client. The pieces, in order: **1. Find the focus.** `.well-known/matrix/client` → `org.matrix.msc4143.rtc_foci: [{type: "livekit", livekit_service_url}]`, or the authenticated `/_matrix/client/unstable/org.matrix.msc4143/rtc/transports`. A call in progress also names one: `focus_active` / `foci_preferred` in the oldest membership, which is what `focus_selection: "oldest_membership"` means. **2. Get a token.** An OpenID token from the homeserver (`POST /_matrix/client/v3/user/{userId}/openid/request_token`), handed to the JWT service beside the SFU. Two generations, both in `element-hq/lk-jwt-service`: - legacy `/sfu/get` — `{room, openid_token, device_id}` → `{url, jwt}` - current `/get_token` — `{room_id, slot_id, openid_token, member: {id, claimed_user_id, claimed_device_id}}` → `{url, jwt}` **3. Speak LiveKit.** `livekit-client` in the renderer: its websocket signalling, its tracks, its participant model. This is the large piece and a real dependency. **4. Encrypt like Element does.** Element Call E2EEs the media per sender and shares the keys over Matrix; a client that joins and sends plaintext is a client nobody can hear. `RTCEncryptionManager` / `ToDeviceKeyTransport` in matrix-js-sdk is the reference. **5. Publish a livekit transport** in our own membership once we can actually serve one, instead of the `moho.mesh` marker we publish today. **Worth knowing before starting:** neither homeserver on this machine advertises a focus — `poa.st` serves no JSON well-known at all, `matrix.salastil.com` has no `rtc_foci` — while `matrix.org` advertises `https://livekit-jwt.call.matrix.org`. So testing needs either an account on a server that has one, or `lk-jwt-service` + LiveKit deployed on `matrix.salastil.com` (which would also give that server working Element calls). **Keep the mesh.** It is what works with no infrastructure at all, which is most self-hosted homeservers, and MSC4143's whole shape is pluggable transports. The SFU path is a second transport, not a replacement.
Salastil added the feature label 2026-09-06 14:23:07 -04:00
Author
Owner

Finished — https://git.salastil.com/Salastil/nobilis/commit/2d96e89 and https://git.salastil.com/Salastil/moho/commit/4e9f6dd

moho is a LiveKit client now, so a Matrix room call goes through a media server where the call has one and stays a mesh where it does not.

Finding the server, in the order that answers best: the call itself (everybody has to be where the media already is — this is what makes joining somebody else's call possible), then this account's own rtc_focus_url for a homeserver that publishes nothing, then the homeserver's .well-known.

Getting in is an OpenID token rather than an access token — the service beside the SFU has no way to check a Matrix access token, so the homeserver vouches for the account in a form the service can verify by asking back. Traded at /sfu/get for a URL and a JWT good for one room, before joining, so a refusal fails the join instead of leaving the account listed in a call it never reached.

The membership then says livekit, which is what makes it legible to Element.

Verified against the real Element stack — livekit/livekit-server and element-hq/lk-jwt-service, both unmodified — with the two Matrix accounts on their two different homeservers:

A joins   → lk-jwt-service verified the OpenID token from poa.st, created the
            LiveKit room, issued a token; A connected, membership published as livekit
B joins   → with its own focus setting cleared, so the only way it could find the
            server was reading A's membership — which is exactly how joining an
            Element call works
both ends → "Connected", 2 tiles, live audio through the SFU

Two bugs found by testing, both fixed. The renderer's CSP refused the websocket (signalling is neither http nor https) — wss: is allowed now, plaintext only to this machine. And the stage was going up after the join, which overwrote the "connected" the call had already reported.

Not done, and it matters for encrypted rooms: Element encrypts a call's media per sender and shares the keys over Matrix. moho joins in the clear, so in an encrypted room everybody would be connected and nobody could hear anybody — the client says so plainly when joining such a call rather than leaving people to wonder. Filed separately.

Worth repeating: neither homeserver here advertises a focus, so real Element interop on your own server still needs lk-jwt-service + LiveKit beside matrix.salastil.com. Until then the per-account setting is the way to point moho at one.

Finished — https://git.salastil.com/Salastil/nobilis/commit/2d96e89 and https://git.salastil.com/Salastil/moho/commit/4e9f6dd moho is a LiveKit client now, so a Matrix room call goes through a media server where the call has one and stays a mesh where it does not. **Finding the server**, in the order that answers best: the call itself (everybody has to be where the media already is — this is what makes joining somebody else's call possible), then this account's own `rtc_focus_url` for a homeserver that publishes nothing, then the homeserver's `.well-known`. **Getting in** is an OpenID token rather than an access token — the service beside the SFU has no way to check a Matrix access token, so the homeserver vouches for the account in a form the service can verify by asking back. Traded at `/sfu/get` for a URL and a JWT good for one room, before joining, so a refusal fails the join instead of leaving the account listed in a call it never reached. **The membership then says `livekit`**, which is what makes it legible to Element. **Verified against the real Element stack** — `livekit/livekit-server` and `element-hq/lk-jwt-service`, both unmodified — with the two Matrix accounts on their two different homeservers: ``` A joins → lk-jwt-service verified the OpenID token from poa.st, created the LiveKit room, issued a token; A connected, membership published as livekit B joins → with its own focus setting cleared, so the only way it could find the server was reading A's membership — which is exactly how joining an Element call works both ends → "Connected", 2 tiles, live audio through the SFU ``` **Two bugs found by testing, both fixed.** The renderer's CSP refused the websocket (signalling is neither http nor https) — `wss:` is allowed now, plaintext only to this machine. And the stage was going up *after* the join, which overwrote the "connected" the call had already reported. **Not done, and it matters for encrypted rooms:** Element encrypts a call's media per sender and shares the keys over Matrix. moho joins in the clear, so in an encrypted room everybody would be connected and nobody could hear anybody — the client says so plainly when joining such a call rather than leaving people to wonder. Filed separately. **Worth repeating:** neither homeserver here advertises a focus, so real Element interop on your own server still needs `lk-jwt-service` + LiveKit beside `matrix.salastil.com`. Until then the per-account setting is the way to point moho at one.
Author
Owner

Tested against Element Desktop 1.12.24 today, with LiveKit + lk-jwt-service local. Three findings, one of them a real bug now fixed — https://git.salastil.com/Salastil/nobilis/commit/fe53f24

1. moho was landing in a different room on the same media server. The LiveKit room is derived, not named: the old /sfu/get derives it from the Matrix room alone, the newer /get_token from sha256(json([room_id, slot_id])) with the slot spelled m.call#ROOM. Element uses the newer one. Same call, same SFU, two rooms, one person in each.

Proven, not reasoned: Element's own attempt asked the JWT service for /ZnnvgF/vlXf9arKRA5Ua2+nPtbTQn5JZV5DhF//DiE, and moho now asks for exactly that string. The old endpoint stays as a fallback for services that only have it.

2. Element does not adopt the call's focus — it requires the joiner's own homeserver to advertise one. Joining moho's call in a room failed with MISSING_MATRIX_RTC_TRANSPORT (Domain: matrix.salastil.com) even though moho's membership named a reachable service in foci_preferred. So focus_selection: oldest_membership decides which of the transports a client already knows to use; it does not teach it a new one.

That makes interop a server-side prerequisite: the Element user's homeserver needs org.matrix.msc4143.rtc_foci in /.well-known/matrix/client. Nothing about the homeserver software is involved — matrix.salastil.com's well-known is served by openresty, so it is a static JSON edit plus a reachable LiveKit + lk-jwt-service.

3. Element gates the call button on widget permission, not on m.call.member. A power-0 member sees "You do not have permission to start video calls"; at moderator the button becomes "Join video call". So moho lowering only org.matrix.msc3401.call.member opens the call to MatrixRTC clients but not to Element users, who also need im.vector.modular.widgets. Worth deciding deliberately — lowering that lets any member add widgets, which is a wider grant than "may join calls".

Also fixed here: power levels are read from the server before being written back. A power-levels event is a read-modify-write of everything a room permits, and writing it from a possibly-partial cache is how a room silently loses a key.

Tested against Element Desktop 1.12.24 today, with LiveKit + lk-jwt-service local. Three findings, one of them a real bug now fixed — https://git.salastil.com/Salastil/nobilis/commit/fe53f24 **1. moho was landing in a different room on the same media server.** The LiveKit room is derived, not named: the old `/sfu/get` derives it from the Matrix room alone, the newer `/get_token` from `sha256(json([room_id, slot_id]))` with the slot spelled `m.call#ROOM`. Element uses the newer one. Same call, same SFU, two rooms, one person in each. Proven, not reasoned: Element's own attempt asked the JWT service for `/ZnnvgF/vlXf9arKRA5Ua2+nPtbTQn5JZV5DhF//DiE`, and moho now asks for exactly that string. The old endpoint stays as a fallback for services that only have it. **2. Element does not adopt the call's focus — it requires the joiner's own homeserver to advertise one.** Joining moho's call in a room failed with `MISSING_MATRIX_RTC_TRANSPORT (Domain: matrix.salastil.com)` even though moho's membership named a reachable service in `foci_preferred`. So `focus_selection: oldest_membership` decides *which* of the transports a client already knows to use; it does not teach it a new one. That makes interop a server-side prerequisite: the Element user's homeserver needs `org.matrix.msc4143.rtc_foci` in `/.well-known/matrix/client`. Nothing about the homeserver *software* is involved — matrix.salastil.com's well-known is served by openresty, so it is a static JSON edit plus a reachable LiveKit + lk-jwt-service. **3. Element gates the call button on widget permission**, not on `m.call.member`. A power-0 member sees "You do not have permission to start video calls"; at moderator the button becomes "Join video call". So moho lowering only `org.matrix.msc3401.call.member` opens the call to MatrixRTC clients but not to Element users, who also need `im.vector.modular.widgets`. Worth deciding deliberately — lowering that lets any member add widgets, which is a wider grant than "may join calls". **Also fixed here:** power levels are read from the server before being written back. A power-levels event is a read-modify-write of everything a room permits, and writing it from a possibly-partial cache is how a room silently loses a key.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Salastil/moho#132