Salastil/nobilis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Carry a call's signalling; leave its media to the client

Browse Source 
Matrix had no calls of any kind. The media cannot live here and should not:
a call is WebRTC - codec negotiation, echo cancellation, jitter buffers,
hardware decode - and the client this daemon serves is Chromium, which has
all of it, while this process would have to grow a browser to keep up. So
what is here is the signalling, which genuinely is the daemon's: an offer,
an answer, the network paths and a hangup are events in a room, and rooms
are what this backend reads and writes.

That is the same line Discord's voice already draws from the other side,
where the daemon holds the audio because the protocol is one no browser
speaks.

Also adds the typed send this needed and the backend did not have.
send_room_event always sends m.room.message - it exists for the things that
are messages without being chat - so a call event routed through it went out
as a message with the type in the room-id slot. A call event has to carry
its own type all the way through, including into the encryption, where the
type is part of what is encrypted.

Verified live between two homeservers: a call placed from
matrix.salastil.com rang on poa.st, was answered, and both ends reported
connected with live audio - then a shared screen arrived as a second track
through m.call.negotiate.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Salastil
2026-09-05 22:10:53 -04:00
co-authored by Claude Opus 5
parent 72a881e5dd
commit c9d06bcb44
3 changed files with 210 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/backend/matrix/calls.rs
+113
View File
@@ -0,0 +1,113 @@
//! Voice and video calls, as far as a daemon can carry them.
//!
//! The media is not here and cannot be. A call is WebRTC: two peers agreeing
//! on codecs and network paths and then sending each other encrypted RTP,
//! with echo cancellation, jitter buffers and hardware decode in the middle.
//! The client this daemon serves is Chromium, which has all of that; this
//! process has none of it, and building it in Rust to keep the media on this
//! side would be building a browser badly.
//!
//! So what lives here is the signalling, which genuinely is the daemon's: the
//! offer, the answer, the network candidates and the hangup are events in a
//! Matrix room, and rooms are what this backend already knows how to read and
//! write - encrypted ones included, which matters because a call in an
//! encrypted room signals through the same ciphertext everything else does.
//!
//! The division is the same one Discord's voice already draws here, from the
//! other side: there the daemon holds the audio because the connection is a
//! bespoke protocol no browser speaks. Where the protocol *is* the browser's,
//! the browser should hold it.
//!
//! One call at a time per account, which is what the events themselves assume:
//! `m.call.invite` names a call id, and every later event carries it.
use serde_json::{json, Value};
use crate::state::AppState;
/// Events a client has to see to be one end of a call.
///
/// Deliberately everything under `m.call.` rather than a list: the ones this
/// client does not act on are still worth passing through, because a client
/// that learns a new one later needs no change here, and because an event
/// that is silently dropped is the hardest kind to find missing.
pub fn handle(
state: &AppState,
account_id: &str,
buffer_id: &str,
own_user_id: &str,
event_type: &str,
sender: &str,
content: &Value,
) {
// Our own signalling, echoed back by the server the way every event is.
// Not silence, though: a call answered on another device has to end the
// ringing here, and the only way this client learns that is by seeing its
// own account answer somewhere else.
let mine = sender == own_user_id;
let call_id = content["call_id"].as_str().unwrap_or_default().to_string();
// Which of this account's devices sent it. A call answered on a phone is
// answered by the same user, and without this the desktop cannot tell
// "somebody else took it" from its own echo.
let device = content["party_id"].as_str().or_else(|| content["device_id"].as_str());
state.events.emit(
"matrixCall",
json!({
"accountId": account_id,
"bufferId": buffer_id,
"kind": event_type,
"callId": call_id,
"from": sender,
"own": mine,
"partyId": device,
// The whole content, because what a client needs from it differs
// per event - an offer carries SDP, candidates carry a list, a
// hangup carries a reason - and re-modelling each here would be
// a second copy of the specification to keep in step.
"content": content,
}),
);
}
/// Sends one call event into a room.
///
/// Every one of them is an ordinary room event, so this is a thin wrapper over
/// the send path the rest of the backend uses - including the encryption,
/// which a call in an encrypted room needs as much as a message does.
pub async fn send(
state: &AppState,
account_id: &str,
buffer_id: &str,
event_type: &str,
content: Value,
) -> anyhow::Result<()> {
// The typed send, not the message-shaped one beside it: a call event sent
// as a message arrives as a blank line rather than as a telephone.
super::send_typed_event(state, account_id, buffer_id, event_type, content).await
}
/// Where to reach this homeserver's TURN servers, and with what.
///
/// Asked of the server rather than configured, because that is where the
/// answer is: a homeserver runs its own relay and hands out short-lived
/// credentials for it. Without one, a call only connects between two people
/// whose networks happen to allow it - which is most of the time on a LAN and
/// almost never between two homes.
pub async fn turn_servers(state: &AppState, account_id: &str) -> anyhow::Result<Value> {
let account = state
.accounts
.get_matrix(account_id)
.ok_or_else(|| anyhow::anyhow!("account not connected"))?;
let base = account.homeserver_url.trim_end_matches('/');
match super::http::get_json(&format!("{base}/_matrix/client/v3/voip/turnServer"), &account.access_token).await {
Ok(answer) => Ok(answer),
// A server with no relay configured answers 404, and that is not an
// error: it means "there is no relay", which a client can still make
// a call without.
Err(e) => {
tracing::debug!("matrix[{account_id}]: no TURN server offered: {e:#}");
Ok(json!({ "uris": [] }))
}
}
}
src/backend/matrix/mod.rs
+59
View File
@@ -28,6 +28,7 @@ pub mod http;
pub mod markup;
pub mod moderation;
pub mod polls;
pub mod calls;
pub mod protocol;
pub mod roomstate;
pub mod rooms;
@@ -706,6 +707,9 @@ async fn handle_timeline_event(
&& !polls::is_poll_event(outer_type)
// And a verification with another person travels through the room.
&& !outer_type.starts_with("m.key.verification.")
// A call is signalled through the room like anything else: the offer,
// the answer and the network candidates are all events in it.
&& !outer_type.starts_with("m.call.")
{
// Membership/name changes were already folded into naming in
// process_sync_response; anything else (typing, receipts, other
@@ -794,6 +798,15 @@ async fn handle_timeline_event(
return;
}
// A call: somebody ringing, answering, hanging up, or telling us how to
// reach them. Handed to the client rather than rendered, because the
// media is the client's - it has a WebRTC stack and this daemon does not.
// See calls::handle.
if effective_type.starts_with("m.call.") {
calls::handle(state, account_id, buffer_id, own_user_id, &effective_type, sender, &content);
return;
}
// A poll: the question, a vote, or the end of the counting. Handled after
// decryption like everything else, because a poll in an encrypted room
// arrives as ciphertext the same way a message does.
@@ -3193,6 +3206,52 @@ pub async fn edit_message(state: &AppState, account_id: &str, buffer_id: &str, a
Ok(())
}
/// Sends an event of whatever type into a room.
///
/// Unlike `send_room_event` below - which is for the things that are messages
/// without being chat, and always sends `m.room.message` - this carries its
/// own type all the way through, including into the encryption, where the
/// type is part of what gets encrypted. That distinction is not academic: a
/// call event sent as a message would arrive as a blank line in the log
/// rather than as a ringing telephone.
///
/// Encrypted where the room is, for the reason everything else here is: a
/// room that hides what is said in it should not make an exception.
pub async fn send_typed_event(
state: &AppState,
account_id: &str,
buffer_id: &str,
event_type: &str,
content: Value,
) -> Result<()> {
let room_id = state.runtime.get_matrix_room(buffer_id).context("no known room id for this buffer")?;
let account = state.accounts.get_matrix(account_id).context("account not connected")?;
let base = account.homeserver_url.trim_end_matches('/');
let access_token = account.access_token.clone();
let (sent_type, body_json) = if state.runtime.is_matrix_room_encrypted(buffer_id) {
let session = state.runtime.get_matrix_machine(account_id).context("crypto session not ready yet")?;
let member_ids = joined_member_ids(base, &access_token, &room_id).await?;
let room_id_ruma = ruma_common::RoomId::parse(&room_id).context("invalid room id")?;
let encrypted = session
.share_and_encrypt_content(&account.homeserver_url, &access_token, &room_id_ruma, member_ids, event_type, content)
.await
.context("encrypting the event")?;
(protocol::EVENT_ROOM_ENCRYPTED.to_string(), encrypted)
} else {
(event_type.to_string(), content)
};
let txn_id = model::next_message_id();
let url = format!(
"{base}/_matrix/client/v3/rooms/{}/send/{sent_type}/{}",
url::form_urlencoded::byte_serialize(room_id.as_bytes()).collect::<String>(),
url::form_urlencoded::byte_serialize(txn_id.as_bytes()).collect::<String>(),
);
http::put_json(&url, &access_token, body_json).await.context("sending the event")?;
Ok(())
}
/// Sends one message-shaped event into a room and says what id it got.
///
/// Shared by the things that are messages without being chat - so far the
src/rpc/methods.rs
+38
View File
@@ -1253,6 +1253,44 @@ pub async fn dispatch(
}
}
// One end of a Matrix call, signalled through the room.
//
// The media never comes near this daemon: the client has a WebRTC
// stack and this process does not, so what crosses here is the offer,
// the answer, the network candidates and the hangup - see
// backend/matrix/calls.rs.
"sendMatrixCallEvent" => {
let (buffer_id, event_type) = match (p_str_opt(params, "bufferId"), p_str_opt(params, "type")) {
(Some(b), Some(t)) => (b, t),
_ => return (None, Some("sendMatrixCallEvent requires \"bufferId\" and \"type\"".to_string())),
};
if !event_type.starts_with("m.call.") {
return (None, Some("that is not a call event".to_string()));
}
let Some(buffer) = state.runtime.get_buffer(buffer_id) else {
return (None, Some("no such conversation".to_string()));
};
let Some(content) = params.get("content").filter(|c| c.is_object()).cloned() else {
return (None, Some("sendMatrixCallEvent requires \"content\"".to_string()));
};
match backend::matrix::calls::send(state, &buffer.account_id, buffer_id, event_type, content).await {
Ok(()) => (Some(ok_node()), None),
Err(e) => (None, Some(format!("{e:#}"))),
}
}
// Where to reach this homeserver's relay, for a call between two
// networks that will not talk to each other directly.
"matrixTurnServers" => {
let Some(account_id) = p_str_opt(params, "accountId") else {
return (None, Some("matrixTurnServers requires \"accountId\"".to_string()));
};
match backend::matrix::calls::turn_servers(state, account_id).await {
Ok(answer) => (Some(answer), None),
Err(e) => (None, Some(format!("{e:#}"))),
}
}
// The threads a Discord channel or forum is holding.
//
// Asked for when it is opened rather than at connect: a guild can be