Salastil/nobilis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Recording somebody, and sending it as a voice message

Browse Source 
The receiving half landed already; this is the half that could not be done
without writing an Ogg container, because a call puts Opus packets straight
on the wire and nothing here had ever needed one.

The pieces were nearly all present: cpal captures the microphone for calls,
and libopus is linked for them too - opus2 was already in the tree as
songbird's own dependency, so asking for it by name costs one line of lockfile
and no new C. What is new is `oggopus`, which is the framing and three rules
from RFC 7845, and `voicenote`, which holds the one recording a machine with
one microphone can have.

The container is hand-written rather than taken from a crate, which is only
defensible because the result is checkable by something nobody here wrote:
ffprobe reads the output as ogg/opus, mono, 48kHz, 2.000000s, and says nothing
on stderr. The checksum has known answers computed by a second implementation
in Python - a self-consistent checksum test proves only that the code agrees
with itself, and Ogg's CRC is not the common one, so a wrong one fails as "not
an Ogg file" rather than as a checksum problem. My first expected value was
wrong and the Rust was right.

The waveform takes the peak of each bucket rather than the mean: a mean over a
fifth of a second of speech is nearly silence, which is what makes some
clients' waveforms look flat.

Three RPCs rather than one, because it is three moments - somebody starts
talking, then decides whether what they said is worth sending. Progress is
polled rather than pushed; it is only wanted while a timer is on screen.

Verified against real Discord, with permission, by sending one to a DM
between two accounts the same person owns. It came back stored with flags=8192
- so Discord kept it as a voice message and not as an .ogg somebody attached -
with duration_secs and waveform intact, and content_type audio/ogg, which is
Discord's own check agreeing about the container. Then read back through
`extract_attachments`, the parser the window actually draws from: voice=true,
duration 2.0, 344 bytes of waveform.

Both probes are #[ignore]d and take the token and channel from the
environment, like the Sneedchat ones, rather than from anywhere in this
repository.

Refs #186.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Salastil
2026-09-13 00:44:31 -04:00
co-authored by Claude Opus 5
parent 54ac74e4fd
commit f9b5be5f1d
7 changed files with 776 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+1
View File
@@ -3789,6 +3789,7 @@ dependencies = [
"matrix-sdk-common",
"matrix-sdk-crypto",
"matrix-sdk-sqlite",
"opus2",
"qrcode",
"rand 0.8.7",
"reqwest",
Cargo.toml
+4
View File
@@ -94,6 +94,10 @@ js_int = "=0.2.2"
songbird = { version = "0.6", default-features = false, features = ["driver", "rustls", "tungstenite", "receive", "gateway"] }
cpal = "0.18.2"
symphonia = { version = "0.5", default-features = false, features = ["pcm"] }
# The Opus encoder, for recording a voice message. Already in the tree as
# songbird's own dependency and already built against the same libopus - this
# only asks for it by name so it can be used directly.
opus2 = "0.4"
async-trait = "0.1.92"
# Two calls, both in main.rs: see keep_the_heap_honest() there for why a chat
src/backend/discord/send.rs
+200
View File
@@ -301,6 +301,206 @@ pub async fn send_attachment(state: &AppState, buffer_id: &str, token: &str, bod
Ok(())
}
/// Sending a recording as a voice message rather than as a sound file.
///
/// Three things make Discord treat it as one, and all three are required: the
/// IS_VOICE_MESSAGE flag on the message, and `duration_secs` and `waveform` on
/// the attachment. Without them the same bytes arrive as an .ogg somebody
/// attached - which is what every other client would then show, because that
/// is what it would be.
///
/// The attachment metadata travels in `payload_json` alongside the file, keyed
/// by the index of the file part. That is the same single multipart POST the
/// ordinary attachment path uses; the separate upload-then-attach flow exists
/// for files too large for one request, which a voice message is not.
///
/// A voice message carries no text: Discord rejects one with content, and
/// there is nowhere in its own client to type any.
pub async fn send_voice_message(
state: &AppState,
buffer_id: &str,
token: &str,
file_path: &std::path::Path,
duration_secs: f64,
waveform: &str,
) -> Result<()> {
const IS_VOICE_MESSAGE: u64 = 1 << 13;
let channel_id = state
.runtime
.get_discord_channel(buffer_id)
.ok_or_else(|| anyhow!("no known Discord channel for this buffer"))?;
let bytes = tokio::fs::read(file_path)
.await
.with_context(|| format!("reading {}", file_path.display()))?;
// The name Discord's own client uses. Its clients key on the flag rather
// than on this, but a file that arrives anywhere else should say what it
// is - and an unnamed part is rejected outright.
let file_name = "voice-message.ogg";
let payload = json!({
"content": "",
"flags": IS_VOICE_MESSAGE,
"attachments": [{
"id": "0",
"filename": file_name,
"duration_secs": duration_secs,
"waveform": waveform,
}],
});
let form = reqwest::multipart::Form::new()
.text("payload_json", payload.to_string())
.part(
"files[0]",
reqwest::multipart::Part::bytes(bytes)
.file_name(file_name)
.mime_str("audio/ogg")
.context("audio/ogg is a valid mime type")?,
);
let resp = send_write(
http_client()
.post(format!("{API_BASE}/channels/{channel_id}/messages"))
.header("Authorization", token)
.multipart(form),
)
.await
.context("uploading the voice message")?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
bail!("Discord API error {status}: {text}");
}
Ok(())
}
#[cfg(test)]
mod live_probe {
//! Sending a real voice message to a real conversation, run by hand.
//!
//! The encoder can be checked offline and is - ffprobe agrees the file is
//! Ogg/Opus of the right length. What cannot be checked offline is whether
//! Discord accepts it *as a voice message* rather than as a sound file
//! somebody attached, because that answer only exists on their side.
//!
//! `#[ignore]`d, and it takes the token and the channel from the
//! environment rather than from anywhere in this repository:
//! MOHO_DISCORD_TOKEN=… MOHO_DISCORD_CHANNEL=… \
//! cargo test --release -- --ignored --nocapture voice_message_probe
#[tokio::test]
#[ignore]
async fn voice_message_probe() {
let _ = rustls::crypto::ring::default_provider().install_default();
let token = std::env::var("MOHO_DISCORD_TOKEN").expect("MOHO_DISCORD_TOKEN");
let channel = std::env::var("MOHO_DISCORD_CHANNEL").expect("MOHO_DISCORD_CHANNEL");
// Two seconds of a tone that rises and falls, so the waveform that
// arrives is one a person can recognise as this recording rather than
// as a flat bar.
let samples: Vec<f32> = (0..crate::oggopus::RATE * 2)
.map(|i| {
let t = i as f32 / crate::oggopus::RATE as f32;
(t * 440.0 * std::f32::consts::TAU).sin() * (t * std::f32::consts::PI / 2.0).sin() * 0.4
})
.collect();
let duration = crate::oggopus::duration_secs(&samples);
let waveform = {
use base64::Engine;
base64::engine::general_purpose::STANDARD.encode(crate::oggopus::waveform(&samples, 256))
};
let bytes = crate::oggopus::encode(&samples).expect("encoding");
println!("{} bytes of ogg, {duration:.2}s", bytes.len());
const IS_VOICE_MESSAGE: u64 = 1 << 13;
let payload = serde_json::json!({
"content": "",
"flags": IS_VOICE_MESSAGE,
"attachments": [{ "id": "0", "filename": "voice-message.ogg", "duration_secs": duration, "waveform": waveform }],
});
let form = reqwest::multipart::Form::new()
.text("payload_json", payload.to_string())
.part(
"files[0]",
reqwest::multipart::Part::bytes(bytes)
.file_name("voice-message.ogg")
.mime_str("audio/ogg")
.unwrap(),
);
let resp = super::http_client()
.post(format!("{}/channels/{channel}/messages", super::API_BASE))
.header("Authorization", &token)
.multipart(form)
.send()
.await
.expect("posting");
let status = resp.status();
let body: serde_json::Value = resp.json().await.expect("a JSON answer");
println!("HTTP {status}");
assert!(status.is_success(), "Discord refused it: {body}");
// The answer is the message Discord stored, so this is its own
// verification: the flag it kept, and what it did with the metadata.
println!("flags={} attachments={}", body["flags"], serde_json::to_string_pretty(&body["attachments"]).unwrap());
assert_eq!(
body["flags"].as_u64().unwrap_or(0) & IS_VOICE_MESSAGE,
IS_VOICE_MESSAGE,
"stored without the voice-message flag, so it is a sound file: {body}"
);
let att = &body["attachments"][0];
assert!(att["duration_secs"].as_f64().is_some(), "no duration came back: {att}");
assert!(att["waveform"].as_str().is_some(), "no waveform came back: {att}");
}
}
#[cfg(test)]
mod receive_probe {
//! The other half of the round trip: reading back what Discord stored and
//! running it through the parser the window draws from.
//!
//! Read-only, and separate from the send probe so it can be pointed at any
//! conversation that already has a voice message in it:
//! MOHO_DISCORD_TOKEN=… MOHO_DISCORD_CHANNEL=… \
//! cargo test --release -- --ignored --nocapture voice_message_read_probe
#[tokio::test]
#[ignore]
async fn voice_message_read_probe() {
let _ = rustls::crypto::ring::default_provider().install_default();
let token = std::env::var("MOHO_DISCORD_TOKEN").expect("MOHO_DISCORD_TOKEN");
let channel = std::env::var("MOHO_DISCORD_CHANNEL").expect("MOHO_DISCORD_CHANNEL");
let messages: serde_json::Value = super::http_client()
.get(format!("{}/channels/{channel}/messages?limit=10", super::API_BASE))
.header("Authorization", &token)
.send()
.await
.expect("fetching")
.json()
.await
.expect("a JSON answer");
let spoken = messages
.as_array()
.expect("a list of messages")
.iter()
.find(|m| m["flags"].as_u64().unwrap_or(0) & (1 << 13) != 0)
.expect("no voice message in the last ten - send one first");
let attachments = crate::backend::discord::messages::extract_attachments(spoken);
let att = attachments.first().expect("a voice message has an attachment");
println!(
"kind={} voice={} duration={:?} waveform={} bytes",
att.kind,
att.voice,
att.duration_secs,
att.waveform.as_deref().map(str::len).unwrap_or(0)
);
assert!(att.voice, "the parser did not read the flag off a real message");
assert!(att.duration_secs.unwrap_or(0.0) > 0.0, "no length came through");
assert!(att.waveform.is_some(), "no waveform came through");
assert_eq!(att.kind, "audio");
}
}
/// PATCH .../messages/{id} - editing your own message. Discord scopes
/// this to the message author (enforced server-side; there's no separate
/// permission check needed here).
src/main.rs
+2
View File
@@ -9,6 +9,7 @@ mod ignores;
mod ipc;
mod model;
mod net;
mod oggopus;
mod profile;
mod rpc;
mod runtime;
@@ -16,6 +17,7 @@ mod secure;
mod state;
mod store;
mod upload;
mod voicenote;
use accounts::AccountStore;
use anyhow::{bail, Context, Result};
src/oggopus.rs
+340
View File
@@ -0,0 +1,340 @@
//! Turning captured microphone audio into the file a voice message is.
//!
//! Discord's own client records Ogg/Opus, and so does everything that plays
//! one back. The pieces were nearly all here already - cpal captures the
//! microphone for calls, and libopus is linked for them too - but nothing in
//! this daemon had ever written an Ogg container, because a call puts Opus
//! packets straight onto the wire with no container at all.
//!
//! So this is the missing part, and it is small: Ogg is a framing format for
//! packets, and Opus-in-Ogg is three rules on top of it (RFC 7845) - a header
//! packet, a tags packet, then the audio, with the granule position counted in
//! 48kHz samples regardless of what the encoder was actually fed.
//!
//! Written here rather than taken from a crate because it is this much code
//! and the alternative is a dependency whose only use would be these eighty
//! lines. What makes that a reasonable trade is that the result is checkable:
//! the tests hand the output to ffprobe, which is a decoder nobody here wrote.
use anyhow::{Context, Result};
use opus2::{Application, Channels, Encoder};
/// The rate everything speaks in: Opus is defined at 48kHz, the capture path
/// already resamples to it, and an Ogg granule position is counted in it.
pub const RATE: u32 = 48_000;
/// 20ms of audio per packet.
///
/// What every voice application uses, and what Discord's own recordings carry.
/// Smaller packets cost proportionally more header; larger ones lose more when
/// one goes missing, which matters for a call and not for a file, but there is
/// no reason to differ from what every decoder is best exercised on.
const FRAME_MS: u32 = 20;
const FRAME_SAMPLES: usize = (RATE as usize / 1000) * FRAME_MS as usize;
/// Encodes mono 48kHz samples into an Ogg/Opus file.
///
/// Mono because it is somebody talking into one microphone: stereo would
/// double the size to say the same thing twice.
pub fn encode(samples: &[f32]) -> Result<Vec<u8>> {
let mut encoder = Encoder::new(RATE, Channels::Mono, Application::Voip)
.map_err(|e| anyhow::anyhow!("starting the Opus encoder: {e}"))?;
let mut ogg = Ogg::new(rand_serial());
// The identification header, RFC 7845 section 5.1. The pre-skip is the
// encoder's own latency, which a player trims from the front; libopus at
// 48kHz reports 312 samples and is asked rather than assumed.
let pre_skip = encoder.get_lookahead().unwrap_or(312) as u16;
let mut head = Vec::with_capacity(19);
head.extend_from_slice(b"OpusHead");
head.push(1); // version
head.push(1); // channels
head.extend_from_slice(&pre_skip.to_le_bytes());
head.extend_from_slice(&RATE.to_le_bytes()); // the original rate, for information
head.extend_from_slice(&0u16.to_le_bytes()); // output gain
head.push(0); // channel mapping family 0: mono or stereo, no table
ogg.page(&[&head], 0, true, false);
// The comment header, which is required even when there is nothing to say.
let vendor = b"moho";
let mut tags = Vec::new();
tags.extend_from_slice(b"OpusTags");
tags.extend_from_slice(&(vendor.len() as u32).to_le_bytes());
tags.extend_from_slice(vendor);
tags.extend_from_slice(&0u32.to_le_bytes()); // no user comments
ogg.page(&[&tags], 0, false, false);
// The audio itself, one page per packet. A page per packet is more header
// than a long recording needs, and is what keeps this simple enough to be
// obviously right; a minute of speech costs about 1.4KB of framing.
let mut buffer = vec![0u8; 4000];
let mut granule: u64 = 0;
let frames: Vec<&[f32]> = samples.chunks(FRAME_SAMPLES).collect();
for (i, frame) in frames.iter().enumerate() {
// The last frame is padded rather than dropped: a partial frame is
// still something somebody said, and Opus only encodes whole ones.
let mut whole;
let input: &[f32] = if frame.len() == FRAME_SAMPLES {
frame
} else {
whole = frame.to_vec();
whole.resize(FRAME_SAMPLES, 0.0);
&whole
};
let n = encoder
.encode_float(input, &mut buffer)
.map_err(|e| anyhow::anyhow!("encoding {FRAME_MS}ms of audio: {e}"))?;
granule += FRAME_SAMPLES as u64;
ogg.page(&[&buffer[..n]], granule, false, i + 1 == frames.len());
}
// A recording of nothing still has to be a valid file: end the stream even
// when no audio packet was ever written.
if frames.is_empty() {
ogg.page(&[&[]], 0, false, true);
}
Ok(ogg.done())
}
/// How long the recording runs, in seconds, as the service wants it.
pub fn duration_secs(samples: &[f32]) -> f64 {
samples.len() as f64 / RATE as f64
}
/// The picture of the sound that travels with the message.
///
/// Discord takes up to 256 bytes, one amplitude per bucket, and draws them as
/// bars. Peak rather than mean within a bucket: a mean over a fifth of a
/// second of speech is close to silence, and produces the flat line that makes
/// some clients' waveforms look broken.
pub fn waveform(samples: &[f32], buckets: usize) -> Vec<u8> {
if samples.is_empty() || buckets == 0 {
return Vec::new();
}
let buckets = buckets.min(samples.len());
let step = samples.len() as f64 / buckets as f64;
(0..buckets)
.map(|i| {
let from = (i as f64 * step) as usize;
let to = (((i + 1) as f64 * step) as usize).max(from + 1).min(samples.len());
let peak = samples[from..to].iter().fold(0.0f32, |m, s| m.max(s.abs()));
(peak.clamp(0.0, 1.0) * 255.0).round() as u8
})
.collect()
}
/// A stream serial number, which only has to differ between streams in one
/// file. There is always exactly one stream here, so anything does - but a
/// fixed number would make two files concatenated by accident look like one
/// stream with a broken sequence, which is a confusing thing to debug.
fn rand_serial() -> u32 {
use std::time::{SystemTime, UNIX_EPOCH};
SystemTime::now().duration_since(UNIX_EPOCH).map(|d| d.subsec_nanos()).unwrap_or(1) | 1
}
/// The Ogg framing itself.
struct Ogg {
out: Vec<u8>,
serial: u32,
sequence: u32,
}
impl Ogg {
fn new(serial: u32) -> Self {
Self { out: Vec::new(), serial, sequence: 0 }
}
/// One page carrying whole packets.
///
/// Every packet here fits in a page - an Opus frame at any sane bitrate is
/// far under the 255 * 255 bytes a page can hold - so this deliberately
/// does not implement continuation across pages. A packet that would not
/// fit cannot be produced by the encoder above, and pretending to handle a
/// case that cannot arise would be untested code in a file format parser.
fn page(&mut self, packets: &[&[u8]], granule: u64, first: bool, last: bool) {
let mut segments: Vec<u8> = Vec::new();
let mut body: Vec<u8> = Vec::new();
for packet in packets {
let mut left = packet.len();
while left >= 255 {
segments.push(255);
left -= 255;
}
segments.push(left as u8);
body.extend_from_slice(packet);
}
let mut header = Vec::with_capacity(27 + segments.len());
header.extend_from_slice(b"OggS");
header.push(0); // stream structure version
header.push(if first { 0x02 } else if last { 0x04 } else { 0x00 });
header.extend_from_slice(&granule.to_le_bytes());
header.extend_from_slice(&self.serial.to_le_bytes());
header.extend_from_slice(&self.sequence.to_le_bytes());
header.extend_from_slice(&0u32.to_le_bytes()); // checksum, filled in below
header.push(segments.len() as u8);
header.extend_from_slice(&segments);
let mut page = header;
page.extend_from_slice(&body);
let crc = crc32(&page);
page[22..26].copy_from_slice(&crc.to_le_bytes());
self.out.extend_from_slice(&page);
self.sequence += 1;
}
fn done(self) -> Vec<u8> {
self.out
}
}
/// Ogg's own CRC32, which is not the common one.
///
/// The polynomial is the same 0x04C11DB7, but with no input or output
/// reflection and a zero initial and final value - so a stock crc32 gives a
/// number every decoder rejects. This is the one detail of the format most
/// worth writing down, because a wrong checksum fails as "not an Ogg file"
/// rather than as a checksum problem.
fn crc32(data: &[u8]) -> u32 {
let mut crc: u32 = 0;
for &byte in data {
crc ^= (byte as u32) << 24;
for _ in 0..8 {
crc = if crc & 0x8000_0000 != 0 { (crc << 1) ^ 0x04C1_1DB7 } else { crc << 1 };
}
}
crc
}
/// Reads a whole recording out of the capture format into what `encode` wants.
///
/// The capture path hands over interleaved stereo at 48kHz because that is
/// what a call needs; a voice message is mono, so the two channels are averaged
/// rather than one being thrown away - dropping a channel loses half the sound
/// on a microphone that happens to be wired to the other one.
pub fn to_mono(interleaved: &[f32], channels: u16) -> Vec<f32> {
if channels <= 1 {
return interleaved.to_vec();
}
interleaved
.chunks(channels as usize)
.map(|frame| frame.iter().sum::<f32>() / frame.len() as f32)
.collect()
}
/// Writes the encoded recording somewhere it can be uploaded from.
pub fn write_temp(bytes: &[u8]) -> Result<std::path::PathBuf> {
let dir = std::env::temp_dir().join("moho-voice");
std::fs::create_dir_all(&dir).with_context(|| format!("creating {}", dir.display()))?;
let path = dir.join(format!("voice-message-{}.ogg", rand_serial()));
std::fs::write(&path, bytes).with_context(|| format!("writing {}", path.display()))?;
Ok(path)
}
#[cfg(test)]
mod tests {
use super::*;
/// A second of a 440Hz tone, which is something a decoder can check the
/// length of and a person can listen to if this ever needs debugging.
fn tone(seconds: f64) -> Vec<f32> {
let n = (RATE as f64 * seconds) as usize;
(0..n).map(|i| (i as f32 * 440.0 * std::f32::consts::TAU / RATE as f32).sin() * 0.5).collect()
}
#[test]
fn the_file_begins_like_an_ogg_opus_file() {
let bytes = encode(&tone(0.2)).expect("encoding");
assert_eq!(&bytes[0..4], b"OggS", "not an Ogg stream");
// The identification header is the first packet of the first page.
assert_eq!(&bytes[28..36], b"OpusHead");
assert!(bytes.windows(8).any(|w| w == b"OpusTags"), "the comment header is required");
}
/// The checksum is the detail most likely to be wrong and least likely to
/// say so: a decoder reports a bad one as "not an Ogg file".
#[test]
fn the_checksum_is_oggs_own() {
// Known answers, computed from the polynomial by a second
// implementation written in Python rather than taken from this one -
// a self-consistent checksum test proves only that the code agrees
// with itself, which is exactly the failure being guarded against.
assert_eq!(crc32(b"OggS"), 0x5fb0_a94f);
assert_eq!(crc32(b"a"), 0xa864_db20);
assert_eq!(crc32(&[]), 0);
}
/// The test that matters for a container written by hand: hand it to a
/// decoder nobody here wrote and see whether it agrees about what it is
/// and how long it lasts.
///
/// Ignored by default because it shells out to ffprobe, which is not
/// something a test suite should require:
/// cargo test --release -- --ignored ffprobe
#[test]
#[ignore]
fn ffprobe_agrees_that_this_is_two_seconds_of_opus() {
let bytes = encode(&tone(2.0)).expect("encoding");
let path = write_temp(&bytes).expect("writing");
let out = std::process::Command::new("ffprobe")
.args(["-v", "error", "-show_entries", "format=format_name,duration:stream=codec_name,channels,sample_rate", "-of", "default=nw=1", "-i"])
.arg(&path)
.output()
.expect("running ffprobe");
let text = String::from_utf8_lossy(&out.stdout).to_string();
let errors = String::from_utf8_lossy(&out.stderr).to_string();
println!("{text}");
assert!(out.status.success(), "ffprobe refused the file: {errors}");
assert!(errors.is_empty(), "ffprobe complained: {errors}");
assert!(text.contains("codec_name=opus"), "not opus: {text}");
assert!(text.contains("format_name=ogg"), "not ogg: {text}");
assert!(text.contains("channels=1"), "not mono: {text}");
assert!(text.contains("sample_rate=48000"), "wrong rate: {text}");
let duration: f64 = text
.lines()
.find_map(|l| l.strip_prefix("duration="))
.and_then(|v| v.parse().ok())
.expect("a duration");
assert!((duration - 2.0).abs() < 0.05, "ffprobe reads {duration}s, not 2s");
let _ = std::fs::remove_file(path);
}
#[test]
fn a_recording_of_nothing_is_still_a_file() {
let bytes = encode(&[]).expect("encoding silence");
assert_eq!(&bytes[0..4], b"OggS");
assert_eq!(duration_secs(&[]), 0.0);
}
#[test]
fn the_length_is_what_was_recorded() {
assert!((duration_secs(&tone(2.5)) - 2.5).abs() < 0.001);
}
/// Peak per bucket, not mean: speech averaged over a fifth of a second is
/// nearly silence, which is what makes a waveform look broken.
#[test]
fn the_waveform_follows_the_sound() {
let mut samples = vec![0.0f32; 4800];
samples[2400] = 1.0;
let bars = waveform(&samples, 4);
assert_eq!(bars.len(), 4);
assert_eq!(bars[0], 0, "silence should read as silence");
assert_eq!(bars[2], 255, "a peak should read as a peak");
}
#[test]
fn the_waveform_of_nothing_is_nothing() {
assert!(waveform(&[], 256).is_empty());
assert!(waveform(&[0.5], 0).is_empty());
}
#[test]
fn two_channels_become_one_without_losing_a_side() {
// Everything on the right channel: averaging keeps it, picking the
// first channel would produce silence.
let stereo = vec![0.0, 1.0, 0.0, 1.0];
assert_eq!(to_mono(&stereo, 2), vec![0.5, 0.5]);
assert_eq!(to_mono(&[0.25, 0.75], 1), vec![0.25, 0.75]);
}
}
src/rpc/methods.rs
+74
View File
@@ -1040,6 +1040,80 @@ pub async fn dispatch(
}
}
// Recording somebody speaking, and sending it as the thing Discord
// calls a voice message rather than as a sound file.
//
// Three calls rather than one because it is three moments: a person
// starts talking, then decides whether what they said is worth
// sending. The microphone is process-wide - there is one - so none of
// these take an account.
"startVoiceMessage" => {
let Some(buffer_id) = p_str_opt(params, "bufferId") else {
return (None, Some("startVoiceMessage requires \"bufferId\"".to_string()));
};
if state.runtime.get_buffer(buffer_id).is_none() {
return (None, Some("no such buffer".to_string()));
}
// The device the calls settings already chose, so there is not a
// second answer to "which microphone" to get wrong.
let device = state.voice_prefs.get().input;
match crate::voicenote::start(buffer_id, device.as_deref().filter(|d| !d.is_empty())) {
Ok(()) => (Some(ok_node()), None),
Err(e) => (None, Some(format!("{e:#}"))),
}
}
/// How long it has been running and whether the microphone is hearing
/// anything - polled by the window rather than pushed, because it is
/// only wanted while somebody is watching a timer.
"voiceMessageProgress" => match crate::voicenote::progress() {
None => (Some(serde_json::json!({ "recording": false })), None),
Some(p) => (
Some(serde_json::json!({
"recording": true,
"bufferId": p.buffer_id,
"seconds": p.seconds,
"level": p.level,
})),
None,
),
},
"cancelVoiceMessage" => (Some(serde_json::json!({ "wasRecording": crate::voicenote::cancel() })), None),
"sendVoiceMessage" => {
let finished = match crate::voicenote::finish() {
Ok(f) => f,
Err(e) => return (None, Some(format!("{e:#}"))),
};
let Some(buffer) = state.runtime.get_buffer(&finished.buffer_id) else {
let _ = std::fs::remove_file(&finished.path);
return (None, Some("the conversation this was recorded for is gone".to_string()));
};
let Some(cfg) = state.accounts.get_discord(&buffer.account_id) else {
let _ = std::fs::remove_file(&finished.path);
return (None, Some("voice messages are a Discord thing for now".to_string()));
};
let result = backend::discord::send_voice_message(
state,
&finished.buffer_id,
&cfg.token,
&finished.path,
finished.duration_secs,
&finished.waveform,
)
.await;
// Sent or not, the recording has served its purpose on disk.
let _ = std::fs::remove_file(&finished.path);
match result {
Ok(()) => (
Some(serde_json::json!({ "durationSecs": finished.duration_secs })),
None,
),
Err(e) => (None, Some(format!("{e:#}"))),
}
}
"addToDiscordGroupDm" | "removeFromDiscordGroupDm" => {
let (account_id, buffer_id, user_id) = match (
p_str_opt(params, "accountId"),
src/voicenote.rs
+155
View File
@@ -0,0 +1,155 @@
//! Recording somebody speaking, so it can be sent as a voice message.
//!
//! Held here rather than in a backend because none of it is Discord's: a
//! recording is a microphone, some samples and a length, and the only part
//! that knows which service it is going to is the call that sends it.
//!
//! Process-wide, because there is one microphone. Two recordings at once is
//! not a thing to support - it is a thing to refuse, clearly, rather than to
//! half-handle by recording the same audio into two buffers.
//!
//! The capture path is the one calls already use, so a voice message is
//! recorded through the same device choice, the same resampling and the same
//! level meter as a call, and needs no second answer to "which microphone".
use crate::audio::{self, Capture, TARGET_CHANNELS};
use crate::oggopus;
use anyhow::{bail, Result};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::Instant;
/// How long a recording may run before it stops itself.
///
/// Discord's own limit, and a backstop besides: a recording is held in memory
/// until it is sent, and a forgotten one should not grow there all afternoon.
/// Five minutes of 48kHz mono is about 57MB before encoding.
const MAX_SECONDS: f64 = 5.0 * 60.0;
struct Recording {
/// Dropping this stops the microphone, so it is held for the duration.
_capture: Capture,
samples: Arc<Mutex<Vec<f32>>>,
/// Where it will be sent, so stopping needs no second answer.
buffer_id: String,
started: Instant,
level: Arc<Mutex<f32>>,
}
fn slot() -> &'static Mutex<Option<Recording>> {
static SLOT: OnceLock<Mutex<Option<Recording>>> = OnceLock::new();
SLOT.get_or_init(|| Mutex::new(None))
}
/// Opens the microphone and starts collecting.
///
/// The captured audio is folded to mono as it arrives rather than at the end:
/// it halves what is held, and a voice message is mono either way.
pub fn start(buffer_id: &str, device_id: Option<&str>) -> Result<()> {
let mut held = slot().lock().unwrap();
if let Some(existing) = held.as_ref() {
bail!("already recording a message for {}", existing.buffer_id);
}
let samples: Arc<Mutex<Vec<f32>>> = Arc::new(Mutex::new(Vec::new()));
let sink_samples = samples.clone();
let capture = audio::start_capture(device_id, move |pcm: &[f32]| {
let mono = oggopus::to_mono(pcm, TARGET_CHANNELS);
let mut held = sink_samples.lock().unwrap();
// The cap is enforced where the samples arrive rather than by a timer:
// it is the memory that matters, and this is the only place that knows
// how much there is.
let room = (MAX_SECONDS * oggopus::RATE as f64) as usize;
if held.len() < room {
let take = (room - held.len()).min(mono.len());
held.extend_from_slice(&mono[..take]);
}
})?;
let level = capture.level.clone();
*held = Some(Recording {
_capture: capture,
samples,
buffer_id: buffer_id.to_string(),
started: Instant::now(),
level,
});
Ok(())
}
/// What a recording in progress looks like from outside.
pub struct Progress {
pub buffer_id: String,
pub seconds: f64,
/// Peak level since the last look, so a window can show that the
/// microphone is actually hearing something.
pub level: f32,
}
pub fn progress() -> Option<Progress> {
let held = slot().lock().unwrap();
held.as_ref().map(|recording| Progress {
buffer_id: recording.buffer_id.clone(),
seconds: recording.started.elapsed().as_secs_f64(),
level: *recording.level.lock().unwrap(),
})
}
/// A finished recording, encoded and ready to be sent.
pub struct Finished {
pub buffer_id: String,
/// The file itself, on disk, because that is what the upload paths take.
pub path: std::path::PathBuf,
pub duration_secs: f64,
/// Discord's own encoding: one amplitude byte per bucket, base64'd.
pub waveform: String,
pub samples: usize,
}
/// Stops the microphone and turns what was said into a file.
///
/// The length comes from the samples rather than from the clock: the two
/// disagree by whatever the device buffered, and the samples are what a player
/// will actually find in the file.
pub fn finish() -> Result<Finished> {
let recording = slot().lock().unwrap().take().ok_or_else(|| anyhow::anyhow!("nothing is being recorded"))?;
let samples = std::mem::take(&mut *recording.samples.lock().unwrap());
if samples.is_empty() {
bail!("the microphone produced no audio");
}
let duration_secs = oggopus::duration_secs(&samples);
let waveform = base64_of(&oggopus::waveform(&samples, 256));
let bytes = oggopus::encode(&samples)?;
let path = oggopus::write_temp(&bytes)?;
Ok(Finished { buffer_id: recording.buffer_id, path, duration_secs, waveform, samples: samples.len() })
}
/// Throws the recording away and closes the microphone.
pub fn cancel() -> bool {
slot().lock().unwrap().take().is_some()
}
/// Base64, of the flavour every service means when it says base64.
fn base64_of(bytes: &[u8]) -> String {
use base64::Engine;
base64::engine::general_purpose::STANDARD.encode(bytes)
}
#[cfg(test)]
mod tests {
use super::*;
/// Nothing may be finished that was never started - and saying so is the
/// difference between an error a window can show and a panic.
#[test]
fn finishing_nothing_is_an_error_rather_than_a_panic() {
// Whatever else has run, this test's precondition is an empty slot.
cancel();
assert!(finish().is_err());
assert!(!cancel(), "cancelling nothing should say so");
}
/// The waveform travels as base64 and has to survive the trip.
#[test]
fn the_waveform_encodes_the_way_the_service_reads_it() {
assert_eq!(base64_of(&[0, 128, 255]), "AID/");
assert_eq!(base64_of(&[]), "");
}
}