Somebody speaking, drawn as somebody speaking
A voice message showed a generic audio player and a filename. The two things a person wants before deciding to listen - how long it is, and what the sound looks like - were both sent by the sender's client and both thrown away. Now it draws the sender's own waveform, the length, a play button and a progress that fills as it plays, and clicking anywhere along the picture seeks there. All of it is available before a byte of audio is fetched, which is the point: fetching it is the thing being decided about. The waveform maths is a plain exported function, because it is the only part that can be wrong quietly - a waveform decoded badly still draws, it just draws a lie about what was said. It averages rather than samples, so a short loud moment does not vanish because the step landed either side of it, and anything unreadable gives a flat line rather than an exception. Checked against Discord's encoding: full and silent samples, averaging, a spike in a quiet recording, no waveform, empty, and somebody else's garbage. Looked at as well as tested, in a throwaway harness under .uicheck, which is where a bug turned up that no unit test would have: the control replaced itself with "could not be played" on sight. The audio element reports an error for a source it cannot load, and a Discord attachment URL expires - so an old message threw away the length and the waveform, which are known here without fetching anything. An error nobody provoked is not a failure to report, and it now only counts after somebody has actually pressed play. This is the receiving half of #186. Sending is not done: it needs recording, an Ogg/Opus encode, and Discord's upload flow with the flag and waveform attached - and a write to Discord to test. Refs #186. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Claude Opus 5
+1
-1
Submodule nobilis updated: 207de58cf7...54ac74e4fd
@@ -3547,6 +3547,76 @@ img.smilie {
|
||||
height: 32px;
|
||||
}
|
||||
|
||||
/* --- a voice message ------------------------------------------------- */
|
||||
|
||||
/* Sized like the audio row it replaces rather than like a media embed: this
|
||||
is a control, not a picture, and it sits inline in a conversation. */
|
||||
.voice-message {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--spacing-s);
|
||||
padding: var(--spacing-xs) var(--spacing-s);
|
||||
background: var(--surface-container-high);
|
||||
border-radius: var(--corner-radius-small);
|
||||
max-width: 420px;
|
||||
}
|
||||
|
||||
.voice-message.failed {
|
||||
gap: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.voice-play {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
flex: none;
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
border: none;
|
||||
border-radius: 50%;
|
||||
color: var(--on-primary);
|
||||
background: var(--primary);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.voice-play:hover {
|
||||
filter: brightness(1.1);
|
||||
}
|
||||
|
||||
/* The picture of the sound. Clickable along its length, which is what makes
|
||||
it a scrubber as well as a display. */
|
||||
.voice-wave {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 2px;
|
||||
flex: 1;
|
||||
height: 28px;
|
||||
min-width: 0;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.voice-bar {
|
||||
flex: 1;
|
||||
min-width: 1px;
|
||||
border-radius: 1px;
|
||||
background: var(--outline-strong);
|
||||
/* Grown from the middle, so a loud moment reads as loud in both directions
|
||||
rather than as a column standing on the floor. */
|
||||
align-self: center;
|
||||
}
|
||||
|
||||
/* How far in the listener is. The played part keeps its shape and changes
|
||||
colour, so the waveform stays readable as a whole while it plays. */
|
||||
.voice-bar.played {
|
||||
background: var(--primary);
|
||||
}
|
||||
|
||||
.voice-time {
|
||||
flex: none;
|
||||
min-width: 34px;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.media-embed.file-card {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { useState } from 'react'
|
||||
import { Icon } from './Icon'
|
||||
import { Lightbox } from './Lightbox'
|
||||
import { VoiceMessage } from './VoiceMessage'
|
||||
import { fullImageFor, knownFullImage } from '../lib/fullimage'
|
||||
import { resolveMediaUrl } from '../lib/util'
|
||||
import type { MediaItem } from '../lib/format'
|
||||
@@ -360,6 +361,14 @@ export function MediaEmbed({
|
||||
)
|
||||
}
|
||||
|
||||
// Somebody speaking rather than a sound file somebody attached. The service
|
||||
// says which, and the two want different controls: a voice message is short,
|
||||
// has a length worth knowing before it is fetched, and comes with a picture
|
||||
// of itself.
|
||||
if (kind === 'audio' && attachment?.voice) {
|
||||
return <VoiceMessage attachment={attachment} />
|
||||
}
|
||||
|
||||
if (kind === 'audio') {
|
||||
return (
|
||||
<div className="media-embed audio">
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
import { useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Icon } from './Icon'
|
||||
import { resolveMediaUrl } from '../lib/util'
|
||||
import type { Attachment } from '../../../shared/wire'
|
||||
|
||||
/**
|
||||
* Somebody speaking, drawn as the thing it is.
|
||||
*
|
||||
* A voice message used to arrive here as an audio file with a generic player
|
||||
* and a name - `voice-message.ogg` - and nothing to say how long it was or
|
||||
* that anybody had spoken it. That is ordinary traffic in a Discord DM, and
|
||||
* the two things a person wants before deciding to listen are exactly the two
|
||||
* the sender's client already provided and this dropped: the length, and the
|
||||
* shape of the sound.
|
||||
*
|
||||
* The waveform is the sender's own, not one computed here. Discord ships up to
|
||||
* 256 bytes of amplitude with the message, so the picture can be drawn before a
|
||||
* single byte of audio is fetched - which is the point, since fetching it is
|
||||
* the thing being decided about.
|
||||
*/
|
||||
export function VoiceMessage({ attachment }: { attachment: Attachment }): JSX.Element {
|
||||
const audio = useRef<HTMLAudioElement>(null)
|
||||
const [playing, setPlaying] = useState(false)
|
||||
const [at, setAt] = useState(0)
|
||||
const [failed, setFailed] = useState(false)
|
||||
/**
|
||||
* Whether anybody has actually asked to hear this yet.
|
||||
*
|
||||
* The element reports an error for a source it cannot load, and a Discord
|
||||
* attachment URL expires - so without this, a message whose link had aged
|
||||
* out replaced itself with "could not be played" on sight, throwing away the
|
||||
* length and the waveform, which are known here without fetching anything
|
||||
* and are most of what the control is for. An error nobody provoked is not
|
||||
* yet a failure to report.
|
||||
*/
|
||||
const asked = useRef(false)
|
||||
|
||||
const src = resolveMediaUrl(attachment.path || attachment.url || '')
|
||||
const bars = useBars(attachment.waveform)
|
||||
// The sender's figure where there is one. An audio element only knows the
|
||||
// length once it has enough of the file to say, which is after the fetch
|
||||
// this display exists to let somebody avoid.
|
||||
const total = attachment.durationSecs ?? audio.current?.duration ?? 0
|
||||
const played = total > 0 ? Math.min(1, at / total) : 0
|
||||
|
||||
useEffect(() => {
|
||||
const el = audio.current
|
||||
if (!el) return
|
||||
const tick = (): void => setAt(el.currentTime)
|
||||
const ended = (): void => {
|
||||
setPlaying(false)
|
||||
setAt(0)
|
||||
}
|
||||
el.addEventListener('timeupdate', tick)
|
||||
el.addEventListener('ended', ended)
|
||||
return () => {
|
||||
el.removeEventListener('timeupdate', tick)
|
||||
el.removeEventListener('ended', ended)
|
||||
}
|
||||
}, [src])
|
||||
|
||||
const toggle = (): void => {
|
||||
const el = audio.current
|
||||
if (!el) return
|
||||
if (el.paused) {
|
||||
asked.current = true
|
||||
void el.play().catch(() => setFailed(true))
|
||||
setPlaying(true)
|
||||
} else {
|
||||
el.pause()
|
||||
setPlaying(false)
|
||||
}
|
||||
}
|
||||
|
||||
/** Scrubbing, by clicking the picture of the sound at the point wanted. */
|
||||
const seek = (e: React.MouseEvent<HTMLDivElement>): void => {
|
||||
const el = audio.current
|
||||
if (!el || !total) return
|
||||
const box = e.currentTarget.getBoundingClientRect()
|
||||
const fraction = Math.min(1, Math.max(0, (e.clientX - box.left) / box.width))
|
||||
el.currentTime = fraction * total
|
||||
setAt(el.currentTime)
|
||||
}
|
||||
|
||||
if (failed) {
|
||||
return (
|
||||
<div className="voice-message failed small muted">
|
||||
<Icon name="mic_off" size={18} />
|
||||
<span>This voice message could not be played.</span>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="voice-message">
|
||||
<button
|
||||
type="button"
|
||||
className="voice-play"
|
||||
disabled={!src}
|
||||
onClick={toggle}
|
||||
title={playing ? 'Pause' : 'Play'}
|
||||
aria-label={playing ? 'Pause' : 'Play'}
|
||||
>
|
||||
<Icon name={playing ? 'pause' : 'play_arrow'} size={20} />
|
||||
</button>
|
||||
|
||||
<div className="voice-wave" onClick={seek} role="presentation">
|
||||
{bars.map((height, i) => (
|
||||
<span
|
||||
key={i}
|
||||
className={i / bars.length <= played ? 'voice-bar played' : 'voice-bar'}
|
||||
// A floor, so silence is still a mark on the line rather than a
|
||||
// gap: the bar says where in the message you are, and a run of
|
||||
// quiet is part of what somebody said.
|
||||
style={{ height: `${Math.max(8, height * 100)}%` }}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<span className="voice-time small muted tabular">
|
||||
{clock(playing || at > 0 ? at : total)}
|
||||
</span>
|
||||
|
||||
<audio
|
||||
ref={audio}
|
||||
src={src}
|
||||
preload="none"
|
||||
onError={() => {
|
||||
if (asked.current) setFailed(true)
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* How many bars to draw.
|
||||
*
|
||||
* Discord sends up to 256 samples and its own client draws far fewer; at the
|
||||
* width this sits in, one bar per sample is a solid block. The samples are
|
||||
* averaged down rather than picked from, so a short spike does not vanish
|
||||
* because the sampling happened to step over it.
|
||||
*/
|
||||
const BARS = 48
|
||||
|
||||
/** The sender's waveform, decoded and reduced to the bars actually drawn. */
|
||||
function useBars(waveform: string | undefined): number[] {
|
||||
return useMemo(() => barsFrom(waveform), [waveform])
|
||||
}
|
||||
|
||||
/**
|
||||
* Somebody else's base64, turned into the heights of the bars drawn.
|
||||
*
|
||||
* Exported for its own sake: this is the only arithmetic in the component and
|
||||
* the only part that can be wrong quietly - a waveform decoded badly still
|
||||
* draws, it just draws a lie about what was said.
|
||||
*
|
||||
* Averaged down rather than sampled, so a short loud moment does not vanish
|
||||
* because the step happened to land either side of it. Anything unreadable
|
||||
* gives a flat line: a voice message with no usable waveform is still a voice
|
||||
* message, and Discord does send them without one.
|
||||
*/
|
||||
export function barsFrom(waveform: string | undefined, bars = BARS): number[] {
|
||||
const flat = (): number[] => new Array(bars).fill(0.25)
|
||||
if (!waveform) return flat()
|
||||
try {
|
||||
const raw = atob(waveform)
|
||||
if (raw.length === 0) return flat()
|
||||
const samples = Array.from(raw, (c) => c.charCodeAt(0) / 255)
|
||||
const step = samples.length / bars
|
||||
return Array.from({ length: bars }, (_, i) => {
|
||||
const from = Math.floor(i * step)
|
||||
const to = Math.max(from + 1, Math.floor((i + 1) * step))
|
||||
const slice = samples.slice(from, to)
|
||||
return slice.reduce((sum, v) => sum + v, 0) / (slice.length || 1)
|
||||
})
|
||||
} catch {
|
||||
return flat()
|
||||
}
|
||||
}
|
||||
|
||||
/** Seconds as a person reads them. */
|
||||
export function clock(seconds: number): string {
|
||||
if (!Number.isFinite(seconds) || seconds <= 0) return '0:00'
|
||||
const whole = Math.floor(seconds)
|
||||
return `${Math.floor(whole / 60)}:${String(whole % 60).padStart(2, '0')}`
|
||||
}
|
||||
@@ -399,6 +399,18 @@ export interface Attachment {
|
||||
path?: string
|
||||
thumbnailPath?: string
|
||||
url?: string
|
||||
/** How long the recording runs, where the service says. */
|
||||
durationSecs?: number
|
||||
/**
|
||||
* The picture of the sound, as the sender's client drew it: Discord sends up
|
||||
* to 256 base64'd bytes, each one an amplitude sample.
|
||||
*/
|
||||
waveform?: string
|
||||
/**
|
||||
* Somebody spoke this rather than attaching a file of it - the service's own
|
||||
* flag, not a guess from the mimetype.
|
||||
*/
|
||||
voice?: boolean
|
||||
}
|
||||
|
||||
export interface Message {
|
||||
|
||||
Reference in New Issue
Block a user