IFM Audio & Voice Architecture
Overview
IFM provides real-time, low-latency, studio-quality voice and audio communication over the decentralized mesh. The audio system follows a strictly modular, composable pipeline architecture where audio sources, mixing, DSP processing, encoding, recording, transport, and peer discovery are independent, replaceable components.
Full Architectural Specification: See Modular Audio Pipeline Architecture
1. Modular Pipeline Architecture
2. Pipeline Composition & Optionality
Components are composed as needed — optional components are never forced into the core pipeline:
- Full Station Studio Pipeline:
Sources → Mixer → Processing (DSP) → Codec → Audio Frames → (Transport + Record) - Direct Source Pipeline (Mixer Bypassed):
Source → Processing → Codec → Audio Frames → Transport - Broadcast Only:
Audio Pipeline → Transport - Record Only:
Audio Pipeline → Record - Broadcast + Record:
Audio Pipeline ──┬──→ Transportand└──→ Record(non-blocking, independently buffered)
3. Component Boundaries
| Stage | Responsibility | Primary Crate / Package |
|---|---|---|
| Sources | Microphone capture (cpal/Web Audio), window / desktop audio (getDisplayMedia), file stream (strip transport: play/pause · stop · speed), soundboard — behind the generic AudioSource interface (MicSource/TabSource/DesktopSource/FileSource) | packages/ifm-station/src/mixer/sources.ts, crates/cli |
| Mixer | Summing and channel balance (optional) | packages/ifm-station/src/mixer/ |
| Processing | DSP, normalization, VAD, timing | crates/audio/src/vad.rs, audioEngine.ts |
| Codec | Opus encoding/decoding (10ms real-time default; 5/20ms runtime-selectable) | crates/audio/src/codec.rs |
| Audio Frames | Timestamped, sequenced frame boundary (EncodedAudioFrame) | crates/protocol/src/frame.rs |
| Record | Independent asynchronous recorder (MediaRecorderRecorder) — the single place the station touches MediaRecorder; broadcast + record buses delegate to it | packages/ifm-station/src/mixer/recorder.ts, crates/core |
| Transport | QUIC datagrams, WebRTC DataChannels, TCP mesh forwarding | crates/transport |
| Discovery | Peer lookup & signaling (headless rendezvous) | crates/rendezvous, crates/discovery |
4. Listener Reverse Pipeline
5. Non-Blocking Recording Principles
- Independent Buffering: Recording consumes audio asynchronously from an internal ring buffer.
- Zero Transport Stalling: Slow disk I/O, file encoding, or storage failure MUST NOT block or introduce latency into the live broadcast path.
- Dual Recording Modes:
- PCM Recorder: Captures processed PCM for local master/studio archive.
- Encoded Frame Recorder: Captures exact
EncodedAudioFramestream for broadcast logging.
6. Live Mode & Low Latency ("feels like FM")
IFM is designed around real-time packet transport, not streaming — the architecture already satisfies most of the low-latency checklist:
| Plan item | IFM implementation |
|---|---|
| Small frames (5–10ms) | Default 10ms Opus frames at 48kHz (100fps, 480 samples) for real-time media — spec docs/features/realtime-media-transport.md §5. 20ms remains valid for recording; 5ms is a runtime-selectable experiment |
| No streaming protocol | Packet-based gossipsub/QUIC — no HLS, no segment buffering |
| No relay re-encode | Relays forward opaque packets (relay neutrality) — never decode/re-encode audio |
| Adaptive jitter buffer | crates/audio/src/jitter.rs adapts target depth 40–60ms (up to 200ms) from observed jitter, with PLC on gaps |
| UDP/QUIC not TCP-style | libp2p QUIC; Opus FEC + DTX + PLC cover small losses (no head-of-line blocking on old packets) |
| Control/media separation | Control plane (discovery/relay/station topics) is separate from frequency voice topics; audio never blocks or is blocked by chat/file/discovery |
| Measure actual latency | JitterBuffer::latency_ms() reports the buffered stage in ms; codec + jitter are pure and cheap enough to instrument per stage |
Live Mode
AudioConfig::low_latency() is the "Live Mode" preset:
- 10ms frames at 48kHz (480 samples) — the real-time media default (spec
docs/features/realtime-media-transport.md§5); 20ms remains valid for recording. - Jitter target 20→30→50ms (adaptive, spec §7) — the buffer grows on arrival jitter and shrinks back when the network recovers; clamped to the configured 2–12 frame bounds.
- FEC + DTX stay enabled, so the lower latency doesn't cost loss resilience.
rust
let cfg = ifm_audio::AudioConfig::low_latency();
let mut enc = ifm_audio::OpusEncoder::new(&cfg)?;
let mut dec = ifm_audio::OpusDecoder::new(&cfg)?;The wire boundary (EncodedAudioFrame + AudioFormat) carries the frame duration per frame, so listeners learn the cadence from the frames themselves; recorders and transport consume identical frames regardless of 10ms vs 20ms.
7. Verification & Test Suite
- Audio Crate Tests:
cargo test -p ifm-audio - Protocol Frame Tests:
cargo test -p ifm-protocol - Station Mixer Tests:
bun testinpackages/sdk/examples/station