Modular Audio Pipeline Architecture

This document specifies the modular, composable audio pipeline architecture for the IFM Protocol system as specified in AUDIO_MIXING.md.

---

1. High-Level Modular Architecture

The audio architecture is designed around clear boundaries and composition, ensuring that audio production, mixing, processing, encoding, recording, and transport evolve as independent, replaceable components.

Discovery remains completely independent:

---

2. Component Boundaries & Responsibilities

A. Audio Sources

• Responsibility: Produce raw PCM audio streams (e.g. microphone input via cpal or Web Audio, file playback, soundboard, or synthetic audio generators).
• Abstraction: The station console implements the generic AudioSource interface (packages/ifm-station/src/mixer/sources.ts) — MicSource (getUserMedia; a deviceId is also Line In), TabSource + DesktopSource (getDisplayMedia: window audio and system/desktop audio), FileSource (decode + loop, with optional strip transport — play/pause · stop · speed via FileTransportState). Every kind attaches into the SAME processing chain, so swapping a source never changes the console. NetworkSource (remote-station channels) arrives with the native IFM transport.
• Rule: Audio sources have zero knowledge of networking, transports, or recording sinks.

B. Mixer (Optional / Replaceable)

• Responsibility: Sum and balance multiple input audio streams into a mixed PCM output.
• Optionality: The pipeline can bypass the mixer entirely when a single audio source is used (Source → Processing → Codec → Audio Frames).
• Rule: The mixer MUST NOT know anything about networking, peers, relays, rendezvous, transports, or recording destinations.

C. Audio Processing (DSP)

• Responsibility: Process post-mixer (or direct source) PCM audio. Performs equalization, normalization, dynamic range compression, acoustic echo cancellation, timing adjustments, and energy VAD.
• Rule: Audio processing operates strictly on PCM audio and MUST NOT depend on network transport or socket logic.

D. Codec

• Responsibility: Encode PCM audio into compressed frames (e.g. Opus 10ms frames at 48kHz mono — real-time default; 20ms valid for recording) and decode compressed frames back to PCM audio (frame-size agnostic).
• Rule: Pure encoder/decoder implementation. The codec boundary is the stable EncodedAudioFrame.

E. Audio Frames (EncodedAudioFrame)

• Responsibility: Authoritative binary representation of timestamped, sequenced audio frames (codec, format, sample_rate, channels, sequence, timestamp, data).
• Rule: Owned authoritatively by crates/protocol. Shared by SDK, station, listener, relay, and recording modules.

F. Record Layer (Optional / Independent)

• Responsibility: Persist audio to disk or storage media.
• Abstraction: The station console's record layer is MediaRecorderRecorder (packages/ifm-station/src/mixer/recorder.ts) — the SINGLE place the station touches MediaRecorder. The engine's broadcast and record buses both delegate to it (startBroadcast/startRecord); the mixer only routes a bus into a MediaStream. The final mix sink is injected by the app (native Tauri save dialog in the desktop app, object-URL download in the web demo), so the record layer never knows where files land.
• Recording Sinks:
  1. PCM Recorder: Captures processed PCM post-DSP for high-quality, lossless local master recording.
  2. Encoded Frame Recorder: Captures exact EncodedAudioFrame streams for broadcast archive logging.
• Non-Blocking Rule: Recording MUST run asynchronously with independent buffering. A slow disk or file encoder MUST NEVER stall or introduce latency into live broadcast transport.

G. Transport (Replaceable)

• Responsibility: Transport EncodedAudioFrame datagrams over P2P mesh, QUIC datagrams, WebRTC DataChannels, or TCP fallback streams.
• Rule: Transport operates solely on protocol-level audio frames. It has no knowledge of microphones, mixers, DSP, or recording destinations.

---

3. Reverse Pipeline: Listener Architecture

The listener side mirrors the sender pipeline in reverse:

---

4. Repository Ownership Mapping

Module / Component	Owning Repository / Crate
Protocol types & EncodedAudioFrame	crates/protocol
Audio Codec (Opus), Jitter Buffer, VAD	crates/audio
Transport abstractions (QUIC, WebRTC, TCP)	crates/transport
Node orchestration	crates/core
Discovery & Rendezvous service	crates/rendezvous
Developer-facing SDK API	packages/sdk
Station UI, Mixer controls, Sources, Recording config	packages/ifm-station
Relay mesh forwarding & pool management	packages/ifm-relay-station
Listener decoding, playback UI, optional recording	packages/ifm-radio-listener
Mobile PWA browser audio player	packages/ifm-pwa

5. Real-Time Audio Transport Design

1. Use very small audio frames

Opus at 48 kHz with 5 ms or 10 ms frames.

Don't make the packet contain 1 second of audio:

BAD:
[ 1 second of audio ]

Instead:

[5ms][5ms][5ms][5ms][5ms]...

That means a packet can be transmitted almost immediately after the audio is captured.

2. Don't use a traditional streaming protocol

Avoid architectures like:

That's great for YouTube/Netflix-style reliability, but terrible for radio latency.

IFM should behave more like a continuous packet stream.

3. Don't re-encode at relays

This is extremely important for your P2P architecture.

A relay should essentially do:

Not:

You want:

This makes relays cheap and keeps latency extremely low.

4. Make the jitter buffer adaptive

This is probably one of the most important pieces.

You don't want:

always buffer 500 ms

Instead:

And when the network stabilizes, gradually reduce the buffer again.

This gives you a radio-like experience without unnecessarily sacrificing reliability.

5. UDP/QUIC rather than TCP-style streaming

For real-time audio, you generally don't want an old packet to block newer packets.

Imagine a lost packet in the stream:

With real-time audio, you'd rather play:

and conceal the missing 102 than wait indefinitely for it.

That's why the transport should support unreliable/low-latency delivery, with Opus packet-loss concealment handling small losses.

6. Separate control traffic from audio

Your IFM architecture should have something like:

The media plane should be extremely simple.

Don't let a chat message, peer discovery operation, logging operation, etc. block audio.

7. Measure the actual latency

This is where your IFM Dashboard idea becomes really useful.

You should be able to see:

Then IFM can expose:

End-to-end audio latency: 70 ms

That's much more useful than simply saying "connected."

The really interesting part

You don't necessarily need to beat FM in physical propagation latency. That's essentially impossible over long Internet distances.

Instead, you want to beat traditional Internet radio by eliminating the unnecessary latency:

Traditional Internet Radio (seconds of latency):

versus IFM (~tens of milliseconds):

That could make IFM feel much more like tuning into a live FM station than listening to an Internet stream.

And I'd make “Live Mode” a fundamental property of the IFM audio protocol, rather than something implemented by the UI/player. That way every IFM application—desktop, mobile, relay, embedded node, etc.—inherits the same low-latency behavior.