Feature Spec — Real-Time Media Transport & P2P-First Mesh

Status: 🚧 Implemented in progress — the real-time media foundation (Phase 1) is implemented and tested; the P2P-first mesh and advanced phases (3–6) are partially built and tracked below. Implementation progress is tracked in the Implementation Tracker at the bottom of this file and in Real-Time Media Refactor Plan.
Repository truth: the code in crates/ and packages/ is authoritative; update this file whenever the implementation changes.
Implementation plan: the file-grounded, phase-by-phase refactor for this spec (protocol wire format, 10 ms pipeline, datagram channel, P2P-first mesh, telemetry) is in Real-Time Media Refactor Plan.

1. Goal & Non-Goals

Goal

Make IFM radio feel like live FM, not like an Internet stream:

Primary target: under 100 ms end-to-end station → listener latency under good conditions.
Secondary target: 50–80 ms as capture, encode, jitter and hop costs are optimized.
Latency is a budget, not a single protocol choice. Every stage must be accounted for.

Non-Goals

We do not replace the packet-based transport with a traditional streaming protocol (HLS/RTMP-style segments + buffers).
We do not guarantee sub-Internet physical propagation latency (station and listener are still governed by geography).
We do not sacrifice Opus quality, PLC resilience, or the modular audio pipeline's boundaries (Invariant-driven).
We do not make the relay a mandatory hop — the opposite (see Part B).

1.1 Station-Owned Content State & Real-Time Stream Model

IFM is built on a core protocol model where stations own their source and content state, and the protocol transports real-time streams and associated data without imposing file transfer or centralized transcoding paradigms:

Stations own the source/content state.
The protocol transports the station's real-time stream and associated data.
Listeners do the media conversion/decoding locally.
Radio is inherently real-time — latency matters more than eventual consistency.
The network prioritizes fast path establishment and continuous delivery, rather than making listeners wait for complete files or centralized processing.
A listener does not need the station to pre-convert audio into the listener's desired format.
Stations do not perform expensive per-listener transcoding.
Once connected, listeners consume the stream continuously and perform required conversion locally.

Conceptual Architecture

Desired Startup Flow

Non-Goal Anti-Pattern:

2. Terminology

Term	Meaning
Peer	Any node on the mesh (listener, source, relay).
Source	A peer that originates a media stream (broadcaster / station).
Listener	A peer that consumes and plays back a media stream.
Forwarder	A peer that re-broadcasts a stream to other peers (temporary or long-lived).
Relay	Dedicated forwarding infrastructure (the shared relay hub/pool). Fallback only.
Live Edge	The playback position at which a listener should be hearing "now".
Media packet	A single `EncodedAudioFrame` (or small group) carrying seq + timestamp.
Path Manager	The component that selects/migrates the delivery path for a stream.

3. Architecture Overview

Two planes over a single packet-based transport:

Distribution principle (new):

text

Direct P2P is the default.
Relays are used only when the P2P mesh cannot maintain the desired service quality.

The relay is not "where the radio lives". It is the emergency infrastructure for when the mesh can't handle the radio — overflow, stabilization, NAT traversal, or regional rescue.

Part A — Real-Time Media Transport

4. Latency Budget Model

Measure the full pipeline, and put a timestamp at every stage:

Example budget:

text

Capture:       0 ms
Encode:        +2 ms
Packetize:     +0.2 ms
Network:       +18 ms
Relay:         +2 ms
Jitter:        +20 ms
Decode:        +1 ms
Output:        +5 ms
--------------------
TOTAL:          48.2 ms

Rule: Never optimize blindly. The dashboard reports where latency actually is.

5. Small Audio Frames

Send small frames, not large blocks:

Opus frame-size strategy:

Frame	Pros	Cons
5 ms	Minimum latency	More packet overhead, higher packet rate
10 ms	Good balance (default for this feature)	—
20 ms	More efficient, lower overhead	More latency

Defaults: 10 ms frames for real-time media (96 frames/s at 48 kHz, 480 samples); 5 ms and 20 ms are runtime-selectable. This aligns with the existing AudioConfig::low_latency() preset (see Audio Pipeline & Codec).

6. Datagram Transport — Reliability for Files, Latency for Audio

No TCP-style retransmission for media:

Traffic	Priority
FILE	reliability > latency
CHAT	reliability
RADIO AUDIO	latency > reliability

Lost packet handling — never wait for the missing packet:

Skip the missing frame and let Opus PLC conceal it — do not stall the stream.

Media flows over UDP/QUIC datagrams (or the equivalent unreliable transport in the IFM transport layer), independent of reliable channels.

7. Adaptive, Small Jitter Buffer

A large fixed buffer trades latency for comfort. Real-time radio wants a small, adaptive buffer:

Adaptive targets:

Never hold a permanently huge buffer "just in case". When the network stabilizes, shrink back toward the low end. (Extends crates/audio/src/jitter.rs.)

8. No Transcoding at Relays

Relays are forwarders, not processors:

Relay forwarding keeps CPU ↓, latency ↓, complexity ↓, and audio quality intact.

9. Relays Forward Immediately

No batching:

Target: forward on the order of micro/milliseconds.

10. No ACKs for Media

The media path never does send-and-wait:

Loss/sequence information travels alongside the stream, not as a gate on it.

11. Sequence Number + Timestamp on Every Frame

Every media frame carries stream metadata (already the shape of EncodedAudioFrame):

text

Stream ID    : RADIO-01
Sequence     : 48392
Timestamp    : 92839201
Codec        : Opus
Duration     : 10 ms
Payload      : ...

The receiver detects loss from sequence gaps immediately:

text

→ 48395 is lost. No waiting, no timeout.

12. Audio Clock, Not Network Clock

Playback follows the audio device clock, never the packet arrival rate:

text

network:  20 ms  18 ms  25 ms  17 ms  30 ms
audio:    10 ms  10 ms  10 ms  10 ms  10 ms

The jitter buffer absorbs the difference between network cadence and audio cadence.

13. Separate Radio Latency from Network Latency

Ping reports network RTT only. Radio latency is the sum of all stages. Expose per-stage telemetry:

This is first-class telemetry for the IFM dashboard (see Observability).

14. Latency Modes

Expose three player modes, chosen manually or automatically from network conditions:

Mode	Jitter target
LOW LATENCY	~20–40 ms
BALANCED	~50–100 ms
STABLE	~100–250 ms

15. Media Priority & No Backpressure Propagation

Audio gets its own priority; lower-priority traffic must never share its queues:

text

Priority 0   AUDIO REALTIME
Priority 1   CONTROL
Priority 2   CHAT
Priority 3   FILES
Priority 4   LOGGING

A slow consumer must never stall the source or other listeners:

Each listener gets its own bounded media queue. No infinite queues.

16. Live Edge (First-Class Concept)

Every listener has a position relative to the live edge:

Rules:

If playback is more than ~1 live-edge worth behind, discard obsolete frames and jump toward the live edge — it's a live broadcast, not a file.
Detect and recover falling-behind listeners automatically.
The live-edge offset is telemetry, more useful than "buffer contains 200 ms".

17. Redundant Packets & Selective FEC

Preferred over retransmission (which arrives too late for live media):

Sliding redundancy:

Selective FEC:

Rule: do not overdo FEC — it costs bandwidth and can add latency. Use it selectively (e.g. on high-loss paths).

18. Capture-Side Latency

Avoid deep OS/application buffers before the encoder:

The desktop app's audio backend/configuration matters as much as the network.

19. Hot Path: Ring Buffer, Not Async Queues

Keep the real-time critical path predictable:

No generic async queue on the audio hot path; no scheduling jitter in the signal path.

20. UI Out of the Critical Path

The UI observes the media system; it never controls media timing:

21. Clock Synchronization (Future)

To reach "every listener hears the same instant", nodes need reasonably synchronized clocks and/or a station timeline:

Not "each node plays whatever arrived next". Deferred — see Open Questions.

Part B — P2P-First Mesh with Relay Fallback

22. Principle: Direct P2P Is the Default

The relay is not the normal path. It is fallback / overflow / stabilization.

23. Distribution Mesh (Listeners Forward)

A station with 1,000 listeners does not maintain 1,000 connections:

The station feeds a small number of peers; those peers propagate the stream. More listeners = more distribution capacity, not more station bandwidth.

24. Don't Blindly Forward — Advertise Capacity

A listener becomes a forwarder only if it has the headroom. Every peer advertises PEER_CAPACITY:

text

upload_capacity      : 20 Mbps
available_upload     : 12 Mbps
latency_to_station   : 18 ms
packet_loss          : 0.2 %
jitter               : 3 ms
relay_capacity       : 5 streams

IFM makes forwarding decisions from this data.

25. Dynamic P2P Mesh, Not a Rigid Tree

A mesh with preferred forwarding relationships, and multiple possible upstreams:

C can switch upstream at any time. If A dies, C immediately switches to B. The mesh is resilient by construction.

26. Automatic Relay Activation

Relay assistance is triggered by mesh health, not by default:

27. Localized Relay Assistance

Only the problematic region uses the relay:

Relay becomes localized infrastructure, not the path for everyone.

28. Roles: Peer = Listener | Source | Forwarder

Internally, don't call listener-forwarders "relays". A peer can transition:

Any listener may stop forwarding whenever it wants. Users contribute network capacity without running dedicated infrastructure.

29. Routing Cost Function

The router optimizes a cost, not a single metric:

text

cost =
    latency
  + jitter
  + packet_loss_penalty
  + hop_penalty
  + congestion_penalty

Example decisions:

30. Continuous Routing (Predict, Don't React)

Peers continuously measure: latency, jitter, packet loss, throughput, queue depth, hop count, peer availability. The network predicts "this P2P path is becoming unhealthy" and migrates before the listener hears an interruption.

31. Seamless Path Switching

The most important feature to design now. Switch at a known media timestamp:

C switches at a known media timestamp — no noticeable interruption.

32. Geographic Local Clusters

The mesh naturally forms regional clusters:

IFM automatically becomes a distributed broadcast system.

33. Path Manager Decision Engine

Explicit control + media plane split, and the engine:

text

IF direct_path.healthy
    use_direct()

ELSE IF alternate_peer.healthy
    switch_peer()

ELSE IF mesh_capacity_available
    establish_new_peer_path()

ELSE
    use_relay()

Plus continuous optimization:

text

IF direct_path.degrading
    prepare_alternate_path()

IF alternate_path.ready
    migrate_stream()

IF mesh.recovered
    migrate_back_to_p2p()

Relay usage is temporary and opportunistic, never an architectural dependency.

34. Wire Format Proposal (Media Packet)

This is the media extension of the existing packet model — see Packet Binary Format and Protocol Overview.

35. Telemetry & Observability

First-class metrics for the IFM dashboard (extend Observability):

end-to-end latency (capture → playback)
per-stage latency (capture, encode, packetize, network, relay, jitter, decode, output)
jitter, packet loss, queue depth
relay hops
encode/decode time
live-edge offset per listener

36. Implementation Phases

Phase 1 — Real-Time Transport Foundation

[x] 10 ms Opus frames as default for real-time media (AudioConfig::default(), low_latency() hardening)
[x] Datagram (unreliable) media channel over the transport layer (Transport::send_media/broadcast_media/on_media, MediaFrame)
[x] seq + timestamp on every media frame (wire format §34 — crates/protocol/src/media.rs)
[x] Adaptive jitter buffer: 20/30/50 ms targets + live-edge jump (crates/audio/src/jitter.rs)

Phase 2 — Forward-Only Relays

[x] Relay hot path: validate → route → forward (no decode/encode) — gossipsub Accept-forwards only verified packets on every traffic lane (crates/transport/src/libp2p_quic.rs, libp2p_browser.rs); media rides the voice lane at MEDIA priority
[x] Bounded per-listener media queues with drop-old behavior — CommandQueue/PriorityQueue per-class bounded queues with drop-old (crates/transport/src/priority.rs, spec §15); wired into the QUIC + browser transports
[~] Per-stage latency telemetry exposed to the dashboard — MediaStats/StreamStats in crates/core/src/media.rs; dashboard surfacing partial

Phase 3 — P2P-First Mesh

[x] Media plane wired into the real transports — send_media/broadcast_media/on_media implemented on QUIC + browser (voice-lane raw-bincode frames, MEDIA priority; crates/transport/src/{libp2p_quic,libp2p_browser}.rs), verified by media_frames_flow_between_quic_nodes_on_the_voice_lane + media_roundtrip_quic_nodes_correlates_waveform
[x] Direct-first listen with relay assist — the core's path scorer ranks direct peers vs pool relays from measured signal (Node::path_ranked, set_signal_quality); a relay is dialed only when the direct signal degrades
[ ] Listener forwarding with PEER_CAPACITY gating — explicit PEER_CAPACITY advertisement not yet in discovery records
[ ] Mesh with multiple upstreams + upstream switch on failure
[ ] Routing cost function + continuous path health measurement

Phase 4 — Automatic Fallback & Seamless Switching

[ ] Relay activation/deactivation from mesh-health thresholds (partially: per-node degraded-signal → relay assist is live)
[ ] Localized relay assistance (per region/peer, not whole mesh)
[ ] Seamless path migration at a known media timestamp
[ ] Migrate-back-to-P2P when the mesh recovers

Phase 5 — Advanced

[ ] Sliding redundancy packets
[ ] Selective FEC
[ ] Latency modes (LOW LATENCY / BALANCED / STABLE) with auto-selection
[ ] Capture-side low-latency backend tuning (desktop) — AudioWorklet capture exists in packages/ifm-station/src/audio/
[ ] Clock synchronization / station timeline (future)

37. Acceptance Criteria

End-to-end station → listener latency is < 100 ms under good conditions, trending toward 50–80 ms.
A lost media packet is never waited on — playback continues with PLC.
Relays never transcode media and forward each packet immediately.
Jitter buffer adapts (20 → 30 → 50 ms) and shrinks back when the network recovers.
Slow listeners are isolated: dropped audio for the slow listener never stalls the source or other listeners.
Direct P2P is the default path; a relay is engaged only when mesh health thresholds are breached, and the stream migrates back to P2P when the mesh recovers.
A peer with insufficient capacity is never selected as a forwarder.
Path switches complete at a known media timestamp with no audible interruption.
The dashboard shows per-stage latency, jitter, loss, queue depth, relay hops, and live-edge offset.

38. Implementation Tracker

Repository truth — updated whenever the implementation changes. The detailed, file-grounded plan with per-workstream status lives in Real-Time Media Refactor Plan.

Workstream	Scope	Status	Where
WS1	Media wire format (`MediaFrame`: stream id, seq u32, ts u64, codec, format, duration, FEC flags, payload)	✅ Done	`crates/protocol/src/media.rs`
WS2	10 ms pipeline: default config, codec frame-size-agnostic decode, arrival-time jitter (20/30/50 ms), audio-clock pop, live edge	✅ Done	`crates/audio/src/{config,codec,jitter}.rs`
WS3	Datagram media channel on `Transport`, `PriorityQueue`, in-memory/shared-mesh media paths	✅ Done	`crates/transport/src/{lib,memory,shared,registry,priority}.rs`
WS4	Core media pipeline: encode→`MediaFrame`→`broadcast_media`; inbound per-stream jitter+decode; media bypasses gossip; Node API (`send_media`, `send_media_f32`, `handle_media_received`, `media_poll`, `media_stats`)	✅ Done	`crates/core/src/media.rs`, `crates/core/src/node.rs`
WS3.5	Media plane wired into the REAL transports: `send_media`/`broadcast_media`/`on_media` on QUIC + browser (voice-lane raw-bincode frames, MEDIA priority queue), inbound media dispatch in the gossipsub handler, media frames proven to flow + decode over real QUIC nodes	✅ Done	`crates/transport/src/{libp2p_quic,libp2p_browser,priority}.rs`; tests `media_frames_flow_between_quic_nodes_on_the_voice_lane`, `media_roundtrip_quic_nodes_correlates_waveform`
WS5	Forward-only relay plane: gossipsub `Accept`-forwards only verified packets; `self_forwarding` transports skip the redundant node-level relay (bandwidth halved); bounded per-class priority queues with drop-old	✅ Done	`crates/transport/src/{libp2p_quic,libp2p_browser,priority}.rs`, `crates/core/src/node.rs`
WS6	SDK path intelligence: direct-first with quality-driven relay assist (core `path_ranked` scorer, `set_signal_quality`)	✅ Partial (`PEER_CAPACITY`/cost function future)	`crates/core/src/path.rs`, `crates/web-core/src/lib.rs`
WS7	Frontends: AudioWorklet capture (desktop), frame-size-aware voice decode	✅ Partial (capture in `packages/ifm-station/src/audio/`; playback still MediaSource/legacy)	`packages/ifm-station/src/audio/`, `crates/desktop/src/core.rs`
WS8	Telemetry: per-stage latency, jitter, live edge, loss	✅ Done (Rust pipeline)	`crates/core/src/media.rs` (`MediaStats`, `StreamStats`)
WS9	Invariants & docs to 10 ms standard	✅ Done	`docs/architecture/invariants.md`, this file, `docs/features/realtime-media-refactor-plan.md`

39. Open Questions

Clock synchronization: initial scope? (NTP-ish discipline vs. station timeline only.)
Forwarder incentives/caps: max concurrent forwarders per peer, max streams per forwarder?
FEC policy: which paths get redundancy, and how much bandwidth budget?
5 ms vs 10 ms: default until measured CPU/packet-rate cost on targets.
Relationship between this media plane and the existing reliable channel's backpressure rules (keep fully separate).

40. References

Audio Pipeline Architecture — modular pipeline, EncodedAudioFrame
Audio Pipeline & Codec — Opus frame cadence, AudioConfig::low_latency(), jitter buffer
Transport & Discovery — transport abstraction
Packet Binary Format — packet wire format
Protocol Overview — protocol spec
Architectural Invariants — boundaries this feature must respect
Observability — dashboard telemetry home
Relay — distributed forwarding infrastructure; relay fallback telemetry surface
Dashboard — internal system observability tool (where relay fallback telemetry surfaces)

Feature Spec — Real-Time Media Transport & P2P-First Mesh ​

1. Goal & Non-Goals ​

Goal ​

Non-Goals ​

1.1 Station-Owned Content State & Real-Time Stream Model ​

Conceptual Architecture ​

Desired Startup Flow ​

2. Terminology ​

3. Architecture Overview ​

Part A — Real-Time Media Transport ​

4. Latency Budget Model ​

5. Small Audio Frames ​

6. Datagram Transport — Reliability for Files, Latency for Audio ​

7. Adaptive, Small Jitter Buffer ​

8. No Transcoding at Relays ​

9. Relays Forward Immediately ​

10. No ACKs for Media ​

11. Sequence Number + Timestamp on Every Frame ​

12. Audio Clock, Not Network Clock ​

13. Separate Radio Latency from Network Latency ​

14. Latency Modes ​

15. Media Priority & No Backpressure Propagation ​

16. Live Edge (First-Class Concept) ​

17. Redundant Packets & Selective FEC ​

18. Capture-Side Latency ​

19. Hot Path: Ring Buffer, Not Async Queues ​

20. UI Out of the Critical Path ​

21. Clock Synchronization (Future) ​

Part B — P2P-First Mesh with Relay Fallback ​

22. Principle: Direct P2P Is the Default ​

23. Distribution Mesh (Listeners Forward) ​

24. Don't Blindly Forward — Advertise Capacity ​

25. Dynamic P2P Mesh, Not a Rigid Tree ​

26. Automatic Relay Activation ​

27. Localized Relay Assistance ​

28. Roles: Peer = Listener | Source | Forwarder ​

29. Routing Cost Function ​

30. Continuous Routing (Predict, Don't React) ​

31. Seamless Path Switching ​

32. Geographic Local Clusters ​

33. Path Manager Decision Engine ​

34. Wire Format Proposal (Media Packet) ​

35. Telemetry & Observability ​

36. Implementation Phases ​

Phase 1 — Real-Time Transport Foundation ​

Phase 2 — Forward-Only Relays ​

Phase 3 — P2P-First Mesh ​

Phase 4 — Automatic Fallback & Seamless Switching ​

Phase 5 — Advanced ​

37. Acceptance Criteria ​

38. Implementation Tracker ​

39. Open Questions ​

40. References ​

Feature Spec — Real-Time Media Transport & P2P-First Mesh

1. Goal & Non-Goals

Goal

Non-Goals

1.1 Station-Owned Content State & Real-Time Stream Model

Conceptual Architecture

Desired Startup Flow

2. Terminology

3. Architecture Overview

Part A — Real-Time Media Transport

4. Latency Budget Model

5. Small Audio Frames

6. Datagram Transport — Reliability for Files, Latency for Audio

7. Adaptive, Small Jitter Buffer

8. No Transcoding at Relays

9. Relays Forward Immediately

10. No ACKs for Media

11. Sequence Number + Timestamp on Every Frame

12. Audio Clock, Not Network Clock

13. Separate Radio Latency from Network Latency

14. Latency Modes

15. Media Priority & No Backpressure Propagation

16. Live Edge (First-Class Concept)

17. Redundant Packets & Selective FEC

18. Capture-Side Latency

19. Hot Path: Ring Buffer, Not Async Queues

20. UI Out of the Critical Path

21. Clock Synchronization (Future)

Part B — P2P-First Mesh with Relay Fallback

22. Principle: Direct P2P Is the Default

23. Distribution Mesh (Listeners Forward)

24. Don't Blindly Forward — Advertise Capacity

25. Dynamic P2P Mesh, Not a Rigid Tree

26. Automatic Relay Activation

27. Localized Relay Assistance

28. Roles: Peer = Listener | Source | Forwarder

29. Routing Cost Function

30. Continuous Routing (Predict, Don't React)

31. Seamless Path Switching

32. Geographic Local Clusters

33. Path Manager Decision Engine

34. Wire Format Proposal (Media Packet)

35. Telemetry & Observability

36. Implementation Phases

Phase 1 — Real-Time Transport Foundation

Phase 2 — Forward-Only Relays

Phase 3 — P2P-First Mesh

Phase 4 — Automatic Fallback & Seamless Switching

Phase 5 — Advanced

37. Acceptance Criteria

38. Implementation Tracker

39. Open Questions

40. References