Cloud Box (UC-CB)

A "Cloud Box" can be understood as a cloud-based private room, similar to the private room we have in our real lives. Friends can come together in these rooms to chat, watch movies, and watching sports events. In sports broadcasts, fans can participate in predictions and discussions about ongoing matches through the "Cloud Box" feature, enabling them to share their insights and forecasts in real-time. The Cloud Box feature bridges the gap between individuals, enhancing their immersive experience and interaction.

In this scenario, participants can engage in real-time interactions through WebRTC.

In order to provide a more immersive user experience, spatial audio codecs similar to Dolby Atmos is utilized in WebRTC when watching Ultra HD sports events.

Real-time Live Commerce (UC-RLC)

Real-time Live Commerce, also known as real-time livestream shopping, is an innovative retail trend that merges real-time streaming with e-commerce. In Live Commerce sessions, businesses and brands showcase their products or services on live online platforms, such as social media and e-commerce websites, while simultaneously engaging with the audience in direct interactions. During these live sessions, hosts or salespeople introduce product features, demonstrate usage, address viewer inquiries, and offer exclusive deals or limited-time promotions.

Viewers can actively participate by using real-time commenting or chat features to interact with the hosts. They can ask questions, express their interest in purchasing, and even share their own product experiences with fellow viewers. When a viewer shows interest in a particular product, they can instantly make a purchase using dedicated links or designated purchasing methods.

Occasionally, invited guests can participate in live interactions with the hosts, aiding in sales through WebRTC-powered video/audio chat.

Metaverse Convetion Center (UC-MCC)

The Metaverse Convention Center is a type of cloud game that incorporates advanced technologies such as WebRTC for interactive communication, 3D rendering, and AI-generated avatars. It transforms traditional audio-video conferencing into virtual, gamified, and interactive experiences for remote meetings, office use, and events.

Users have the ability to create their own virtual characters, which can range from cartoon-style to realistic representations, and select from a variety of meeting rooms to align with their preferences.

In this scenario, all the resources are rendered on the cloud and then streamed to the browser using WebRTC. Any control commands in the browser are executed based on WebRTC.

Generic Fragmentation and Reassembly Mechanism for Audio Transmission

This requirement is for UC-CB.

During audio transmission via WebRTC, RTP packets encapsulate each audio frame. However, if the frame size exceeds the network's MTU limit (typically 1500 bytes), packet transmission can fail. To overcome this, packet fragmentation and reassembly are needed. Fragmentation occurs at the sender, and reassembly happens at the receiver.

WebRTC handles fragmentation and reassembly for video transmission, but not for audio. It is the responsibility of the application or developer to handle fragmentation and reassembly of audio packets when using WebRTC for audio transmission.

A proposed fragmentation and reassembly workflow can be outlined as follows:

Statistics for Audio and Video Freeze

This requirement is for UC-CB, UC-MCC, and UC-RLC.

In the W3C Identifiers for WebRTC's Statistics API spec, the logic for marking a video freeze, is defined as follows(see freezeCount):

1. Calculate the linear average of the past 30 rendering frame durations and label it as avg_frame_duration_ms.
2. Calculate the interval between the current two frames. If this value is greater than Max(3 * avg_frame_duration_ms, avg_frame_duration_ms + 150), it is considered as a freeze.

Meanwhile, The W3C spec does not define metrics for audio-related freeze , although the underlying WebRTC code has already implemented them with the fields totalInterruptionDuration and interruptionCount. The metrics can be exposed to developers.

Receiving sounds from other devices when in background

This requirement is for UC-CB.

An iOS device that has the microphone enabled and then switches to the background is unable to hear sounds from other devices when some device unmutes.

This issue can be reproduced as follows:

• Devices A and B join a WebRTC conference, B is an iOS device, A is any device
• Device A activates the microphone.
• Device B goes to the background (e.g., Home button).
• Device A's voice is audible to B.
• Device A mutes its microphone.
• Device A unmutes and speaks.
• Device B can't hear A, even if B goes back to the foreground.

Fixed GOP encode interval

This requirement is for UC-RLC.

In the live-streaming scene, the encoder needs to output video frames at a fixed GOP interval (such as GOP=2 seconds). We hope that an API can be provided to set GOP interval.

Enable RTCP-XR (RRTR/DLRR) at recv-only mode

This requirement is for UC-RLC.

In the live-streaming scene, the player endpoint is generally in recv-only mode. If the player endpoint wants to calculate RTT (round-trip time), one possible approach is to use RTCP-XR [RFC3611]. Currently, WebRTC already supports RTCP-XR, but the offer generated by browser doesn't contain it. We hope the browser adds an API to enable the RTCP-XR.

Related issue