Next-Gen Media SDK Solution Design of Tencent RTC

1. Immersive Convergence

/1.1 Higher definition

According to statistics from Tencent Cloud, the average bitrate of internet streaming media played on PCs, tablets, mobile phones, and other terminals has been increasing since H1 2018. As people require higher definition, the compression rate has also improved as the bitrate increases. This is due to the developments from H.264 and H.265 to the recent H.266 with its over 100 technical proposals, which delivers 50% higher compression rates than H.265.

/1.2 Stronger immersiveness

Advances have been made in the immersive experience of many applications, such as 3D audio effects, 3D guides, 3D modeling, AR/VR games, and multi-angle sports viewing.

/1.3 Enhanced interaction

Real-time interaction is stronger. In particular, face point cloud data is collected from a mobile phone and then sent back to the audience member's device from the cloud. TRTC provides real-time audio and video, helping games create extraordinary gaming interaction experiences.

/1.4 Lower latency

Latency has achieved the greatest improvement. A few years ago, the latency on webpages was counted in seconds, but now it is measured in milliseconds, low enough for users to sing duets together in live rooms. We provide a global end-to-end latency of less than 300 ms.

/1.5 Four elements of the all-true internet

The all-true internet features a higher definition, enhanced interaction, stronger immersiveness, and lower latency. But this entails challenges and unavoidable difficulties in the cloud and on the terminal.

2. Technical Challenges

Let's take a look at the challenges and how to overcome them.

/2.1 Challenge 1: RT-Cube™ architecture design

It's hard to coordinate internal modules no matter what you are working on, from an operating system to something smaller like an SDK. An SDK has many modules. The image shows a simplified version of the SDK module architecture, but you can still imagine the large number of modules that are actually involved. The bottom-left corner shows audio/video engine modules, the bottom-right corner TIM modules, and the top TUI components. When multiple modules are working together, they tend to scramble for CPU resources and encounter other conflicts.

The image above depicts the architecture design of the audio/video engine in RT-Cube™, which consists of many core modules with their respective submodules. Between those modules, there are much data communication and control logic. When the system runs stably, everything works well in unison. However, if the CPU frequency is reduced or the memory becomes insufficient, competition between modules will soon cause the entire system to crash. Therefore, a central control module is adapted to monitor and coordinate the modules in real-time and take intervention measures when necessary to better coordinate them and prevent an avalanche.

/2.2 Challenge 2: RT-Cube™ version management