Enhancing Real-Time Audio and Video Quality with Tencent's Advanced Encoding Technologies
Looking back at the year 2023, Tencent Multimedia Lab and Tencent RTC team continued to cooperate in real-time audio and video scenarios. From the perspectives of product link and technical foundation, they conducted further in-depth optimization on video quality and compression performance, continuously polishing various underlying technologies including the self-developed H.264 real-time encoder - O264rt. This article will unveil the optimization details behind the product from a technical perspective.
In the field of audio and video, people are always pursuing higher clarity, more extreme experiences, and more vivid effects in order to obtain a more immersive feeling. Therefore, explorations starting from these points - such as 8K, VR, HDR, etc., have undoubtedly sparked a lot of discussion and research. More realistic images can greatly enhance the user experience, but this is not an easy task in the field of real-time audio and video.
Challenges in Real-Time Audio and Video
An important distinction between real-time audio and video and other live scenarios is extremely low latency. This provides users with a more barrier-free remote interaction and communication experience. As a result, users can enjoy a more extreme experience in the following scenarios: such as more efficient video conferences, more immersive live effects, and more comfortable video call experiences. However, extremely low latency also implies the need for efficient encoding on the client side, and very strict requirements for codec delay, CPU performance consumption, and jitter resistance.
Figure 1: Latency illustration for different audio and video scenarios
Generally speaking, the technical requirements for improving clarity mean that higher resolution images and better picture quality need to be processed. This leads to higher encoding complexity and greater bandwidth occupancy, which can easily cause other issues that affect the user experience, such as increased stuttering and excessive terminal CPU performance consumption. Taking Tencent RTC user data as an example, the top device distribution for terminal devices is still dominated by mid-range devices from three or four years ago. In this situation, hastily increasing encoding complexity and bandwidth to obtain high-definition images will inevitably reduce the actual QoE
