Building a Web-based Audio and Video Call Engine with WebAssembly
As web technology advances and audio and video call demands evolve, it's essential to explore and implement the value of new web technologies in practical applications, bringing us greater benefits. We will introduce the value and advantages of WebAssembly, WebCodecs, WebTransport, and other technologies in the audio and video industry from practical experience.
We will introduce the topic from four aspects: background, WebAssembly engine, WebAssembly implementation, and issues and prospects.
Background
With the upgrade of network infrastructure, the iteration of audio and video transmission technology, and the change of audio and video consumption habits, multimedia technology has developed from the initial on-demand and live broadcast to ultra-low-latency live broadcast and real-time audio and video interaction. WebRTC has laid the technical foundation in the development process.
This is a schematic diagram of WebRTC's architecture. WebRTC provides a rich set of Web APIs. Audio and video capture, encoding and decoding, preprocessing and postprocessing, transmission, and rendering are all made possible by WebRTC. When developing web-based audio and video applications, the use of WebRTC reduces development difficulty and cost.
WebRTC also has some shortcomings. Firstly, WebRTC does not allow custom codecs. Secondly, WebRTC cannot reuse existing service frameworks and optimization capabilities. Lastly, WebRTC has a low level of customization.
Are there any new web technologies that can replace WebRTC and solve its problems? The following are some new technologies that can be used.
WebAssembly is a new type of code that runs in modern browsers and provides new performance features and effects. Its design goals are fast, efficient, portable, readable, debuggable, secure, and non-disruptive to the network. WebAssembly can solve performance issues of JavaScript in complex scenarios, such as 3D games, computer vision, image and video editing, and many other areas that require native performance. Using WebAssembly in some scenarios that previously used JavaScript can significantly improve efficiency. Thanks to WebAssembly's small size, it can also solve the problem of high costs for downloading and parsing JavaScript applications.
WebCodecs provides developers with a way to use existing media components in browsers, not only solving the low-latency problem of codecs but also providing more flexible configuration interfaces. The image on the right shows the configuration options for a video encoder, with many configurable options provided, such as choosing between software and hardware encoding, VBR/CBR selection, quality priority/low latency priority, etc. When using H264 encoding with HighProfile, WebCodes can easily support it, providing great convenience at the encoding level.
WebTransport is a new pluggable communication protocol that supports reliable and unreliable transmission. It can be used in applications that require reliable transmission. WebTransport's goal is to be faster, more efficient, secure, and low-latency. It can solve the problem of connection migration. WebTransport has flexible congestion control and better weak network capabilities. When dealing with head-of-line blocking, more flexible transmission methods can be used.
WebAssembly Engine
New technologies and architectures aim to provide users with more possibilities. Custom codecs, custom transmission methods, custom data encryption, custom audio and video preprocessing and postprocessing, and custom QoS operations have all been implemented in practical projects.
