Back to Learning

Encoding and Decoding in OTT and RTC: A Comprehensive Guide to AAC and H.264

Tencent RTC-Dev Team
Spt 26, 2024

In the rapidly evolving landscape of Over-the-Top (OTT) and Real-Time Communication (RTC) technologies, a thorough understanding of encoding and decoding processes is crucial. These processes form the backbone of efficient data transmission and storage in audio-video applications. This blog post will explore the concepts of encoding and decoding, with a particular focus on two widely used formats: AAC for audio and H.264 for video.

The Essence of Encoding and Decoding

Encoding is the process of converting information into a different data format according to specific rules, while decoding is the reverse process. The primary purpose of encoding in audio-video technology is data compression, which is essential for efficient transmission and storage.

To illustrate the importance of encoding, let's consider an example:

Imagine an uncompressed 720×1280 video at 25fps using the RGBA color format. Without any processing, each second of this video would require:

720 × 1280 × 25 × 2 bytes ≈ 44 Mbytes

This translates to a bit rate of 352 Mbps. Such high data volumes would put immense pressure on network transmission if left uncompressed.

Codecs: The Workhorses of Audio-Video Processing

In audio-video technology, the tools used for encoding and decoding are called codecs (coder-decoder). These are primarily divided into video codecs and audio codecs.

Codecs can be likened to compression tools like WinRAR or 7-Zip, but they are specialized for audio-video data. The key differences are:

  1. Specialization: Audio-video codecs are designed specifically for multimedia data.
  2. Efficiency: They typically achieve compression ratios of over 100:1.
  3. Quality Preservation: Despite high compression, they maintain visual and auditory quality that's often indistinguishable from the original to human perception.

AAC: Advanced Audio Coding

AAC, established in 1997, is a high-compression audio encoding algorithm. It's based on MPEG-2 audio coding technology and was later integrated into the MPEG-4 standard in 2000.

Key Features of AAC:

  1. High Compression Ratio: AAC offers superior compression compared to formats like AC3 or MP3.
  2. Quality: It can maintain CD-quality sound despite high compression.
  3. Variants: AAC has several variants to suit different needs, including AAC LC, AAC HE, and AAC HEv2.

AAC Types:

AAC can be further categorized into two types:

ADIF (Audio Data Interchange Format):

  • Used primarily for local file storage.
  • Decoding must start from a specified header.

ADTS (Audio Data Transport Stream):

  • Commonly used for internet applications.
  • Allows decoding to start from any point in the audio stream.
  • Contains synchronization words for easy identification of ADTS headers in the bitstream.

The structure of an ADTS audio stream looks like this:

... ADTS Header | AAC ES | ADTS Header | AAC ES ...

Where:

  • ADTS Header contains information necessary for decoding, such as stream identifier, bit rate, sample rate, channel count, and buffer size.
  • AAC ES (Elementary Stream) contains the actual encoded audio data.

For more detailed information on AAC, including its structure and implementation details, you can refer to this comprehensive document.

H.264: Advanced Video Coding

H.264, also known as AVC (Advanced Video Coding), is a high-performance video codec that has become a standard in the industry due to its excellent compression capabilities.

Key Features of H.264:

  1. High Compression Ratio: H.264 offers significantly higher compression ratios compared to earlier standards like MPEG-2 and MPEG-4.
  2. Efficiency: At the same image quality, H.264's compression ratio is more than twice that of MPEG-2 and 1.5 to 2 times that of MPEG-4.
  3. Size Reduction: On average, H.264 files are about 61% the size of equivalent MPEG-4 files and 36% the size of MPEG-2 files.

Frame Types in H.264:

H.264 defines three types of frames:

I-frames (Intra-coded frames):

  • Fully encoded pictures.
  • Do not depend on other frames for decoding.

P-frames (Predictive frames):

  • Encoded based on differences from previous I-frames or P-frames.
  • Smaller than I-frames.

B-frames (Bi-predictive frames):

  • Encoded based on differences from both previous and subsequent frames.
  • Typically the smallest in size.

Group of Pictures (GoP):

A Group of Pictures (GoP) is the sequence from one I-frame to the next. All frames within a GoP depend on the I-frame for decoding. If an I-frame is lost, the entire GoP becomes undecodable.

H.264 Structure:

H.264 is structured into two layers:

Video Coding Layer (VCL):

  • Contains the compressed video data.

Network Abstraction Layer (NAL):

  • Packages the video data for transmission or storage.
  • VCL data is encapsulated in NAL Units (NALUs) before transmission or storage.

The basic structure of a NAL Unit (NALU) is:

... NALU header | RBSP | NALU header | RBSP ...

Where:

  • NALU header contains information about the type of data in the RBSP.
  • RBSP (Raw Byte Sequence Payload) contains the actual video data.

For a more in-depth explanation of H.264/AVC, including its encoding process, frame types, and structure, you can refer to this detailed document.

Conclusion

Understanding encoding and decoding processes is crucial for anyone working with OTT and RTC technologies. AAC and H.264 are currently the most widely supported formats for audio and video respectively, offering excellent compression ratios while maintaining high quality.

As the field of audio-video technology continues to evolve, new codecs and standards are constantly being developed. For instance, H.265 (HEVC) offers even better compression than H.264, and AV1 is emerging as a royalty-free alternative. On the audio side, formats like Opus are gaining popularity for their flexibility and efficiency.

Staying updated with the latest codec technologies will be essential for developers and engineers working on OTT and RTC applications. By leveraging these advanced encoding and decoding techniques, we can continue to push the boundaries of what's possible in digital media transmission and storage, enabling more immersive and high-quality audio-video experiences for users around the world.