Understanding HTTP: The Backbone of Web Communication

Tencent RTC-Dev Team

Spt 26, 2024

HTTP (Hypertext Transfer Protocol) is the foundation of data communication on the World Wide Web. As a crucial component in Over-the-Top (OTT) and Real-Time Communication (RTC) technologies, understanding HTTP is essential for developers and engineers working in these fields. This blog post will delve into the intricacies of HTTP, its structure, and its application in streaming technologies like HLS (HTTP Live Streaming).

What is HTTP?

HTTP is an application-layer protocol built on top of TCP. It's the most widely used protocol on the internet, facilitating the transfer of hypertext documents, images, videos, and other multimedia content. HTTPS (HTTP Secure) is a secure version of HTTP that uses SSL/TLS encryption for data transfer.

HTTP URL Structure

HTTP resources are identified using Uniform Resource Locators (URLs). A typical HTTP URL has the following structure:

http://host[:port]/[path/]filename[?param1=value1&param2=value2][#fragment]

http://: Indicates the use of the HTTP protocol (or https:// for HTTPS)
host: The domain name or IP address of the server
port: The port number (default is 80 for HTTP, 443 for HTTPS)
path: The path to the resource on the server
filename: The name of the resource
param, value: Query parameters
fragment: A specific part of the resource, often called an anchor

HTTP Communication

HTTP communication involves requests from clients (User Agents) and responses from servers (Origin Servers). Intermediaries like proxy servers, gateways, or tunnels may exist between the client and server.

HTTP Request Structure

An HTTP request consists of three parts:

Request Line
Message Headers
Request Body

Example of an HTTP request:

POST /index.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
Accept: text/html,application/xhtml+xml, application/xml; q=0.9,*/*; q=0.8
Accept-Language: zh-cn,zh;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Referer: http://localhost/
Content-Length: 26
Content-Type: application/x-www-form-urlencoded

username=abc&password=1234

HTTP Response Structure

An HTTP response also consists of three parts:

Status Line
Message Headers
Response Body

Example of an HTTP response:

HTTP/1.1 200 OK
Date: Sun, 17 Mar 2013 08:12:54 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 4393
Keep-Alive: timeout=5, max=100
Connection: keep-alive
Content-Type: text/html; charset=utf-8

<html>
<head>
<title>HTTP Response Example</title>
</head>
<body>
Hello HTTP!
</body>
</html>

HTTP Status Codes

HTTP status codes are three-digit numbers returned by a server in response to a client's request. They are grouped into five classes:

Status Code	Description	Meaning
1XX	Informational	Request received, continuing process
2XX	Successful	The action was successfully received, understood, and accepted
3XX	Redirection	Further action needs to be taken in order to complete the request
4XX	Client Error	The request contains bad syntax or cannot be fulfilled
5XX	Server Error	The server failed to fulfill an apparently valid request

Here are some common status codes:

Status Code	Text	Description
200	OK	Request successful
206	Partial Content	Partial content delivered
301	Moved Permanently	Requested URL has been permanently moved
302	Found	Temporary URL redirection
304	Not Modified	Resource not modified since last request
400	Bad Request	Server cannot process due to client error
403	Forbidden	Server refuses to authorize the request
404	Not Found	Requested resource could not be found
500	Internal Server Error	Generic error message when server fails
502	Bad Gateway	Server received an invalid response from upstream server
503	Service Unavailable	Server is temporarily unable to handle the request

For more information, you can refer to

HTTP in Streaming: HLS (HTTP Live Streaming)

HLS is a streaming protocol developed by Apple that uses HTTP for media delivery. It works by breaking the overall stream into a sequence of small HTTP-based file downloads.

HLS Workflow

The client first retrieves a master playlist file (M3U8).
The M3U8 file contains URLs of media segment files (TS files).
The client then downloads these TS files via HTTP.
For live streaming, the client continuously updates the M3U8 file to get new TS file URLs.

M3U8 File Structure

M3U8 files are text-based and consist of a series of tags. Here's an example:

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:10,
http://media.example.com/segment0.ts
#EXTINF:10,
http://media.example.com/segment1.ts
#EXTINF:10,
http://media.example.com/segment2.ts
#EXT-X-ENDLIST

Here's a table of common M3U8 tags:

Tag	Meaning
#EXTM3U	Must be the first line of every M3U8 file
#EXTINF	Specifies the duration of the TS file
#EXT-X-TARGETDURATION	Specifies the maximum duration of any TS file
#EXT-X-MEDIA-SEQUENCE	Specifies the sequence number for the first TS file
#EXT-X-KEY	Specifies the encryption method for TS files
#EXT-X-PROGRAM-DATE-TIME	Specifies the date/time of the first sample in a TS file
#EXT-X-ALLOW-CACHE	Specifies whether the client may cache the downloaded media files
#EXT-X-PLAYLIST-TYPE	Provides information about the playlist type (EVENT or VOD)
#EXT-X-ENDLIST	Indicates the end of the M3U8 file

HTTP Dynamic Live Streaming (HDL)

HTTP Dynamic Live Streaming (HDL), more commonly known as HTTP-FLV, is a streaming media protocol developed by Adobe. It leverages HTTP for the online transmission of FLV (Flash Video) files, making it an important technology in the realm of web-based streaming.

HDL's working principle is similar to HLS (HTTP Live Streaming). Both protocols use HTTP to retrieve relevant files and subsequently fetch the corresponding audio and video stream data. However, where HLS uses M3U8 and TS files, HDL utilizes SWF (Shockwave Flash) and FLV files. These file formats, designed by Adobe, traditionally required Flash technology support, which limited their direct use in HTML5 environments. (In browsers supporting MSE, libraries like flv.js can be used to enable support, though compatibility issues should be considered.)

SWF (Shockwave Flash) is a proprietary format developed by Macromedia (later acquired by Adobe) for their Flash animation design software. It has been widely used in web design and animation production. The structure of an SWF file is as follows:

After the Header, there are several tagged data blocks. The Tag format is consistent with FLV encapsulation. During online playback, typically only one SWF file needs to be retrieved.

The HDL playback process follows these steps:

Retrieve the SWF file via HTTP
Use the SWF file to obtain FLV files
Decapsulate, decode, and display the FLV file content
Continuously fetch and process FLV file data, decoding and displaying it until the EndTag of the SWF file is reached

This process demonstrates how HDL utilizes HTTP not just for initial file retrieval, but for the entire streaming process, making it an integral part of the protocol's functionality.

For more detailed information about SWF files, you can refer to the Adobe Developer Center: http://www.adobe.com/devnet/swf.html

Conclusion

HTTP forms the backbone of web communication and plays a crucial role in modern streaming technologies like HLS. Understanding its structure, status codes, and application in streaming protocols is essential for developers working in OTT and RTC fields. As the web continues to evolve, HTTP remains a fundamental technology, adapting to new requirements and use cases in the ever-changing landscape of digital communication.