please select
RTC Engine
  • Overview
  • Web
    • Run Sample Demo
    • Integration
    • Basic Features
      • Screen Sharing
      • Live Streaming
      • Media Device
      • Audio Volume
      • Set Encoding Profile
      • Detect Network Quality
      • Detect Capabilities
    • Advance Features
      • Enable AI Denoiser
      • Enable Audio Mixer
      • Enable Watermark
      • Enable Virtual Background
      • Enable Beauty and Effects
      • Data Messages
      • Custom Capturing and Rendering
    • API List
    • Released Notes
    • Supported Platforms
    • Web FAQs
      • Optimize Multi-Person Video Calls
      • Handle Autoplay Restriction
      • Handle Firewall Restriction
      • Others
  • Android
    • Run Sample Code
    • Integration
    • Basic Features
      • Sensing Network Quality
      • Enabling Screen Sharing
      • Setting Video Quality
      • Rotating Videos
    • Advanced Features
      • Testing Newwork Quality
      • Custom Capturing and Rendering
      • Custom Audio Capturing and Playback
    • Client APIs
      • Overview
      • TRTCCloud
      • TRTCStatistics
      • TRTCCloudListener
      • TXAudioEffectManager
      • TXBeautyManager
      • TXDeviceManager
      • Type Definition
      • Deprecated Interface
      • Error Codes
    • Solution
      • Real-Time Chorus (TUIKaraoke)
        • Quick Integration
        • Implementation Steps
        • Song Synchronization
        • Lyric Synchronization
        • Vocal Synchronization
        • Mixing Stream Solution
        • TRTCKaraoke APIs
        • FAQs
    • Release Notes
  • iOS
    • Run Sample Code
    • Integration
    • Basic Features
      • Sensing Network Quality
      • Enabling Screen Sharing
      • Setting Video Quality
      • Rotating Videos
    • Advanced Features
      • Testing Network Quality
      • Custom Capturing and Rendering
      • Custom Audio Capturing and Playback
    • Client APIs
      • Overview
      • TRTCCloud
      • TRTCCloudDelegate
      • TRTCStatistics
      • TXAudioEffectManager
      • TXBeautyManager
      • TXDeviceManager
      • Type Definition
      • Deprecated Interface
      • ErrorCode
    • Solution
      • Quick Integration
      • Implementation Steps
      • Song Synchronization
      • Lyric Synchronization
      • Vocal Synchronization
      • Mixing Stream Solution
      • TRTCKaraoke APIs
      • FAQs
    • Release Notes
  • macOS
    • Run Sample Code
    • Integration
    • Basic Features
      • Sensing Network Quality
      • Enabling Screen Sharing
      • Sharing Computer Audio
      • Setting Video Quality
      • Rotating Videos
    • Advanced Features
      • Testing Hardware Devices
      • Testing Network Quality
      • Custom Capturing and Rendering
      • Custom Audio Capturing and Playback
    • Client APIs
      • Overview
      • TRTCCloud
      • TRTCCloudDelegate
      • TRTCStatistics
      • TXAudioEffectManager
      • TXBeautyManager
      • TXDeviceManager
      • Type Definition
      • Deprecated Interface
      • ErrorCode
      • Release Notes
    • Release Notes
  • Windows
    • Run Sample Code
    • Integration
    • Basic Features
      • Sensing Network Quality
      • Enabling Screen Sharing
      • Setting Video Quality
      • Rotating Videos
    • Advanced Features
      • Testing Hardware Devices
      • Testing Network Quality
      • Custom Capturing and Rendering
      • Custom Audio Capturing and Playback
    • Client APIs
      • Overview
      • ITRTCCloud
      • ITRTCStatistics
      • TRTCCloudCallback
      • ITXAudioEffectManager
      • ITXDeviceManager
      • Type Definition
      • Deprecated Interface
      • Error Codes
    • Release Notes
  • Electron
    • Integration
      • 1.API Examples
      • 2.Importing the SDK
      • 3.Entering a Room
      • 4.Subscribing to Audio/Video Streams
      • 5.Publish Audio/Video Streams
      • 6.Exiting a Room
      • 7.Sensing Network Quality
      • 8.Enabling Screen Sharing
      • 9.Sharing Computer Audio
      • 10.Setting Video Quality
      • 11.Rotating Videos
    • Client APIs
      • Overview
      • Error Codes
  • Flutter
    • Integration
      • 1.API Examples
      • 2.Importing the SDK
      • 3.Entering a Room
      • 4.Subscribing to Audio/Video Streams
      • 5.Publish Audio/Video Streams
      • 6.Exiting a Room
      • 7.Sensing Network Quality
      • 8.Enabling Screen Sharing
      • 9.Sharing Computer Audio
      • 10.Setting Video Quality
      • 11.Rotating Videos
    • Client APIs
      • Overview
      • Error Codes
  • Unity
    • Integration
      • 1.API Examples
      • 2Importing the SDK
    • Client APIs
      • Overview
      • Error Codes
  • Qt
    • Integration
      • 1.Importing the SDK
  • Overview
    • Overview
  • Concepts
  • Features
  • Performance Statistics
  • Pricing
    • Free Minutes
    • RTC-Engine Monthly Packages
    • Billing Explanation for Subscription Package Duration
    • Pay-as-you-go
      • Billing of Audio and Video Duration
      • Billing of On-Cloud Recording and Recording Delivery
      • Billing of MixTranscoding and Relay to CDN
  • FAQs
    • FAQs for Beginners
    • Migration Guide
      • Twilio Video to Tencent RTC
      • Billing
      • Features
      • UserSig
      • Firewall Restrictions
      • How to Downsize Installation Package
      • TRTCCalling for Web
      • Audio and Video Quality
      • Others
RTC Engine

Lyric Synchronization

Implementation process

In the lyrics synchronization scheme, the actions of three different roles are as follows:
Lead Singer
Chorus
Audience
NTP time calibration
Turn on black frame compensation
Send SEI message
Local lyrics synchronization
Update lyrics widget
NTP time calibration
Local lyrics synchronization
Update lyrics widget
NTP time calibration
Receive SEI message
Update lyrics widget
The lead singer and chorus update the lyrics progress locally according to the synchronized song playback progress; the audience needs to receive the SEI message sent by the lead singer, which contains the latest lyrics progress, to update the local lyrics progress.




Timing diagram





The lyrics synchronization timing can be mainly divided into three parts: NTP time calibration, turning on black frame compensation, and local and remote lyrics synchronization. The code implementation of NTP time calibration has been given in the song synchronization, and the specific code implementation for the latter two parts will be provided below.

Key code implementation

1. Turn on black frame compensation

// In pure audio mode, the main instance (vocal instance) needs to turn on black frame compensation to carry the SEI message.
mTRTCCloud.callExperimentalAPI("{\"api\":\"enableBlackStream\",\"params\": {\"enable\":true}}");
Note:
The experimental interface enableBlackStream needs to be called after entering the room.
On Android, the value type of the enable parameter is boolean, and on iOS, it is integer.
The receiver needs to call startRemoteView(userId, null) when onUserVideoAvailable(userId, true) is received.

2. Send song progress through SEI message

mAudioEffectManager.setMusicObserver(mCurPlayMusicId, new TXAudioEffectManager.TXMusicPlayObserver() {
@Override
public void onPlayProgress(int id, long curPtsMS, long durationMS) {
JSONObject jsonObject = new JSONObject();
// Current NTP time
long ntpTime = TXLiveBase.getNetworkTimestamp();
jsonObject.put("bgmProgressTime", curTime);
jsonObject.put("ntpTime", ntpTime);
jsonObject.put("musicId", musicId);
jsonObject.put("duration", duration);
jsonObject.toString().getBytes();
mTRTCCloud.sendSEIMsg(jsonObject.toString().getBytes(), 1);
}
}

Note:
The frequency of the lead singer sending SEI messages is determined by the frequency of background music playback event callbacks, usually 200ms;
The reason for not directly using CMD messages to send song progress: The signaling transmitted by the SEI channel can be transmitted to the live CDN along with the video frames, providing better compatibility for the audience pulling the CDN stream.

3. Local and remote lyrics synchronization

// Local lyrics synchronization
mAudioEffectManager.setMusicObserver(mCurPlayMusicId, new TXAudioEffectManager.TXMusicPlayObserver() {
@Override
public void onPlayProgress(int id, long curPtsMS, long durationMS) {
...
// TODO Update lyrics widget logic:
// Determine whether to seek the lyrics widget based on the latest progress and local lyrics progress error
...
}
}

// Remote lyrics synchronization
@Override
public void onRecvSEIMsg(String userId, byte[] data) {
String result = new String(data);
JSONObject jsonObject = new JSONObject(result);
long bgmProgressTime = jsonObject.getLong("bgmProgressTime");
long ntpTime = jsonObject.getLong("ntpTime");
String musicId = jsonObject.getString("musicId");
long duration = jsonObject.getLong("duration");
...
// TODO Update lyrics widget logic:
// If you reuse the TUIKaraoke component's lyrics widget,
//please refer to the code logic of the TUIKaraoke LyricsView section to synchronize the lyrics widget progress.
...
}
Note:
If you reuse the TUIKaraoke component's lyrics widget, please refer to the code logic of the TUIKaraoke LyricsView section to synchronize the lyrics widget progress.