Custom Audio Capturing and Playback
This document describes how to use the TRTC SDK to implement custom audio capturing and rendering.
Custom Audio Capturing
The custom audio capturing feature of the TRTC SDK can be used in two steps: enabling the feature and sending audio frames to the SDK. For detailed directions of specific APIs, see below. We also provide API examples for different platforms:
Enabling custom audio capturing
To enable the custom audio capturing feature of the TRTC SDK, you need to call the
enableCustomAudioCapture API of TRTCCloud. Below is the sample code:TRTCCloud mTRTCCloud = TRTCCloud.shareInstance();mTRTCCloud.enableCustomAudioCapture(true);
self.trtcCloud = [TRTCCloud sharedInstance];[self.trtcCloud enableCustomAudioCapture:YES];
liteav::ITRTCCloud* trtc_cloud = liteav::ITRTCCloud::getTRTCShareInstance();trtc_cloud->enableCustomAudioCapture(true);
Sending custom audio frames
You can use the
sendCustomAudioData API of TRTCCloud to populate the TRTC SDK with your own audio data. Below is the sample code:TRTCCloudDef.TRTCAudioFrame trtcAudioFrame = new TRTCCloudDef.TRTCAudioFrame();trtcAudioFrame.data = data;trtcAudioFrame.sampleRate = sampleRate;trtcAudioFrame.channel = channel;trtcAudioFrame.timestamp = timestamp;mTRTCCloud.sendCustomAudioData(trtcAudioFrame);
TRTCAudioFrame *audioFrame = [[TRTCAudioFrame alloc] init];audioFrame.channels = audioChannels;audioFrame.sampleRate = audioSampleRate;audioFrame.data = pcmData;[self.trtcCloud sendCustomAudioData:audioFrame];
liteav::TRTCAudioFrame frame;frame.audioFormat = liteav::TRTCAudioFrameFormatPCM;frame.length = buffer_size;frame.data = array.data();frame.sampleRate = 48000;frame.channel = 1;getTRTCShareInstance()->sendCustomAudioData(&frame);
notice
Using
sendCustomAudioData may cause AEC to fail.Getting Raw Audio Data
The audio module is a highly complex module, and the TRTC SDK needs to strictly control the capturing and playback logic of audio devices. In some cases, to get the audio data of a remote user or audio captured by the local mic, you can use the APIs of
TRTCCloud for different platforms. We also provide API examples for those platforms: Setting audio callback
mTRTCCloud.setAudioFrameListener(new TRTCCloudListener.TRTCAudioFrameListener() {@Overridepublic void onCapturedRawAudioFrame(TRTCCloudDef.TRTCAudioFrame trtcAudioFrame) {}@Overridepublic void onLocalProcessedAudioFrame(TRTCCloudDef.TRTCAudioFrame trtcAudioFrame) {}@Overridepublic void onRemoteUserAudioFrame(TRTCCloudDef.TRTCAudioFrame trtcAudioFrame, String s) {}@Overridepublic void onMixedPlayAudioFrame(TRTCCloudDef.TRTCAudioFrame trtcAudioFrame) {}@Overridepublic void onMixedAllAudioFrame(TRTCCloudDef.TRTCAudioFrame trtcAudioFrame) {// For more information, see the custom rendering tool class `com.tencent.trtc.mediashare.helper.CustomFrameRender` in `TRTC-API-Example`}});
[self.trtcCloud setAudioFrameDelegate:self];// MARK: - TRTCAudioFrameDelegate- (void)onCapturedRawAudioFrame:(TRTCAudioFrame *)frame {NSLog(@"onCapturedRawAudioFrame");}- (void)onLocalProcessedAudioFrame:(TRTCAudioFrame *)frame {NSLog(@"onLocalProcessedAudioFrame");}- (void)onRemoteUserAudioFrame:(TRTCAudioFrame *)frame userId:(NSString *)userId {NSLog(@"onRemoteUserAudioFrame");}- (void)onMixedPlayAudioFrame:(TRTCAudioFrame *)frame {NSLog(@"onMixedPlayAudioFrame");}- (void)onMixedAllAudioFrame:(TRTCAudioFrame *)frame {NSLog(@"onMixedAllAudioFrame");}
// Set custom audio data callbackliteav::ITRTCCloud* trtc_cloud = liteav::ITRTCCloud::getTRTCShareInstance();trtc_cloud->setAudioFrameCallback(callback)// Callback APIs for custom audiovirtual void onCapturedRawAudioFrame(TRTCAudioFrame* frame) {}virtual void onLocalProcessedAudioFrame(TRTCAudioFrame* frame) {}virtual void onPlayAudioFrame(TRTCAudioFrame* frame, const char* userId) {}virtual void onMixedPlayAudioFrame(TRTCAudioFrame* frame) {}
notice
Do not perform time-consuming operations with any of the above callback functions. We recommend that you copy the data to another thread for processing to avoid AEC failure and choppy audio.
The data called back by the above callback functions can only be read and copied. Modifying the data may lead to unexpected results.