Docs Overview/
Converting Audio to Expression/

Converting Audio to Expression

Overview

This capability processes audio data and outputs blendshape data that meets the standards of Apple ARKit. For details, see ARFaceAnchor. You can pass the data to Unity to drive your model or use the data to implement other features.

Integration

Method 1: Integrate the Tencent Effect SDK

The capability of converting audio to expressions is built into the Tencent Effect SDK, so to use the capability, you can integrate the Tencent Effect SDK.
1. Download the complete edition of the Tencent Effect SDK.
2. Follow the directions in Integrating Tencent Effect SDK to integrate the SDK.

Method 2: Integrate the Audio-to-Expression SDK

If you do not need other Tencent Effect capabilities, you can integrate the Audio-to-Expression SDK, the AAR file of which is about 6 MB. You can contact us to get the SDK.

Directions

1. Set the license. For detailed directions, see Integrating Tencent Effect SDK - Step 1. Authenticate.
2. Configure the model files: Copy the model files from assets to a private directory of your application, such as context.getFilesDir() + "/my_models_dir/audio2exp". Then, call the init(String modelPath) API of Audio2ExpApi, passing in context.getFilesDir() + "/my_models_dir". You can find the model files at the following location of the SDK package:


APIs

API
Description
public int Audio2ExpApi.init(String modelPath);
Initializes the SDK. You need to pass in the path of the model files. 0 indicates the initialization is successful.
public float[] Audio2ExpApi.parseAudio(float[] inputData);
The input is audio, which must be one-channel and have a sample rate of 16 Kbps and an array length of 267 (267 sampling points). The output is a float array with 52 elements, which correspond to 52 blendshapes. The value range of each element is 0-1, and their sequence is specified by Apple{"eyeBlinkLeft","eyeLookDownLeft","eyeLookInLeft","eyeLookOutLeft","eyeLookUpLeft","eyeSquintLeft","eyeWideLeft","eyeBlinkRight","eyeLookDownRight","eyeLookInRight","eyeLookOutRight","eyeLookUpRight","eyeSquintRight","eyeWideRight","jawForward","jawLeft","jawRight","jawOpen","mouthClose","mouthFunnel","mouthPucker","mouthRight","mouthLeft","mouthSmileLeft","mouthSmileRight","mouthFrownRight","mouthFrownLeft","mouthDimpleLeft","mouthDimpleRight","mouthStretchLeft","mouthStretchRight","mouthRollLower","mouthRollUpper","mouthShrugLower","mouthShrugUpper","mouthPressLeft","mouthPressRight","mouthLowerDownLeft","mouthLowerDownRight","mouthUpperUpLeft","mouthUpperUpRight","browDownLeft","browDownRight","browInnerUp","browOuterUpLeft","browOuterUpRight","cheekPuff","cheekSquintLeft","cheekSquintRight","noseSneerLeft","noseSneerRight","tongueOut"}
public int Audio2ExpApi.release();
Releases resources. Call this API when you no longer need the capability.

Integration Code Sample

@Override
protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        findViewById(R.id.button).setOnClickListener(new OnClickListener() {
                @Override
                public void onClick(View view) {
                        TELicenseCheck.getInstance().setTELicense(MainActivity.this, licenseUrl, licenseKey, new TELicenseCheckListener() {
                                @Override
                                public void onLicenseCheckFinish(int errorCode, String s) {
                                        Log.d(TAG, "onLicenseCheckFinish: errorCode = "+errorCode+",msg="+s);
                                        if (errorCode == TELicenseCheck.ERROR_OK) {
                                                //license check success
                                                Audio2ExpApi audio2ExpApi = new Audio2ExpApi();
                                                int err = audio2ExpApi.init(MainActivity.this.getFilesDir() +"/models");
                                                Log.d(TAG, "onLicenseCheckFinish: err="+err);
                                                //TODO start record and parse audio data
                                        } else {
                                                // license check failed
                                        }
                                }
                        });
                }
        });
}
Note
For the full code sample, see Demos.
For audio recording, see com.tencent.demo.avatar.audio.AudioCapturer.
For more information on how to use the APIs, see com.tencent.demo.avatar.activity.Audio2ExpActivity and its related classes.