Android
Overview
This capability processes the OpenGL textures of the camera and outputs 3D body data. You can pass the data to Unity to drive your model or use the data to implement other features.
Integration (Android)
API calls
1. Enable the feature (XmagicApi.java).
public void setFeatureEnableDisable(String featureName, boolean enable);
Set
featureName to XmagicConstant.FeatureName.BODY_3D_POINT.2. Configure the data callback (XmagicApi.java).
Version 2.6.0 and earlier versions use the following method.
void setYTDataListener(XmagicApi.XmagicYTDataListener ytDataListener)public interface XmagicYTDataListener {void onYTDataUpdate(String data)}
onYTDataUpdate returns a JSON string. You can find an example below.face_info is facial data, which is not used by this capability.For the meanings of fields in
body_3d_info, see below.
Version 3.0.0 uses the following method.
void setAIDataListener(XmagicApi.OnAIDataListener aiDataListener)public interface OnAIDataListener {void onFaceDataUpdated(List<TEFaceData> faceDataList);void onHandDataUpdated(List<TEHandData> handDataList);void onBodyDataUpdated(List<TEBodyData> bodyDataList);void onAIDataUpdated(String data); //This method is a new method added in version 3.0.0, and the data structure is consistent with the XmagicYTDataListener interface used in previous versions.}
onAIDataUpdated returns a JSON string. You can find an example below.face_info is facial data, which is not used by this capability.For the meanings of fields in
body_3d_info, see below.Body Keypoints and Descriptions
SMPL keypoints
SMPL-X hand keypoints
Here is an example of the JSON string output by the SDK:
The following section details the fields in
body_3d_info:imageWidth and imageHeight: The width and height of images sent to the SDK.items: An array. Currently, it contains only one element.index: A reserved field. You can ignore it.pose:1. Location [0,2]. The 3D location (xyz) of the body root bone with the camera at the center.
2. Location [3,12]. The body shape. It includes 10 floating numbers, which are based on 10 meshes of SMPL.
3. Location [13]. The focal length, which is 5000.
4. Location [14,29]. The OpenGL projection matrix in a 3D space, which is generated based on the focal length. A 4 x 4 projection matrix is calculated as follows in the algorithm:
matrix={2 * focal_length / img_wid, 0, 0, 0,0, 2 * focal_length / img_hei, 0,0,0,0, (zf + zn) / (zn - zf), -1,0, 0, (2.0f * zf * zn) / (zn - zf), 0};}
5. Location [30,33]. Whether the left toe, left heel, right toe, and right heel are on the ground.
position_x,position_y,position_z：
1. Location [0,23]. 2D body keypoints (figure 1 above). The value of
position_z for all 2D keypoints is 0.2. Location [24,47]. 3D body keypoints (figure 1 above).
rotation
1. Location [0,23]. The body bone rotation quaternions (wxyz).
2. Location [25,54]. The hand bone rotation quaternions (wxyz). There are 15 quaternions for each hand.
Bone Names
No. | Bone Name | Bone Name 2 |
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | "pelvis", "left_hip", "right_hip", "spine1", "left_knee", "right_knee", "spine2", "left_ankle", "right_ankle", "spine3", "left_foot", "right_foot", "neck", "left_collar", "right_collar", "head", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hand" "right_hand" | "Hips" "LeftUpLeg" "RightUpLeg" "Spine" "LeftLeg" "RightLeg" "Spine1" "LeftFoot" "RightFoot" "Spine2" "" "" "Neck" "LeftShoulder" "RightShoulder" "Head" "LeftArm" "RightArm" "LeftForeArm" "RightForeArm" "LeftHand" "RightHand" "" "" |
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | "left_index1" "left_index2" "left_index3" "left_middle1" "left_middle2" "left_middle3" "left_pinky1" "left_pinky2" "left_pinky3" "left_ring1" "left_ring2" "left_ring3" "left_thumb1" "left_thumb2" "left_thumb3 | IndexFinger1_L IndexFinger2_L IndexFinger3_L MiddleFinger1_L MiddleFinger2_L MiddleFinger3_L PinkyFinger1_L PinkyFinger2_L PinkyFinger3_L RingFinger1_L RingFinger2_L RingFinger3_L ThumbFinger1_L ThumbFinger2_L ThumbFinger3_L |
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | "right_index1" "right_index2" "right_index3" "right_middle1" "right_middle2" "right_middle3" "right_pinky1" "right_pinky2" "right_pinky3" "right_ring1" "right_ring2" "right_ring3" "right_thumb1" "right_thumb2" "right_thumb3" | IndexFinger1_R IndexFinger2_R IndexFinger3_R MiddleFinger1_R MiddleFinger2_R MiddleFinger3_R PinkyFinger1_R PinkyFinger2_R PinkyFinger3_R RingFinger1_R RingFinger2_R RingFinger3_R ThumbFinger1_R ThumbFinger2_R ThumbFinger3_R |