Adding Lip-Sync to your Character

Step 1: Add Lip-Sync Component

To add Lip-Sync to your character, follow these steps.

Convai's Lip-Sync uses OVR or Reallusion CC4 Extended (Reallusion CC4+) Blendshapes. The lip-sync will work best with models that have OVR or Reallusion CC4 Extended (Reallusion CC4+) compatible Blendshapes.

Using Unity Inspector

Select the character GameObject and click Add Component in the Inspector.

Select Convai Lip Sync to add the Lip Sync component.

Using Convai NPC Component

You can also select Add Components.

Select the Lip-Sync checkmark and click Apply Changes.

Step 2: Set up the Lip Sync Component

For Reallusion Character Models

In the new component, select the type of lipsync, and assign the Skinned Mesh Renderer with the Facial Blendshapes (here for Reallusion Characters, CC_Base_Body, CC_Base_Teeth and CC_Base_Tongue) and the game objects corresponding to the bones for Jaw and Tongue (here for Reallusion Characters, CC_Base_JawRoot and CC_Base_Tongue01). Feel free to adjust the position of the tongue with the Tongue Bone Offset field.

The screenshot shows "Reallusion Plus" already selected. You must now manually change this option from "OVR" to "Reallusion Plus" for the lip sync to work. This was due to a recent Unity update that changed the default settings.

If your character's mouth seems not to be opening enough for a proper lip sync, this could indicate an issue with the animator or animations.

For Ready Player Me Character Models

For Ready Player Me models, the type of lipsync will be OVR and the appropriate facial skinned mesh renderers for the Head will be Renderer_Head and the appropriate facial skinned mesh renderer for the Teeth will be Renderer_Teeth. There is no need for adding any of the other mesh renderers or bones.

The above process of assigning the skinned mesh renderer is not necessary. The ConvaiLipSync.cs script is capable of assigning the necessary skinned mesh renderers by itself if you have left the fields empty.

To resolve common issues caused by the animator and animations, follow the troubleshooting guides below.

Animations have Facial Blendshapes