Learn to add lip sync to your Unity characters using Convai. Improve realism and interactivity.
Convai sends Visemes or Blend Shape Frame from back-end depending upon the face model developer chooses to use and when return Convai SDK out of the box extracts and parses it and provides it to the Convai LipSync Component
, after which the component relies of the SkinMeshRen derer
's Blendshape Effectors
and Bone Effectors
to give Convai powered NPC's realistic lipsync.
This is where developer will tell the Convai SDK, which index of Blendshape Array will be effector how much from which value. To better explain its working lets understand it with a diagram.
Here, its saying that whatever value is coming from the server will affect Blendshape at 116th index by 0.2 multiplier and Blendshape at 114th index by 0.5 multiplier. The engine representation of this would look something like this.
So, you can make you own Effector list or use one of the many that we ship in the SDK.
How to Create your own Viseme Effector List
Right click inside project panel and head over to Create > Convai > Expression > Viseme Skin Effector
which will create a Viseme Effector List Scriptable Object and now you can define your own values
This is where developer will tell the Convai SDK, how much each value coming from server will affect the rotation of the bone. To better explain its working lets understand it with a diagram.
Here, bone's rotation will be effect by the values coming from server multiplied by the values in effects. Example, for TH the value will affect bone's rotation by 0.2 multiplier and etc. The engine representation of this would look something like this.
So, you can make you own Bone Effector list or use one of the many that we ship in the SDK.
We use this formula to calucate the roatation
How to Create your own Viseme Bone Effector List
Right click inside project panel and head over to Create > Convai > Expression > Viseme Bone Effector
which will create a Viseme Bone Effector List Scriptable Object and now you can define your own values.
When you attach this component to your Convai Character, you will see something like this.
Let's learn what these learns are
Facial Expression Data
Head | Teeth | Tongue
Renderer: Skin Mesh Renderer which correspond to that specified part of the body
Viseme Effectors List: How the SkinMeshRenderer's Blendshape will be affected by values coming from server.
Jaw | Tongue Bone Effector
How much Bone's rotation will be affected by values coming from server.
Jaw | Tongue Bone
Reference to the bone which control jaw and tongue respectively
Weight Blending Power
Percentage to interpolate between two frames in late update.
Character Emotions
Learn More about Character Emotions here Character Emotion
Now you can configure the Component according to your custom configuration or use one of the many Presets Convai ships with the SDK
Now your lipsync component would be ready to use in your application.