Documentation
Search
K

Facial Expressions

This section provides comprehensive information on integrating and handling facial expressions and lipsync within your web applications using the convai-web-sdk.

Initialization

To kickstart facial expression functionality, initialize the ConvaiClient with the necessary parameters. The enableFacialData flag must be set to true to enable facial expression data.
convaiClient.current = new ConvaiClient({
apiKey: '<apiKey>',
characterId: '<characterId>',
enableAudio: true,
enableFacialData: true,
faceModel: 3, // OVR lipsync
});
faceModel : 3 is standard and actively maintained.

Receiving Viseme Data

Retrieve viseme data by incorporating the provided callback. The example code demonstrates how to handle and update facial data.
const [facialData, setFacialData] = useState([]);
const facialRef = useRef([]);
convaiClient.current.setResponseCallback((response) => {
if (response.hasAudioResponse()) {
let audioResponse = response?.getAudioResponse();
if (audioResponse?.getVisemesData()?.array[0]) {
//Viseme data
let faceData = audioResponse?.getVisemesData().array[0];
//faceData[0] implies sil value. Which is -2 if new chunk of audio is recieve.
if (faceData[0] !== -2) {
facialRef.current.push(faceData);
setFacialData(facialRef.current);
}
}
}

Modulating Morph Targets

Utilize the useFrame hook from react-three-fiber to modulate morph targets based on the received facial data.
import {OvrToMorph} from 'convai-web-sdk'
const blendShapeRef = useRef([]);
const currentBlendFrame = useRef(0);
useFrame((state, _delta) => {
// Initiate blendshapes
if(client?.facialData.length > 0){
// OvrToMorph is required for mapping reallusion morphs
OvrToMorph(client?.facialData[currentBlendFrame.current],blendShapeRef);
}
if (currentBlendFrame.current <= blendShapeRef?.current?.length) {
// Logic to adjust morph targets based on facial data
currentBlendFrame.current += 1;
}
});
In addition to facial expressions, the convai-web-sdk allows developers to modulate bone adjustments for specific facial features. Receive bone adjustments for "Open_Jaw," "Tongue," and "V_Tongue_Out" and apply them to your character as demonstrated below:
// If current viseme value is Open_Jaw
characterRef.current.getObjectByName("CC_Base_JawRoot").setRotationFromEuler(jawRotation);
// The jaw rotation value can be modulated using THREE.lerp().
// example
const jawRotation = new THREE.Euler(0,0,1.57);
// Here 1.57 is the base value for closed Jaw.
jawRotation.z = THREE.MathUtils.lerp(jawRotation.z,1.57 + blendShapeRef.current[currentBlendFrame.current-1][blend]*0.2,0.8);
These code examples are specific to reallusion characters.

Handling 100fps Animation

Implement throttling using lodash to ensure smooth animations at 100fps. The provided example demonstrates how to maintain a consistent animation frame rate.
const throttledUpdate = _.throttle(updateAnimation, 10);
const [tick, setTick] = useState(true);
const updateAnimation = () => {
setTick((tick) => {
if (tick) {
return tick;
}
return true;
});
requestAnimationFrame(throttledUpdate);
};
// Start the animation loop when the component mounts
useEffect(() => {
requestAnimationFrame(throttledUpdate);
// Clean up the animation loop when the component unmounts
return () => {
cancelAnimationFrame(throttledUpdate);
};
}, []);
// Use the frame hook to update animations
useFrame((state, _delta) => {
if (tick) {
// Logic to get viseme data and alter morph targets accordingly
setTick(false);
}
});
Note: Throttle function is not 100% accurate

Handling 100fps Edge Cases

Throttle accuracy may lead to edge cases. Implement the clock setup and handle both above and below 100fps scenarios using the elapsed time.
const [startClock, setStartClock] = useState(false);
useFrame((state, _delta) => {
if (!startClock || !client?.isTalking) {
state.clock.elapsedTime = 0;
if (startClock) setStartClock(false);
}
if (client?.isTalking) {
setStartClock(true);
}
});
// Handle both above and below 100fps scenarios
if (startClock) {
if (Math.abs(Math.floor(state.clock.elapsedTime * 100) - currentBlendFrame.current) > 15) {
if (Math.floor(state.clock.elapsedTime * 100) - currentBlendFrame.current > 0) {
// Below 100fps
for(let i=0;i<15;i++){
blendShapeRef.current.push(0)
}
currentBlendFrame.current += 15;
} else {
// Above 100fps
if(blendShapeRef.current.length > 15){
blendShapeRef.current.splice(-15);
currentBlendFrame.current -= 15;
}
}
}
}