Initialization

To kickstart facial expression functionality, initialize the ConvaiClient with the necessary parameters. The enableFacialData flag must be set to true to enable facial expression data.

convaiClient.current = new ConvaiClient({
  apiKey: '<apiKey>',
  characterId: '<characterId>',
  enableAudio: true,
  enableFacialData: true,
  faceModel: 3, // OVR lipsync
});

Receiving Viseme Data

Retrieve viseme data by incorporating the provided callback. The example code demonstrates how to handle and update facial data.

const [facialData, setFacialData] = useState([]);
const facialRef = useRef([]);

convaiClient.current.setResponseCallback((response) => {
if (response.hasAudioResponse()) {
    let audioResponse = response?.getAudioResponse();
      if (audioResponse?.getVisemesData()?.array[0]) {
      //Viseme data
        let faceData = audioResponse?.getVisemesData().array[0];
        //faceData[0] implies sil value. Which is -2 if new chunk of audio is recieve.
          if (faceData[0] !== -2) {
            facialRef.current.push(faceData);
            setFacialData(facialRef.current);
          }
      }
}

Modulating Morph Targets

Utilize the useFrame hook from react-three-fiber to modulate morph targets based on the received facial data.

import {OvrToMorph} from 'convai-web-sdk'
const blendShapeRef = useRef([]);
const currentBlendFrame = useRef(0);

useFrame((state, _delta) => {
 // Initiate blendshapes
    if(client?.facialData.length > 0){
// OvrToMorph is required for mapping reallusion morphs
    OvrToMorph(client?.facialData[currentBlendFrame.current],blendShapeRef);
    }
  if (currentBlendFrame.current <= blendShapeRef?.current?.length) {
    // Logic to adjust morph targets based on facial data
    currentBlendFrame.current += 1;
  }
});

In addition to facial expressions, the convai-web-sdk allows developers to modulate bone adjustments for specific facial features. Receive bone adjustments for "Open_Jaw," "Tongue," and "V_Tongue_Out" and apply them to your character as demonstrated below:

// If current viseme value is Open_Jaw
characterRef.current.getObjectByName("CC_Base_JawRoot").setRotationFromEuler(jawRotation);
// The jaw rotation value can be modulated using THREE.lerp().
// example
const jawRotation = new THREE.Euler(0,0,1.57);
// Here 1.57 is the base value for closed Jaw.
jawRotation.z = THREE.MathUtils.lerp(jawRotation.z,1.57 + blendShapeRef.current[currentBlendFrame.current-1][blend]*0.2,0.8);

Handling 100fps Animation

Implement throttling using lodash to ensure smooth animations at 100fps. The provided example demonstrates how to maintain a consistent animation frame rate.

const throttledUpdate = _.throttle(updateAnimation, 10);
const [tick, setTick] = useState(true);

const updateAnimation = () => {
  setTick((tick) => {
    if (tick) {
      return tick;
    }
    return true;
  });
  requestAnimationFrame(throttledUpdate);
};

// Start the animation loop when the component mounts
useEffect(() => {
  requestAnimationFrame(throttledUpdate);
  // Clean up the animation loop when the component unmounts
  return () => {
    cancelAnimationFrame(throttledUpdate);
  };
}, []);

// Use the frame hook to update animations
useFrame((state, _delta) => {
  if (tick) {
    // Logic to get viseme data and alter morph targets accordingly
    setTick(false);
  }
});

Handling 100fps Edge Cases

Throttle accuracy may lead to edge cases. Implement the clock setup and handle both above and below 100fps scenarios using the elapsed time.

const [startClock, setStartClock] = useState(false);

useFrame((state, _delta) => {
  if (!startClock || !client?.isTalking) {
    state.clock.elapsedTime = 0;
    if (startClock) setStartClock(false);
  }
  if (client?.isTalking) {
    setStartClock(true);
  }
});

// Handle both above and below 100fps scenarios
if (startClock) {
  if (Math.abs(Math.floor(state.clock.elapsedTime * 100) - currentBlendFrame.current) > 15) {
    if (Math.floor(state.clock.elapsedTime * 100) - currentBlendFrame.current > 0) {
      // Below 100fps
           for(let i=0;i<15;i++){
              blendShapeRef.current.push(0)
            }
            currentBlendFrame.current += 15;
    } else {
      // Above 100fps
       if(blendShapeRef.current.length > 15){
            blendShapeRef.current.splice(-15);
            currentBlendFrame.current -= 15;
       }
    }
  }
}

To set up the Actions you need to follow the following steps:

1. Sign in to Convai's website and navigate to your Character Details.

2. Navigate to Actions, enable the Action Generation and select the actions you want your NPC to perform.

3. Go back to your code and Initialize an actionText state that will store the action that you want NPC to perform.

const [actionText, setActionText] = useState("");

1. Inside the same useEffect where we check the audio response. Refer to the Getting Started page to quickly understand how and where we check audio response.

useEffect(()=>{

------------------------------------------------------------------------
//First Initialize Convai Client here. Refer Getting Started Page
//setResponseCallback method to be called after initializing the client
//onAudioPlay and onAudioStop methods need to be setup
------------------------------------------------------------------------

if (response.hasActionResponse()) {
        let actionResponse = response.getActionResponse();
        let parsedActions = actionResponse.getAction().trim().split("\n");
        setActionText(parsedActions[0].split(", "));
  }
 },[])

Actions have been set up and now you can use the ActionText to perform the required action.

At first import Convai client from convai-web-sdk.

Declare the convaiClient variable using the useRef hook provided by the React library. The useRef hook returns a mutable ref object, which can be used to store a value that persists across component renders.

Initialize the Convai client inside a useEffect hook to ensure it runs only once when the component is mounted. By providing an empty dependency array as the second parameter to the useEffect hook, the initialization code will be executed only on the initial render.

Your Convai Client has been initialized. Now you can use Convai Client methods to set up a conversation with your NPC.

These are the main methods that allow you to interact and converse with your NPC using the ConvaiClient.

1. setResponseCallback

Description: Sets the response callback function for the ConvaiClient instance. This callback function will be invoked when a response is received from the Convai API. This part of the code should be also under the same use effect as the initialization one.

Parameters: callback (function): A callback function that will be executed when a response is received. It takes one parameter representing the received response data.

Example:

This part of the code gets the user Query from the response and the finalized text is set as userText which is further used to generate a response from the NPC which is stored as npcText.

Also, remember to declare both the onAudioPlay and onAudioStop methods described below inside the useEffect after the setResposeCallback method to avoid facing errors.

1. startAudioChunk

Description: Initiates the client to start accepting audio chunks for voice input. This method signals the client to begin receiving and processing audio data.

Parameters: None

Example:

You can use this method to make the client listen and take input of the audio only when the user presses some particular key.

1. endAudioChunk

Description: Instructs the client to stop taking user audio input and finalize the transmission of audio chunks for voice input. This method indicates the end of the audio input and allows the client to process the received audio data.

Parameters: None

Example:

You can use this method to make the client stop listening to the audio on release of some particular key.

1. onAudioPlay

Description: Notifies whenever the NPC starts speaking.

Parameters: None

Example:

1. onAudioStop

Description: Notifies whenever the NPC stops speaking.

Parameter: None

Example:

1. sendTextChunk

Description: Can be used to send a text chunk to the client which will further be processed and the NPC output will be generated.

Parameter: text (string): Takes a text chunk of type string as input.

Example: Can be used when your are using textbox to take user's input. You can set up the text box in the following way where

You can use this method to get the input from a text box and then send it to the client for processing.

1. resetSession

Description: Used for resetting the current session.

Parameter: None

Example:

1. toggleAudioVolume

Description: Can be used to toggle audio mode from on to off or vice versa.

Parameter: None

Example:

Overview

The convai-web-sdk is a powerful npm package that empowers developers to seamlessly integrate lifelike characters into web applications. This SDK facilitates the capture of user audio streams and provides appropriate responses in the form of audio, actions, and facial expressions. Whether you're building an interactive website, a chatbot, or a game, this package adds a human touch to your user experience.

Before you begin

Sign in to and copy your API key. This will help you to converse with the avatar at a later step.

Install the package

is available as an npm package. Run the following command in the root of your React project to install the package.

npm install convai-web-sdk@latest