1 of 7

PlayCanvas Plugin

PlayCanvas template for Convai integration.

Overview

Integrating ConvAI with PlayCanvas

ConvAI is a powerful tool that enables developers to incorporate natural language processing (NLP) and conversational AI capabilities into their PlayCanvas projects. By following this guide, you'll learn how to seamlessly integrate ConvAI into your PlayCanvas project, allowing you to create engaging and interactive experiences for your users.

To help you get started, we've created a reference PlayCanvas project that demonstrates the integration of ConvAI. This project serves as a foundation for you to build upon and understand the necessary steps to incorporate ConvAI into your own projects.

Adding External Script

Adding External Script - PlayCanvas Plugin Guide for Convai integration.

Here we will add the Convai Web SDK to our project, a JavaScript library that enables integration of conversational AI capabilities.

Create a blank project after loggin in.
Open up settings section in SETTINGS panel.
Increase the Array Size to 1 for adding 1 external script.
Add Convai web-sdk-cdn link in url section.

First Person View

First Person View - PlayCanvas Plugin Guide for Convai integration.

Adding Physics components to plane

Scale up the plane and add physics component to it. Add both collision and rigidbody.
Import Ammo Js (enables physics). Scale up the Collision component according to the plane size.

Player Body

Create a New Entity
- In the PlayCanvas Editor, right-click in the Hierarchy panel and select "Create New Entity".
Add Physics Component
- With the new entity selected, click the "Add Component" button in the top-right corner of the Editor.
- Search for "Physics" and add the "Physics" component to the entity.
Add Collision Capsule Component
- With the entity still selected, click "Add Component" again.
- Search for "Collision" and add the "Collision Capsule" component to the entity.
Adjust Entity Y-Position
- Adjust the "Y" value of the Translation to position the entity above the plane.
Add Rigidbody Component
- With the entity still selected, click "Add Component" again.
- Search for "Rigidbody" and add the "Rigidbody" component to the entity.
- Set the "Type" of the rigidbody to "Dynamic".
Adjust Angular Factors
- In the Rigidbody component, locate the "Angular Factor" section.
- Set the "X", "Y", and "Z" values of the Angular Factor to 0.

Camera controls for first person view

Adding characters to scene

Adding Characters to Scene - PlayCanvas Plugin Guide for Convai integration.

When preparing your .glb file, ensure that the model has morph targets for lip synchronization. Morph targets animate facial expressions and lip movements.

Download preferred glb character.

Once glb file is created. Drag and drop it to PlayCanvas Editor.

Character Animations

Add character animations in PlayCanvas with Convai. Enhance your web projects with interactive AI.

Upon completing the creation/upload of all desired animations within the PlayCanvas environment, construct an Animation State Graph to effectively manage and control the animation states and transitions

Reference : https://developer.playcanvas.com/tutorials/anim-blending/

Attach the state graph and animation files to the character. Create a Anim component and drag-drop the files to the placeholders.

Convai Integration

Convai Integration - PlayCanvas Plugin Guide for seamless integration.

After the addition of Convai web-sdk-cdn to url section, ConvaiClient class will be available to the browser directly.

Add all the scripts bellow to your Character entity.

Convai Initialization Script

Replace the empty "" with you API key and Character ID.

The ConvaiNpc script is responsible for handling the interaction between the user and a virtual character powered by the Convai AI.

The script initializes the Convai client by providing an API key and character ID. It sets up necessary callbacks to handle various events, such as errors, user queries, and audio responses from the Convai service.

The initializeConvaiClient function is the entry point for setting up the Convai client. It creates a new instance of the ConvaiClient and configures it with the provided API key, character ID, and other settings like enabling audio and facial data.

The script handles user input through two methods: text input via a form and voice input using the "T" key. For voice input, the handleKeyDown and handleKeyUp functions are used to detect when the "T" key is pressed and released, respectively. When the "T" key is pressed, the script starts recording audio and sends it to the Convai service for processing.

The ConvaiNpc.prototype.initialize function is called once per entity and sets up the Convai client. It also registers callbacks for handling audio playback events, updating the isTalking and conversationActive flags accordingly.

The ConvaiNpc.prototype.handleAnimation function updates the character's animation based on the isTalking state, allowing for synchronized lip movements and facial expressions.

var ConvaiNpc = pc.createScript('convaiNpc');

/** Loading convai-sdk cdn */
var convaiClient = null;
var userTextStream = ""; // Stores the user's text input
var npcTextStream = ""; // Stores the NPC's text response
var keyPressed = false; // Keeps track of whether the "T" key is pressed
var isTalking = false; // Indicates if the NPC is currently talking
var conversationActive = false; // Indicates if a conversation is currently active
var visemeData = []; // Stores viseme data for lip sync animation
var visemeDataActive = false; // Indicates if viseme data is available
var formLoaded = false; // Indicates if the form has been loaded
let timeoutId;
/**
 * Initializes the ConvaiClient with an API key, a character ID, and flags for enabling the audio recorder and player. Call this first from Start()
 * Sets up several callbacks for handling different types of responses from the Convai Client.
 * @param {string} apiKey - The API key for the Convai service.
 * @param {string} characterId - The ID of the character to use in the Convai service.
 * @param {boolean} enableAudioRecorder - Flag to enable the audio recorder.
 * @param {boolean} enableAudioPlayer - Flag to enable the audio player.
 * @param {number} faceModal - The face modal to use for the character.
 * @param {boolean} enableFacialData - Flag to enable facial data.
*/
function initializeConvaiClient () {
    console.log("Convai client initiated.");
    convaiClient = new ConvaiClient({
        apiKey: "", // Replace with your API key
        characterId: "", // Replace with your character ID
        enableAudio: true,
        faceModal: 3,
        enableFacialData: true,
    });

    // Error callback
    convaiClient.setErrorCallback(function(type, statusMessage) {
        console.log("Error Callback", type, statusMessage);
    });

    // Response callback
    convaiClient.setResponseCallback(function(response) {
        // Handle user query
        if (response.hasUserQuery) {
            var transcript = response.getUserQuery();
            if (transcript) {
                var isFinal = transcript?.getIsFinal();
                var transcriptText = transcript.getTextData();
                if (isFinal) {
                    userTextStream += " " + transcriptText;
                }
                userTextStream = transcriptText;
            }
        }

        // Handle audio response
        if (response.hasAudioResponse()) {
            var audioResponse = response.getAudioResponse();
            npcTextStream += " " + audioResponse.getTextData();

            // Handle viseme data for lip sync animation
            if (audioResponse.hasVisemesData()) {
                let lipsyncData = audioResponse.getVisemesData().array[0];
                if (lipsyncData[0] !== -2) {
                    visemeData.push(lipsyncData);
                    visemeDataActive = true;
                }
            } else {
                visemeDataActive = false;
            }
        }
    });
}

// Handle key down event for starting audio input
function handleKeyDown(e) {
    const convaiFormEl = document.getElementById("convai-input");
    if (convaiClient && e.keyCode === 84 && !keyPressed && !(document.activeElement === convaiFormEl)) {
        e.stopPropagation();
        e.preventDefault();
        keyPressed = true;
        userTextStream = "";
        npcTextStream = "";
        convaiClient.startAudioChunk();
        conversationActive = true;
    }
}

// Handle key up event for stopping audio input
function handleKeyUp(e) {
    const convaiFormEl = document.getElementById("convai-input");
    if (convaiClient && e.keyCode === 84 && keyPressed && !(document.activeElement === convaiFormEl)) {
        e.preventDefault();
        keyPressed = false;
        clearTimeout(timeoutId);
        timeoutId = setTimeout(() => {
        convaiClient.endAudioChunk();
        }, 500);
    }
}

// Add event listeners for key down and key up
window.addEventListener("keydown", handleKeyDown);
window.addEventListener("keyup", handleKeyUp);

// Initialize code called once per entity
ConvaiNpc.prototype.initialize = function() {
    initializeConvaiClient();

    // Set callbacks for audio playback
    convaiClient.onAudioPlay(() => {
        isTalking = true;
    });
    convaiClient.onAudioStop(() => {
        isTalking = false;
        conversationActive = false;
    });
};

// Handle animation based on the talking state
ConvaiNpc.prototype.handleAnimation = function() {
    this.entity.anim.setBoolean("Talking", isTalking);
};

// Update code called every frame
ConvaiNpc.prototype.update = function(dt) {
    this.handleAnimation();

    // Check if the form is loaded and add event listener
    if (!formLoaded && document.getElementById("convai-form")) {
        this.formListener();
        formLoaded = true;
    }
};

// Add event listener for form submission
ConvaiNpc.prototype.formListener = function() {
    document.getElementById("convai-form").addEventListener("submit", (e) => {
        e.preventDefault();
        var inputValue = document.getElementById("convai-input").value;
        window.convaiClient.sendTextChunk(inputValue);
        userTextStream = inputValue;
        document.getElementById("convai-form").reset();
        conversationActive = true;
        npcTextStream = "";
    });
}

PlayerAnimationHandler Script

The PlayerAnimationHandler script is responsible for controlling the animations of a player character based on certain conditions, such as velocity or other factors.

The script defines three attributes:

blendTime: This attribute controls the blend time between animations, which determines how smoothly the transition between animations occurs. The default value is set to 0.2.
velMin: This attribute represents the minimum velocity required to trigger a specific animation. The default value is set to 10.
velMax: This attribute represents the maximum velocity required to trigger a specific animation. The default value is set to 50.

// Create a new script called 'playerAnimationHandler'
var PlayerAnimationHandler = pc.createScript('playerAnimationHandler');

// Add an attribute called 'blendTime' with a default value of 0.2
// This attribute controls the blend time between animations
PlayerAnimationHandler.attributes.add('blendTime', { type: 'number', default: 0.2 });

// Add an attribute called 'velMin' with a default value of 10
// This attribute represents the minimum velocity for a specific animation
PlayerAnimationHandler.attributes.add('velMin', { type: 'number', default: 10 });

// Add an attribute called 'velMax' with a default value of 50
// This attribute represents the maximum velocity for a specific animation
PlayerAnimationHandler.attributes.add('velMax', { type: 'number', default: 50 });

// This function is called when the script is initialized
PlayerAnimationHandler.prototype.initialize = function() {
    // Play the 'Idle' animation with the specified blend time
    this.entity.animation.play('Idle', this.blendTime);
};

These attributes can be adjusted in the editor or through code to fine-tune the animation behavior for the player character.

The initialize function is called when the script is initialized. In this implementation, it plays the 'Idle' animation with the specified blend time (this.blendTime). This animation will be played when the player character is not moving or when the velocity is outside the range defined by velMin and velMax.

The script is designed to be extended further to handle different animation states based on the player character's velocity or other conditions. For example, you could add additional functions or logic to check the player's velocity and play different animations (e.g., 'Walk', 'Run') based on the velocity range defined by velMin and velMax.

By utilizing this script, you can easily manage and transition between different animations for the player character, providing a more immersive and realistic experience in your game or application.

Lipsync Script

The Lipsync script is responsible for animating the character's mouth and facial expressions based on the received viseme data. Visemes are the key mouth shapes and facial positions used to represent speech sounds. The script applies morph target animations to the character's head and teeth components to achieve realistic lip-syncing effects.

The script works by accessing the visemeData array, which contains the viseme weights for each frame of the animation. It then applies these weights to the corresponding morph targets on the head and teeth components. The runVisemeData function handles this process by looping through the viseme weights and setting the morph target weights accordingly.

The script keeps track of the current viseme frame using the currentVisemeFrame variable and a timer variable. This ensures that the viseme animations are synchronized with the audio playback. When the viseme data has finished playing, the zeroMorphs function is called to reset all morph target weights to zero, effectively resetting the character's facial expression.

// Create a new script called 'lipsync'
var Lipsync = pc.createScript('lipsync');

// Array to store the lipsync data
var lipsyncData = [];

// initialize code called once per entity
Lipsync.prototype.initialize = function() {
    // Find the head component of the entity
    this.headComponent = this.entity.findByName("Wolf3D_Head").render;

    // Find the teeth component of the entity
    this.teethComponent = this.entity.findByName("Wolf3D_Teeth").render;

    // Initialize the current viseme frame to 0
    this.currentVisemeFrame = 0;

    // Initialize the timer to 0
    this.timer = 0;
};

// Run morph targets on the viseme data received
Lipsync.prototype.runVisemeData = function(dt) {
    // Sync frames with time
    if (this.currentVisemeFrame <= lipsyncData.length && lipsyncData.length !== 0) {
        if (this.headComponent && this.teethComponent) {
            // Get the morph instance for the head and teeth components
            var headMorphInstance = this.headComponent.meshInstances[0].morphInstance;
            var teethMorphInstance = this.teethComponent.meshInstances[0].morphInstance;

            if (lipsyncData[this.currentVisemeFrame] !== undefined) {
                // Loop through the morph targets and set their weights
                for (let i = 0; i < 15; i++) {
                    if (lipsyncData[this.currentVisemeFrame][i] !== undefined) {
                        headMorphInstance.setWeight(i, lipsyncData[this.currentVisemeFrame][i]);
                        teethMorphInstance.setWeight(i, lipsyncData[this.currentVisemeFrame][i]);
                    }
                }
            }

            // Update the current viseme frame based on the timer
            this.currentVisemeFrame = Math.floor(this.timer * 100);
        }
    }
}

// Reset all morph targets to 0
Lipsync.prototype.zeroMorphs = function(dt) {
    if (this.headComponent && this.teethComponent) {
        // Get the morph instance for the head and teeth components
        var headMorphInstance = this.headComponent.meshInstances[0].morphInstance;
        var teethMorphInstance = this.teethComponent.meshInstances[0].morphInstance;

        // Loop through the morph targets and set their weights to 0
        for (let i = 0; i < 15; i++) {
            headMorphInstance.setWeight(i, 0);
            teethMorphInstance.setWeight(i, 0);
        }
    }
}

// update code called every frame
Lipsync.prototype.update = function(dt) {
    // Check if there is new viseme data and update the lipsyncData array
    if (window.visemeData && window.visemeDataActive) {
        lipsyncData = window.visemeData;
    }

    if (lipsyncData.length > 0) {
        // Update the timer
        this.timer += dt;

        // Run the viseme data
        this.runVisemeData();

        // Check if the viseme data has finished playing
        if (this.timer * 100 >= lipsyncData.length) {
            // Reset all morph targets to 0
            this.zeroMorphs();

            // Clear the lipsyncData array and reset the timer and currentVisemeFrame
            lipsyncData = [];
            this.timer = 0;
            this.currentVisemeFrame = 0;
            window.visemeData = [];
        }
    }
};

HeadTracking Script

The HeadTracking script is responsible for controlling the rotation of a character's head and eyes based on the position of the camera (representing the user's viewpoint). The script achieves this by calculating the angle between the forward vector of the head component and the forward vector of the camera. If this angle is within a specified threshold (45 degrees in this case), the head and eyes are rotated to look towards the camera's position.

// Create a new script called 'headTracking'
var HeadTracking = pc.createScript('headTracking');

// Helper function to find a child entity by name recursively
HeadTracking.prototype.findChild = function(children, matchName) {
    for (let child of children) {
        if (child.name.includes(matchName)) {
            return child;
        }

        if (child.children) {
            const foundChild = this.findChild(child.children, matchName);
            if (foundChild) {
                return foundChild;
            }
        }
    }

    return null;
};

// Function to calculate the radian angle between two vectors
HeadTracking.prototype.getRadianAngle = function(vecA, vecB) {
    dot = vecA.normalize().dot(vecB.normalize());
    return Math.acos(dot);
};

// initialize code called once per entity
HeadTracking.prototype.initialize = function() {
    // Find the head, left eye, and right eye components
    this.headComponent = this.entity.findByName("Head");
    this.leftEye = this.entity.findByName("LeftEye");
    this.rightEye = this.entity.findByName("RightEye");
};

// Function called after the update loop
HeadTracking.prototype.postUpdate = function(dt) {
    // Check if the camera position is available
    if (window.fpsCamera?.position !== undefined) {
        const cameraPosition = window.fpsCamera.getPosition().clone();
        const angle = this.getRadianAngle(this.headComponent.forward, window.fpsCamera.forward);
        const degree = angle * pc.math.RAD_TO_DEG;

        // If the angle is within 45 degrees, look at the camera position
        if (degree <= 45) {
            this.headComponent.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
            this.leftEye.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
            this.rightEye.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
        }
    }
};

Add all the above scripts to your playcanvas project and attach convaiNPC, lipsync, Headtracking to the convi (your model) component.

Chat Overlay

Chat Overlay - PlayCanvas Plugin Guide for Convai integration.

Let's add a chat window to enhance user interaction and immersion. Create a New entity called Convai Chat.

Convai Chat script

The ConvaiChat script is responsible for managing the chat interface and displaying the conversation between the user and an AI-powered virtual character. It handles rendering user messages and AI responses, maintaining a chat history, and ensuring smooth scrolling behavior within the chat container.

Add the following files as an attachment to convaiChat script after parsing the script.

HTML

CSS

Convai Integration

Convai Integration - PlayCanvas Plugin Guide for seamless integration.

After the addition of Convai web-sdk-cdn to url section, ConvaiClient class will be available to the browser directly.

Add all the scripts bellow to your Character entity.

Convai Initialization Script

Replace the empty "" with you API key and Character ID.

The ConvaiNpc script is responsible for handling the interaction between the user and a virtual character powered by the Convai AI.

The ConvaiNpc.prototype.handleAnimation function updates the character's animation based on the isTalking state, allowing for synchronized lip movements and facial expressions.

var ConvaiNpc = pc.createScript('convaiNpc');

/** Loading convai-sdk cdn */
var convaiClient = null;
var userTextStream = ""; // Stores the user's text input
var npcTextStream = ""; // Stores the NPC's text response
var keyPressed = false; // Keeps track of whether the "T" key is pressed
var isTalking = false; // Indicates if the NPC is currently talking
var conversationActive = false; // Indicates if a conversation is currently active
var visemeData = []; // Stores viseme data for lip sync animation
var visemeDataActive = false; // Indicates if viseme data is available
var formLoaded = false; // Indicates if the form has been loaded
let timeoutId;
/**
 * Initializes the ConvaiClient with an API key, a character ID, and flags for enabling the audio recorder and player. Call this first from Start()
 * Sets up several callbacks for handling different types of responses from the Convai Client.
 * @param {string} apiKey - The API key for the Convai service.
 * @param {string} characterId - The ID of the character to use in the Convai service.
 * @param {boolean} enableAudioRecorder - Flag to enable the audio recorder.
 * @param {boolean} enableAudioPlayer - Flag to enable the audio player.
 * @param {number} faceModal - The face modal to use for the character.
 * @param {boolean} enableFacialData - Flag to enable facial data.
*/
function initializeConvaiClient () {
    console.log("Convai client initiated.");
    convaiClient = new ConvaiClient({
        apiKey: "", // Replace with your API key
        characterId: "", // Replace with your character ID
        enableAudio: true,
        faceModal: 3,
        enableFacialData: true,
    });

    // Error callback
    convaiClient.setErrorCallback(function(type, statusMessage) {
        console.log("Error Callback", type, statusMessage);
    });

    // Response callback
    convaiClient.setResponseCallback(function(response) {
        // Handle user query
        if (response.hasUserQuery) {
            var transcript = response.getUserQuery();
            if (transcript) {
                var isFinal = transcript?.getIsFinal();
                var transcriptText = transcript.getTextData();
                if (isFinal) {
                    userTextStream += " " + transcriptText;
                }
                userTextStream = transcriptText;
            }
        }

        // Handle audio response
        if (response.hasAudioResponse()) {
            var audioResponse = response.getAudioResponse();
            npcTextStream += " " + audioResponse.getTextData();

            // Handle viseme data for lip sync animation
            if (audioResponse.hasVisemesData()) {
                let lipsyncData = audioResponse.getVisemesData().array[0];
                if (lipsyncData[0] !== -2) {
                    visemeData.push(lipsyncData);
                    visemeDataActive = true;
                }
            } else {
                visemeDataActive = false;
            }
        }
    });
}

// Handle key down event for starting audio input
function handleKeyDown(e) {
    const convaiFormEl = document.getElementById("convai-input");
    if (convaiClient && e.keyCode === 84 && !keyPressed && !(document.activeElement === convaiFormEl)) {
        e.stopPropagation();
        e.preventDefault();
        keyPressed = true;
        userTextStream = "";
        npcTextStream = "";
        convaiClient.startAudioChunk();
        conversationActive = true;
    }
}

// Handle key up event for stopping audio input
function handleKeyUp(e) {
    const convaiFormEl = document.getElementById("convai-input");
    if (convaiClient && e.keyCode === 84 && keyPressed && !(document.activeElement === convaiFormEl)) {
        e.preventDefault();
        keyPressed = false;
        clearTimeout(timeoutId);
        timeoutId = setTimeout(() => {
        convaiClient.endAudioChunk();
        }, 500);
    }
}

// Add event listeners for key down and key up
window.addEventListener("keydown", handleKeyDown);
window.addEventListener("keyup", handleKeyUp);

// Initialize code called once per entity
ConvaiNpc.prototype.initialize = function() {
    initializeConvaiClient();

    // Set callbacks for audio playback
    convaiClient.onAudioPlay(() => {
        isTalking = true;
    });
    convaiClient.onAudioStop(() => {
        isTalking = false;
        conversationActive = false;
    });
};

// Handle animation based on the talking state
ConvaiNpc.prototype.handleAnimation = function() {
    this.entity.anim.setBoolean("Talking", isTalking);
};

// Update code called every frame
ConvaiNpc.prototype.update = function(dt) {
    this.handleAnimation();

    // Check if the form is loaded and add event listener
    if (!formLoaded && document.getElementById("convai-form")) {
        this.formListener();
        formLoaded = true;
    }
};

// Add event listener for form submission
ConvaiNpc.prototype.formListener = function() {
    document.getElementById("convai-form").addEventListener("submit", (e) => {
        e.preventDefault();
        var inputValue = document.getElementById("convai-input").value;
        window.convaiClient.sendTextChunk(inputValue);
        userTextStream = inputValue;
        document.getElementById("convai-form").reset();
        conversationActive = true;
        npcTextStream = "";
    });
}

PlayerAnimationHandler Script

The PlayerAnimationHandler script is responsible for controlling the animations of a player character based on certain conditions, such as velocity or other factors.

The script defines three attributes:

blendTime: This attribute controls the blend time between animations, which determines how smoothly the transition between animations occurs. The default value is set to 0.2.
velMin: This attribute represents the minimum velocity required to trigger a specific animation. The default value is set to 10.
velMax: This attribute represents the maximum velocity required to trigger a specific animation. The default value is set to 50.

// Create a new script called 'playerAnimationHandler'
var PlayerAnimationHandler = pc.createScript('playerAnimationHandler');

// Add an attribute called 'blendTime' with a default value of 0.2
// This attribute controls the blend time between animations
PlayerAnimationHandler.attributes.add('blendTime', { type: 'number', default: 0.2 });

// Add an attribute called 'velMin' with a default value of 10
// This attribute represents the minimum velocity for a specific animation
PlayerAnimationHandler.attributes.add('velMin', { type: 'number', default: 10 });

// Add an attribute called 'velMax' with a default value of 50
// This attribute represents the maximum velocity for a specific animation
PlayerAnimationHandler.attributes.add('velMax', { type: 'number', default: 50 });

// This function is called when the script is initialized
PlayerAnimationHandler.prototype.initialize = function() {
    // Play the 'Idle' animation with the specified blend time
    this.entity.animation.play('Idle', this.blendTime);
};

These attributes can be adjusted in the editor or through code to fine-tune the animation behavior for the player character.

By utilizing this script, you can easily manage and transition between different animations for the player character, providing a more immersive and realistic experience in your game or application.

Lipsync Script

// Create a new script called 'lipsync'
var Lipsync = pc.createScript('lipsync');

// Array to store the lipsync data
var lipsyncData = [];

// initialize code called once per entity
Lipsync.prototype.initialize = function() {
    // Find the head component of the entity
    this.headComponent = this.entity.findByName("Wolf3D_Head").render;

    // Find the teeth component of the entity
    this.teethComponent = this.entity.findByName("Wolf3D_Teeth").render;

    // Initialize the current viseme frame to 0
    this.currentVisemeFrame = 0;

    // Initialize the timer to 0
    this.timer = 0;
};

// Run morph targets on the viseme data received
Lipsync.prototype.runVisemeData = function(dt) {
    // Sync frames with time
    if (this.currentVisemeFrame <= lipsyncData.length && lipsyncData.length !== 0) {
        if (this.headComponent && this.teethComponent) {
            // Get the morph instance for the head and teeth components
            var headMorphInstance = this.headComponent.meshInstances[0].morphInstance;
            var teethMorphInstance = this.teethComponent.meshInstances[0].morphInstance;

            if (lipsyncData[this.currentVisemeFrame] !== undefined) {
                // Loop through the morph targets and set their weights
                for (let i = 0; i < 15; i++) {
                    if (lipsyncData[this.currentVisemeFrame][i] !== undefined) {
                        headMorphInstance.setWeight(i, lipsyncData[this.currentVisemeFrame][i]);
                        teethMorphInstance.setWeight(i, lipsyncData[this.currentVisemeFrame][i]);
                    }
                }
            }

            // Update the current viseme frame based on the timer
            this.currentVisemeFrame = Math.floor(this.timer * 100);
        }
    }
}

// Reset all morph targets to 0
Lipsync.prototype.zeroMorphs = function(dt) {
    if (this.headComponent && this.teethComponent) {
        // Get the morph instance for the head and teeth components
        var headMorphInstance = this.headComponent.meshInstances[0].morphInstance;
        var teethMorphInstance = this.teethComponent.meshInstances[0].morphInstance;

        // Loop through the morph targets and set their weights to 0
        for (let i = 0; i < 15; i++) {
            headMorphInstance.setWeight(i, 0);
            teethMorphInstance.setWeight(i, 0);
        }
    }
}

// update code called every frame
Lipsync.prototype.update = function(dt) {
    // Check if there is new viseme data and update the lipsyncData array
    if (window.visemeData && window.visemeDataActive) {
        lipsyncData = window.visemeData;
    }

    if (lipsyncData.length > 0) {
        // Update the timer
        this.timer += dt;

        // Run the viseme data
        this.runVisemeData();

        // Check if the viseme data has finished playing
        if (this.timer * 100 >= lipsyncData.length) {
            // Reset all morph targets to 0
            this.zeroMorphs();

            // Clear the lipsyncData array and reset the timer and currentVisemeFrame
            lipsyncData = [];
            this.timer = 0;
            this.currentVisemeFrame = 0;
            window.visemeData = [];
        }
    }
};

HeadTracking Script

// Create a new script called 'headTracking'
var HeadTracking = pc.createScript('headTracking');

// Helper function to find a child entity by name recursively
HeadTracking.prototype.findChild = function(children, matchName) {
    for (let child of children) {
        if (child.name.includes(matchName)) {
            return child;
        }

        if (child.children) {
            const foundChild = this.findChild(child.children, matchName);
            if (foundChild) {
                return foundChild;
            }
        }
    }

    return null;
};

// Function to calculate the radian angle between two vectors
HeadTracking.prototype.getRadianAngle = function(vecA, vecB) {
    dot = vecA.normalize().dot(vecB.normalize());
    return Math.acos(dot);
};

// initialize code called once per entity
HeadTracking.prototype.initialize = function() {
    // Find the head, left eye, and right eye components
    this.headComponent = this.entity.findByName("Head");
    this.leftEye = this.entity.findByName("LeftEye");
    this.rightEye = this.entity.findByName("RightEye");
};

// Function called after the update loop
HeadTracking.prototype.postUpdate = function(dt) {
    // Check if the camera position is available
    if (window.fpsCamera?.position !== undefined) {
        const cameraPosition = window.fpsCamera.getPosition().clone();
        const angle = this.getRadianAngle(this.headComponent.forward, window.fpsCamera.forward);
        const degree = angle * pc.math.RAD_TO_DEG;

        // If the angle is within 45 degrees, look at the camera position
        if (degree <= 45) {
            this.headComponent.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
            this.leftEye.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
            this.rightEye.lookAt(-cameraPosition.x, cameraPosition.y, -cameraPosition.z);
        }
    }
};

Add all the above scripts to your playcanvas project and attach convaiNPC, lipsync, Headtracking to the convi (your model) component.

Chat Overlay

Chat Overlay - PlayCanvas Plugin Guide for Convai integration.

Let's add a chat window to enhance user interaction and immersion. Create a New entity called Convai Chat.

Convai Chat script

// Create a new script called 'convaiChat'
var ConvaiChat = pc.createScript('convaiChat');

// Add an attribute to store the CSS asset
ConvaiChat.attributes.add('css', { type: 'asset', assetType: 'css', title: 'CSS Asset' });

// Add an attribute to store the HTML asset
ConvaiChat.attributes.add('html', { type: 'asset', assetType: 'html', title: 'HTML Asset' });

// Initialize code called once per entity
ConvaiChat.prototype.initialize = function() {
    // Create a new div element for the chat box container
    this.htmlEl = document.createElement('div');
    this.htmlEl.classList.add('chatbox__container');
    this.htmlEl.setAttribute("id", "convai-chat");
    document.body.appendChild(this.htmlEl);

    // Set the HTML content of the chat box container
    this.htmlEl.innerHTML = this.html.resource || '';

    // Create a STYLE element for the CSS
    this.cssEl = document.createElement('style');
    document.head.appendChild(this.cssEl);
    this.cssEl.innerHTML = this.css.resource || '';

    // Initialize variables
    this.asset = null;
    this.assetId = 0;
    this.currentNpcText = "";
    this.currentUserText = "";
    this.ChatHistory = [];
    this.createNewUserChatEl = true;
    this.createNewNpcChatEl = true;
    this.userChatEl = null;
    this.npcChatEl = null;
};

// Function to add a conversation to the chat history
ConvaiChat.prototype.addToChatHistory = function(userQuery, npcResponse) {
    // Create an object representing the conversation
    const conversation = {
        userQuery: userQuery.trim(),
        response: npcResponse.trim(),
    };

    // Push the conversation object to the chat history array
    this.ChatHistory.push(conversation);
};

// Function to add a user message to the chat container
ConvaiChat.prototype.addUserMessage = function(message) {
    const chatContainer = document.getElementById("chatbot__activeChat");
    if (this.createNewUserChatEl) {
        const userMessage = document.createElement("div");
        userMessage.classList.add("user_message");
        this.userChatEl = userMessage;
        chatContainer.appendChild(this.userChatEl);
    }
    this.userChatEl.textContent = message;
};

// Function to add an AI response to the chat container
ConvaiChat.prototype.addNpcMessage = function(message) {
    const chatContainer = document.getElementById("chatbot__activeChat");
    if (this.createNewNpcChatEl) {
        const convaiMessage = document.createElement("div");
        convaiMessage.classList.add("convai_message");
        this.npcChatEl = chatContainer.appendChild(convaiMessage);
    }
    this.npcChatEl.textContent = message;
};

// Function to handle the chat updates
ConvaiChat.prototype.handleChat = function() {
    // Set userText when the conversation is active
    if (window.conversationActive && (window.userTextStream !== this.currentUserText) && window.userTextStream !== "") {
        this.currentUserText = window.userTextStream;
        // UI for current user chat (also trim in the UI)
        this.addUserMessage(this.currentUserText);
        this.createNewUserChatEl = false;
    }

    // Set npcText when the conversation is active
    if (window.conversationActive && (window.npcTextStream !== this.currentNpcText) && window.npcTextStream !== "") {
        this.currentNpcText = window.npcTextStream;
        // UI for current NPC chat (also trim in the UI)
        this.addNpcMessage(this.currentNpcText);
        this.createNewNpcChatEl = false;
    }

    // Add the conversation to the chat history when it's finished
    if (!window.conversationActive && this.currentUserText !== "" && this.currentNpcText !== "") {
        this.addToChatHistory(this.currentUserText, this.currentNpcText);
        this.currentUserText = "";
        this.currentNpcText = "";
        this.createNewUserChatEl = true;
        this.createNewNpcChatEl = true;
    }

    // Scroll to the bottom of the chat container
    this.scrollTop();
};

// Function to scroll to the bottom of the chat container
ConvaiChat.prototype.scrollTop = function(dt) {
    if (window.conversationActive) {
        var chatBox = document.getElementById("chatbot__activeChat");
        chatBox.scrollTop = chatBox.scrollHeight - chatBox.clientHeight;
    }
};

// Update code called every frame
ConvaiChat.prototype.update = function(dt) {
    this.handleChat();
};

Add the following files as an attachment to convaiChat script after parsing the script.

HTML

<div class="chatbot__activeChat" id="chatbot__activeChat">    
</div>
<form id="convai-form">
      <input
        type="text"
        placeholder="Hold [T] to talk"
        class="convai_input"
        id="convai-input"
        name="convai-input"
      />
      <button type="submit" class="convai_chat_submit send-button">
        <svg
          xmlns="http://www.w3.org/2000/svg"
          height="35px"
          viewBox="0 0 24 24"
          width="35px"
          fill="#e5e5e5"
        >
          <path d="M0 0h24v24H0z" fill="none" />
          <path d="M2.01 21L23 12 2.01 3 2 10l15 2-15 2z" />
        </svg>
      </button>
</form>

CSS

*{
  box-sizing: border-box;
}

.chatbox__container{
    position: fixed;
    bottom: 1vw;
    left: 1vw;
    background-color: rgba(0, 0, 0, 0.7);
    border-radius: 16px;
    width: 30vw;
    z-index: 10;
}
.chatbot__activeChat{
  width: 100%;
  display: flex;
  flex-direction: column;
  overflow-y: auto;
  min-height: 20vh;
  max-height: 60vh;
  padding: 2%;
}
/* width */
::-webkit-scrollbar {
  width: 0px;
}
/* Track */
::-webkit-scrollbar-track {
  background: transparent;
}

/* Handle */
::-webkit-scrollbar-thumb {
  background: transparent;
}

/* Handle on hover */
::-webkit-scrollbar-thumb:hover {
  background: transparent;
}

.user_message, .convai_message {
    margin-bottom: 8px;
    padding: 8px;
    color: white;
    background: rgba(255, 255, 255, 0.07);
    backdrop-filter: blur( 2px );
    -webkit-backdrop-filter: blur( 2px );
    border-radius: 16px;
}
.user_message{
  color: #e5e5e5;
  align-self: flex-start;
  max-width: 80%;
}
.convai_message {
  background-color: rgba(14, 114, 75, 0.4);
  color: #e5e5e5;
  align-self: flex-end;
  max-width: 80%;
}
@media only screen and (min-device-width: 0px) {
  #convai-form {
    box-sizing: border-box;
    display: flex;
    position: relative;
    padding: 2%;
    height: auto;
    bottom: 0;
    width: 100%;
    max-width: 100vw;
    column-gap: 2%;
    justify-content: space-between;
    align-items: center;
    background-color: transparent;
  }
  #convai-input {
    width: 90%;
    background: rgba(255, 255, 255, 0.07);
    border-radius: 30px;
    padding: 2%;
    color: #e5e5e5;
    outline: none;
    box-sizing: border-box;
    transition: 0.3s;
    border: none;
  }
  #convai-input::placeholder{
    color: #e5e5e5;
  }

  .active_input {
    width: 70%;
    border: 1px solid #ca7f1c;
    box-shadow: 0 0 10px rgba(255, 225, 137, 0.6);
    transition: 0.4s;
  }
}

/* Form buttons  */

.convai_chat_submit {
  box-sizing: border-box;
  display: flex;
  align-items: center;
  justify-content: center;
  border-radius: 100%;
  height: 100%;
  aspect-ratio: 1/1;
  color: #e5e5e5;
  border: 2px solid #e5e5e5;
  background-color: transparent;
  font-weight: 500;
}
.send-button {
  width: 30px;
  height: 30px;
  box-sizing: border-box;
  display: flex;
  border: 2px solid #e5e5e5;
  transition: 0.4s;
  cursor: pointer;
}

.convai-logo{
  position: fixed;
  bottom: 1vw;
  right: 1vw;
  height: 5%;
  object-fit: contain;
  width: auto;
}

First Person View

First Person View - PlayCanvas Plugin Guide for Convai integration.

Adding Physics components to plane

Scale up the plane and add physics component to it. Add both collision and rigidbody.
Import Ammo Js (enables physics). Scale up the Collision component according to the plane size.

Player Body

Create a New Entity
- In the PlayCanvas Editor, right-click in the Hierarchy panel and select "Create New Entity".
Add Physics Component
- With the new entity selected, click the "Add Component" button in the top-right corner of the Editor.
- Search for "Physics" and add the "Physics" component to the entity.
Add Collision Capsule Component
- With the entity still selected, click "Add Component" again.
- Search for "Collision" and add the "Collision Capsule" component to the entity.
Adjust Entity Y-Position
- Adjust the "Y" value of the Translation to position the entity above the plane.
Add Rigidbody Component
- With the entity still selected, click "Add Component" again.
- Search for "Rigidbody" and add the "Rigidbody" component to the entity.
- Set the "Type" of the rigidbody to "Dynamic".
Adjust Angular Factors
- In the Rigidbody component, locate the "Angular Factor" section.
- Set the "X", "Y", and "Z" values of the Angular Factor to 0.

Camera controls for first person view

Create a FirstPersonView.js script and add the bellow code. You can find examples for implementing camera controls on and examples as well. Attach this script to Player capsule.

var FirstPersonMovement = pc.createScript('firstPersonMovement');
var fpsCamera = null
var overChat = false
FirstPersonMovement.attributes.add('power', {
    type: 'number',
    default: 2500,
    description: 'Adjusts the speed of player movement'
});

FirstPersonMovement.attributes.add('lookSpeed', {
    type: 'number',
    default: 0.07,
    description: 'Adjusts the sensitivity of looking'
});
// initialize code called once per entity
FirstPersonMovement.prototype.initialize = function() {
    this.force = new pc.Vec3();
    this.eulers = new pc.Vec3();

    var app = this.app;
    app.mouse.on("mousemove", this._onMouseMove, this);
    app.mouse.on("mousedown", function () { 
        if(!overChat){
            app.mouse.enablePointerLock();
        }   
    },this);
    // Check for required components
    if (!this.entity.collision) {
        console.error("First Person Movement script needs to have a 'collision' component");
    }

    if (!this.entity.rigidbody || this.entity.rigidbody.type !== pc.BODYTYPE_DYNAMIC) {
        console.error("First Person Movement script needs to have a DYNAMIC 'rigidbody' component");
    }
};

// update code called every frame
FirstPersonMovement.prototype.update = function(dt) {
// If a camera isn't assigned from the Editor, create one
    if (!this.camera) {
        this._createCamera();
    }

    var force = this.force;
    var app = this.app;

    fpsCamera = this.camera;
    var forward = this.camera.forward;
    var right = this.camera.right;

    // movement 
    var x = 0;
    var z = 0;

    const convaiFormEl = document.getElementById("convai-input");
    // use w-a-s-d
    if(app.keyboard.isPressed(pc.KEY_A) && !(document.activeElement === convaiFormEl)){
         x -= right.x;
         z -= right.z;
    }

      if (app.keyboard.isPressed(pc.KEY_D) && !(document.activeElement === convaiFormEl)) {
        x += right.x;
        z += right.z;
    }

    if (app.keyboard.isPressed(pc.KEY_W) && !(document.activeElement === convaiFormEl)) {
        x += forward.x;
        z += forward.z;
    }

    if (app.keyboard.isPressed(pc.KEY_S) && !(document.activeElement === convaiFormEl)) {
        x -= forward.x;
        z -= forward.z;
    }

    // use direction from keypresses to apply a force to the character
    if (x !== 0 || z !== 0) {
        force.set(x, 0, z).normalize().scale(this.power);
        this.entity.rigidbody.applyForce(force);
    }

    // update camera angle from mouse events
    this.camera.setLocalEulerAngles(this.eulers.y, this.eulers.x, 0);
};

FirstPersonMovement.prototype._onMouseMove = function (e) {
    // If pointer is disabled
    // If the left mouse button is down update the camera from mouse movement
    if (pc.Mouse.isPointerLocked() || e.buttons[0]) {
        this.eulers.x -= this.lookSpeed * e.dx;
        this.eulers.y -= this.lookSpeed * e.dy;
    }

    const convaiChat = document.getElementById("convai-chat")
    convaiChat.addEventListener("mouseover",()=>{
    overChat = true;
    })

    convaiChat.addEventListener("mouseout",()=>{
    overChat = false;
    })
};

FirstPersonMovement.prototype._createCamera = function () {
    // If user hasn't assigned a camera, create a new one
    this.camera = new pc.Entity();
    this.camera.setName("First Person Camera");
    this.camera.addComponent("camera");
    this.entity.addChild(this.camera);
    this.camera.translateLocal(0, 0.5, 0);
};