Overview

Convai's Unity SDK provides you with all the tools you need to integrate conversational AI into your Unity projects. Convai offers specialized NLP-based services to build intelligent NPCs for your games and virtual worlds. Our platform is designed to seamlessly integrate with your game development workflow, enhancing the interactivity and depth of your virtual environments.

Key Features

• Conversational AI: Leverage advanced NLP capabilities to create NPCs that can understand and respond to player input in natural, engaging ways.

• Intelligent NPCs: Build characters with dynamic dialogue and behaviors that adapt to player actions and the game world.

• Easy Integration: Our SDK is designed for quick and simple integration with your Unity projects, allowing you to focus on creating compelling gameplay experiences.

• Cross-Engine Support: In addition to Unity, Convai supports other popular game engines, ensuring broad compatibility and flexibility for your development needs.

This is the Core version of the plugin. It has a sample scene for anyone to get started. This version of the plugin only contains the basic Convai scripts and Character Downloader.

Visit convai.com for more information and support.

Quick Setup Tutorial

Setting up Unity Plugin

In the Menu Bar, go the Convai > API Key Setup.

Go to convai.com, and sign in to your Convai account. Signing in will redirect you to the Dashboard. From the dashboard, grab your API key.

Enter the API Key and click begin.

This will create an APIKey asset in the resources folder. This contains your API Key.

Open the demo scene by going to Convai > Demo > Scenes > Full Features

Click the Convai NPC Amelia and add the Character ID (or you can keep the default character ID). You can get the character ID for your custom character from this page Create Character. Now you can converse with the character. The script is set up so that you have to go near the character for them to hear you.

Now you can test out the Convai Demo Scene and talk to the character present there. Her name is Amelia and she loves hiking!

You can open the Convai NPC Script to replicate or build on the script to create new NPCs.

Settings Panel consists of two main sections.

• Audio Settings

• Interface Settings

Audio Settings

Microphone Settings

The Microphone Settings section is primarily for troubleshooting and testing the microphone when using the Convai plugin.

• In the Input section, you can view the microphones connected to your computer and select the desired one.

• In the Test Input field, you can record your voice using the selected microphone in the Input section. After clicking Stop, you can listen to the recorded voice and observe the sound levels.

Interface Settings

Appearance

The first setting that greets us here is the Appearance setting.

In the Appearance section, you can switch between Transcript UI designs.

There are three Transcript UI options:

• ChatBox

• QuestionAnswer

• Subtitle

Display Name

The second section in Interface Settings is the Display Name section. This section allows you to change how the user's name appears in the Transcript UI.

Notifications Checkmark

The last section in Interface Settings is the Notifications Checkmark.

Convai sometimes displays notifications on the screen to inform the user. If you want to disable these notifications, you can click the checkbox here. ( If the box is green, it's active. If empty, it's inactive )

For more information about notifications, you can refer to this link.

Convai offers comprehensive transcript and voice support for a wide range of languages. To facilitate seamless integration, our Unity plugin comes with a custom TextMeshPro (TMP) package, which includes essential fonts and required settings for major languages.

Setup

To implement these language-specific features in your project:

1. Navigate to the Convai Setup Window within Unity.

2. Locate the Package Management section.

3. Click on the "Convai Custom TMP Package" button.

Once installed, just import the character for which you require the language support, talk with it and the font will automatically render in the transcript.

For now, we provide fonts for these languages:

• Arabic

• Japanese

• Korean

• Chinese

RTL Support

We also provide support for Right-to-Left languages, like Urdu, Persian and Arabic through our Chat UIs. So, for example, if you talk with an Arabic character or if the character's name is in Arabic, the text will automatically enable the RTL feature provided by unity to reflect proper transcripts.

If the Lip-sync from characters are either not visible or are very faint, if could be a result of character's animations overriding the blendshape changes made by the script. We recommend deleting the relevant components in the animation dopesheet.

If you see the microphone indicator turning on in the top left corner but no user transcript in the chat UI and the character's response doesn't seem coherent to what you said, then it is likely that the game or Unity is not accessing the correct microphone or does not have sufficient microphone privilege. To fix this, please follow along.

Check Notification System page to learn more about various issues related to microphone

You can point at Interactable Objects and Characters and ask your characters about them.

To enable this, simply drag and drop the Convai Crosshair Canvas prefab into the scene.

Unity Version​

The minimum supported Unity version is 2022.3.x. Earlier versions may not be compatible.

Supported Platform​s

Disabling Assembly Validation

We will create a simple scenario where the character welcomes the player and asks them about their evening or morning based on the player's time of day.

Step 1

Activate the Narrative Design for your character in the Playground. Then, create a new Section.

Step 2

In the Objective section of the new Section, add the following text:

The time of day currently is {TimeOfDay}. Welcome the player and ask him how his {TimeOfDay} is going.

Step 3

Now, let’s back to Unity and make the necessary adjustments. Click on your NPC.

Click the Add Component button and add the Narrative Design Key Controller Component.

Step 4

In the Name field, enter TimeOfDay. In the Value field, specify the corresponding value for that variable, which could be Morning, Evening, or anything else you choose.

Our plugin has various scripts and dependencies that use Newtonsoft Json. If Newtonsoft Json is missing from the plugin, it could lead to a large number of errors as shown below:

Ensure that NewtonSoft.Json is present in your packages. Go to your project folder.

Then navigate to Packages folder. In the Packages folder. Click on manifest.json. A json file containing the project dependacies should open up.

Add the Newtonsoft Json Package on top.

    "com.unity.nuget.newtonsoft-json": "3.2.1",

The final manifest.json should look like this.

{  
    "dependencies": {
        "com.unity.nuget.newtonsoft-json": "3.2.1", 
        "com.unity.animation.rigging": "1.1.1",
        "com.unity.ide.rider": "3.0.16",
        "com.unity.ide.visualstudio": "2.0.16",
        "com.unity.ide.vscode": "1.2.5",
        "com.unity.test-framework": "1.1.33",
        "com.unity.textmeshpro": "3.0.6",
        "com.unity.timeline": "1.6.4",
        .
        .
        .
    }
}

How to Change the Talk Button or Any Input?

1. Double click on the "Controls" asset in your project tab.
2. You can setup multiple control schemes for different devices here, currently we have it for PC (Keyboard & Mouse) and Gamepad. For mobile, we have provided joystick and buttons, which are mapped to Gamepad controls for functionality, but you can directly add touchscreen and use its different features to trigger an Input Action. You can also add your own control scheme if you want support for a different device by clicking on "Add Control Scheme".

1. Find the Input Action you want to change in the above window. If you want to add a new Input Action, refer to the other section in documentation. In this case, we selected "Talk Key Action" to change the talk button. Click on "T [Keyboard]". In the Binding Properties window, click on the " T [Keyboard] " button in the Path field.

2. Press the " Listen " button in the top left of the opened window. If you prefer, you can choose your desired input from the categories below.

1. Press the key you want to assign and it will be reflected in the control asset.

How to Add a New Input Action?

1. First, go to the controls asset mentioned above and use the add button to create an input action. For this example, we will call it interact and provide it the binding with [E] button.

2. Then, click on the <No Binding> item to setup binding for this action. As before, you can use the listen button (has a UI bug for Windows but works) or you can select the key from dropdown. After selecting the binding (we will select [E] key for this), don't forget to press on the Save Asset option on the top menu.
3. You will now get an error saying ConvaiInputManager does not implement OnInteract. We need to implement this. Open the " ConvaiInputManager.cs " script to do so. ( " Convai / Scripts / Runtime / Core / ConvaiInputManager.cs " )
4. Your IDE might suggest you to implement missing members. If it doesn't we can manually write the OnInteract function like in the last figure shown. You receive a callback context which shows which frame input started, performed or got cancelled which you can use for different purposes. And that's it the error should be gone and you are good to go!

This Unity tutorial demonstrates how to implement NPC-to-NPC conversations in 3D virtual environments using Convai's NPC2NPC feature. This functionality enables two NPCs to engage in dynamic, low-latency dialogues on predetermined topics, while allowing players or other game entities to interrupt and interact with them. Thus, this feature can create more engaging and interactive game worlds with use cases like storytelling, tutorial sequences or quest initiation

Check out this demo and tutorial on how to build this in Unity. We are also releasing the sample project source code along with the tutorial to get you started.

Download the free Sample project files:

Unity Asset Store plugin link:

Character Details

Name: Christina Smith ID: 64247ac6-74f8-11ef-be3e-42010a7be011

Name: Kevin Shaw ID: 7ffaee42-74f8-11ef-8980-42010a7be011

Resources and References for your help

NPC2NPC Walkthrough:

Unity Factory Scene HDRP:

Sign up at Convai to get started. For any questions or bug reports, please visit the Convai Developer Forum where our community and support team will be happy to assist you.

To import your custom characters into your Convai-powered Unity project, you will first need to bring your model into your project. The model needs at least two animations: one for talking and one for Idle.

Prerequisites

When you want to set up your custom character with Convai, you will need your character model and two animations: Idle and Talking.

Create an animator controller with the two animations that looks like this. You should also add a 'Talk' Boolean to ensure that you can trigger the animation. Here is a YouTube tutorial on how to set up an animator controller. This is the bare minimum animator setup that you need to do.

Step 1: Add Animator to your custom character

Select your character from the Hierarchy and Add Animator Component

Convai Plugin ships with two pre-made animation controller, you can choose these controllers or can assign your custom controller, whatever fits your need. For this demo we are going with Feminine NPC Animator

Step 2: Adding a Trigger Volume

With your custom character selected, add a Collision shape of your choice, for this demo we are going with a Capsule Collider

We will make this Collider a trigger, for this we will enable the Is Trigger option in the inspector panel

We will adjust the Center, Radius and Height of the collider such that it fits our character

Step 3: Add ConvaiNPC Component

With your Custom Character Selection add ConvaiNPC component. By doing so, your Game objectgame should look like this:

Copy your character's ID and name from Convai Playground and paste them here.

Now your Custom Character is all set to work with Convai Plugin.

Step 1) Open the New Scene window

You can open the new scene window by two ways, first by pressing Ctrl + N for windows or CMD + N for Mac on your keyboard, second way is to navigate to File -> New Scene

Step 2) Select Convai Scene Template

There will be many scene templates depending upon your project, but in this guide, we are interested in Convai Scene Template so select that and click on Create button.

Step 3) Save Created Scene

You can now save the newly created scene in the project at your desired location by either pressing Ctrl + S on Windows or CMD + S on Mac. Another method is to navigate to File -> Save Scene

This is open the Save Scene Window, choose your desired location, for this demo we will save it inside Demo folder, but you can save it anywhere in the assets directory.

Give your scene a name and then click on Save Scene button

Now you can import your Convai Character or your Custom Characters by following our complete guide on it

If the Lip Sync does not seem to cause any facial animations, even after removing all blendshapes from animations, then the following steps should help resolve the issue.

Select the Character and head to the Animator component.

Click the Avatar Field once to select the character's avatar in the Project window.

Select the Avatar and click Configure Avatar.

Select the Head option in the Mapping tab.

Select the Jaw Mapping and set it to None.

Finally scroll down and click Apply.

This will free the avatar's jaw mapping and allow the script to manipulate the Jaw bones.

Import RPM characters from the Convai Playground

This is how you can import characters from the Convai Playground into your Unity Project.

In the Menu Bar, go the Convai > Character Importer.

Enter the Character ID and click Import.

If you are unsure how to get the character ID, click the "How do I create a character?".

You can get the character ID from the Character Description.

The downloading will take a while. On successful download, you will see the character in the scene with the same GameObject as the character ID.

This character will automatically be set up with the basic Convai Setup including the ConvaiNPC Script and Out-Of-Box Animations.

If you are facing issues with the animations in your imported character, make sure to change the animation type of Ellen@IdleNew and Ellen@TalkingNew Animations in the Assets/Convai/Animations folder to Humanoid.

Now you are ready to set up the character with transcriptions.

Convai Playground

Step 1: Select your Character in which you want to enable Narrative Design

Step 2: Open Narrative Design in Convai Playground

Select the Narrative Design option from the side panel and create your narrative design

For this sample we have created the following Narrative design

You are all set to bring your character from Convai Playground to Unity, let's hope over to Unity to continue the guide

Unity Setup

Step 1: Add the Narrative Design Manager Component

Using Add Components Button in Convai NPC (Recommended Way)

1: Select your Convai Character in the scene and look for ConvaiNPC component in the inspector panel. Click on Add Components button

2: Select Narrative Design Manager checkbox and then click on Apply Changes button

Using Unity Inspector

1: Select your Convai Character and find Add Component button in the inspector panel

2: Search for Narrative Design Manager in the search box and select it

Step 2: Setup the Narrative Design Component

After adding the Narrative Design Component, you will be able to be the following component

Getting to know the Narrative Design Component

Expanding the section event, you will see two unity events you can subscribe to, one is triggered when section starts, and another one is triggered when section ends

Getting to know about Section Triggers

Section triggers are a way to directly invoke a section in narrative design and can be used to jump to a different section in your narrative design

Step 1: Select the game object you want to make a trigger, in this example we have selected a simple cube, but it's up to your imagination.

Step 2: Add Narrative design Trigger from Add Component menu by searching for it

Step 3: Make the collider a trigger.

Step 4: Assign your Convai NPC to Convai NPC field

Now you can select from the "Trigger" dropdown which trigger should be invoked when player enters this trigger box.

We have added a way for you to manually invoke this trigger also, you can use InvokeSelectedTrigger function to invoke the trigger from any where

Invoke Trigger from any script

You can use this code block as a reference to invoke the trigger from anywhere

if(convaiNPC.TryGetComponent(out NarrativeDesignTrigger narrativeDesignTrigger))
{
    //Optional message parameter if you want to send some message while invoking
    //the trigger 
    string message = "Player has collected enough resources";
    narrativeDesignTrigger.InvokeSelectedTrigger(message);
}

The Convai plugin comes with default notifications, totaling four. Here they are:

Notifications

Not Close Enough to the Character

Appears when you press the talk button but there is no active NPC nearby.

Talk Button Released Early

Appears if you release the talk button in less than 0.5 seconds.

Microphone Issue Detected

Appears when the recorded audio input level is below the threshold.

Connection Problem

Appears when there is no internet connection upon launching the application.

How to Add Your Own Notification?

Adding your custom notification is straightforward.

Let's go through the steps to add a " CharacterStartedListening" notification as an example.

1. Open the script "Convai/Scripts/Notification System/Notification Type.cs." This script stores Notification Types as enums. Give a name to your desired Notification type and add it here.

1. Right-click on "Convai / Scripts / Notification System / Scriptable Objects" and select "Create > Convai > Notification System > Notification" then create a "Notification Scriptable Object".

1. Name the created Notification Scriptable Object. Click on it, and fill in the fields in the Inspector as desired.

1. Add the created Notification Scriptable Object to "Convai/Scripts/Notification System/Scriptable Objects" under "Convai Default Notification Group" (details of Notification Group here****).

1. Your Notification is now ready. The last step is to call this Notification from where you need it. For example, if you created the " CharacterStartedTalking " Notification, find the location where your character listens and write the code.

NotificationSystemHandler.Instance.NotificationRequest(NotificationType.CharacterStartedListening);

1. Replace the parameter with the NotificationType you created. (For our example, NotificationType.CharacterStartedListening)

2. Ensure that the Convai Notification System is present in your scene. (accessible from "Convai/Prefabs/ Notification System")

All steps are complete, and you're ready to test!🙂✅

Notification Scriptable Object

This Scriptable Object stores information about a Notification

• Notification Type

• Notification Icon

• Notification Title

• Notification Message

To create a new Notification Scriptable Object, right-click anywhere in the Project Window and select "Create > Convai > Notification System > Notification"

Notification Group Scriptable Object

This Scriptable Object stores Notification Scriptable Objects as groups. When a Notification is requested, it searches for the Notification using the specified Notification Group in the Convai Notification System prefab's Notification System Handler script.

You can create different Notification groups based on your needs. Note: If your referenced Notification Group does not have the Notification you want, that Notification won't be called.

The Convai Default Notification Group has four Notifications, but you can add more or create a new group with additional notifications.

Lip Sync System

Convai sends Visemes or Blend Shape Frame from back-end depending upon the face model the developer chooses to use and when returned Convai SDK out of the box extracts and parses it and provides it to the Convai LipSync Component, after which the component relies on the SkinMeshRenderer's Blendshape Effectors and Bone Effectors to give Convai powered NPC's realistic lipsync.

Components of LipSync System

Viseme Effector List

This is where the developer will tell the Convai SDK, which index of Blendshape Array will be effector how much from which value. To better explain its working let's understand it with a diagram.

Here, it is saying that whatever value is coming from the server will affect Blendshape at the 116th index by 0.2 multipliers and Blendshape at the 114th index by 0.5 multipliers. The engine representation of this would look something like this.

So, you can make your own Effector list or use one of the many that we ship in the SDK.

How to Create your own Viseme Effector List

Right click inside project panel and head over to Create > Convai > Expression > Viseme Skin Effector which will create a Viseme Effector List Scriptable Object and now you can define your own values.

Viseme Bone Effector List

This is where developer will tell the Convai SDK, how much each value coming from the server will affect the rotation of the bone. To better explain its working let's understand it with a diagram.

Here, bone's rotation will be affected by the values coming from the server multiplied by the values in effects. For example, for TH the value will affect the bone's rotation by a 0.2 multiplier and etc. The engine representation of this would look something like this.

So, you can make your own Bone Effector list or use one of the many that we ship in the SDK.

We use this formula to calculate the rotation

UpdateJawBoneRotation(
new Vector3(
        0.0f, 
        0.0f, 
        -90.0f - CalculateBoneEffect(FacialExpressionData.JawBoneEffector) * 30f
    )
);
UpdateTongueBoneRotation(
new Vector3(
        0.0f,
        0.0f,
        CalculateBoneEffect(FacialExpressionData.TongueBoneEffector) * 80f - 5f
    )
);

How to Create Your Own Viseme Bone Effector List

Right click inside the project panel and head over to Create > Convai > Expression > Viseme Bone Effector which will create a Viseme Bone Effector List Scriptable Object and now you can define your own values.

Convai Lipsync Component

When you attach this component to your Convai Character, you will see something like this.

Let's learn what these learns are

1. Facial Expression Data

   1. Head | Teeth | Tongue

      1. Renderer: Skin Mesh Renderer which corresponds to that specified part of the body

      2. Viseme Effectors List: How the SkinMeshRenderer's Blendshape will be affected by values coming from server.

   2. Jaw | Tongue Bone Effector

      1. How much of Bone's rotation will be affected by values coming from server?

   3. Jaw | Tongue Bone

      1. Reference to the bone which controls jaw and tongue respectively

2. Weight Blending Power

   1. Percentage to interpolate between two frames in late update.

3. Character Emotions

   1. Learn More about Character Emotions here Character Emotion

Steps to add Lipsync to your Convai Character

Now you can configure the Component according to your custom configuration or use one of the many Presets Convai ships with the SDK

Now your lipsync component would be ready to use in your application.

Setting Up Action Configurations

1. Select the Convai NPC character from the hierarchy.

2. Scroll down to the ConvaiNPC script attached to your character.

3. Click the "Add Component" button.

1. Use the checkbox to add the action script to the NPC Actions.

2. Click "Apply Changes" to confirm.

Pre-defined Actions

Convai offers predefined actions for a quick start.

1. Click the "+" button to add a new action.

2. From the dropdown menu, select "Move To."

1. Enter the action name as "Move To" (the name doesn't have to match the action choice name).

2. Leave the Animation Name field empty for now.

Repeat these steps to add more actions like "Pickup" and "Drop" etc.

Adding an Object in the Scene

1. Add any object into the scene—a sphere, a cube, a rock, etc.—that can be interacted with

2. Resize and place the object in your scene.

Adding the Convai Interactables Data Script

• Create an empty GameObject and name it "Convai Interactables."

• Attach the Convai Interactables Data script to this GameObject.

• Add characters and objects to the script by clicking the "+" button and attaching the corresponding GameObjects.

• Add the "There" object in Objects list, so that we can use the Dynamic Move Target indicator.

Setting Up NavMesh

To ensure your NPCs can navigate the scene:

1. Bake a NavMesh for your scene if you haven't already:

   • Go to Window > AI > Navigation.

   • In the Navigation window, under the Bake tab, adjust the settings as needed.

   • Click "Bake" to generate the NavMesh.

2. Ensure that the NPC character has a NavMeshAgent component:

   • If not already attached, click "Add Component" and search for NavMeshAgent.

   • Adjust the Agent Radius, Speed, and other parameters according to your NPC's requirements.

Adding a Dynamic Move Target Indicator

To visually indicate where your NPC will move:

• Create a new empty GameObject in the scene and name it accordingly or use the pre-made prefab named Dynamic Move Target Indicator.

• Link this Move Target Indicator to your NPC's action script so it updates dynamically when you point the cursor to the ground and ask the NPC to move to "There".

Test the Setup

1. Click "Play" to start the scene.

2. Ask the NPC, "Bring me the Box."

3. If setup properly, the NPC should walk upto the box and bring it to you

Adding Custom Actions to Your Unity NPC in Convai

Introduction

Make your NPC perform custom actions like dancing.

Action that Only Requires an Animation

1. Locate the dance animation file within our plugin.
2. Incorporate this animation into your NPC's actions.

Setting Up the Animator Controller

1. Open the Animator Controller from the Inspector window.

2. Drag and drop the dance animation onto the controller, creating a new node named "Dancing."

Adding custom Animation Action

1. Go to the Action Handler Script attached to your Convai NPC.

2. Add a new action named "Dancing."

3. In the Animation Name field, enter "Dancing" (it must exactly match the Animator Controller node name).

4. Leave the enum as "None."

Testing the Custom Action

1. Click "Play" to start the scene.

2. Instruct the NPC, "Show me a dance move," and the NPC should start dancing.

Creating Complex Custom Actions in Unity with Convai: Throwing a Rock

Introduction

Adding advanced custom actions, such as a throw action, to your NPC.

Animation Requirement

2. Import it into Unity.

Setting Up the Animator Controller

Action Handler Script Setup

1. Add the "Throw" enum to the script.
2. In the "Do Action" function, add a switch case for the throw action.
3. Define the "Throw()" function.

Adding the Throw Action

1. Add a new action named "Throw" and select the "Throw" enum.

2. Leave the animation name field empty.

Adding the Object (Rock) to the Convai Interactables Data script

1. Add any rock prefab into the scene.

2. Add the rock to the Convai Interactable Data script.

Adding a location to Convai Interactables Data script

1. Add a stage/new location in the ground of the scene.

2. Add that new location game object in the Convai Interactable Data.

Testing the Complex Action

1. Click "Play" to start the scene.

2. Instruct the NPC, "Pick up the rock and throw it from the stage."

3. If everything is set up properly, the NPC should pick up the rock and throw it from the stage.

1. iOS/iPadOS

2. Virtual Reality

3. Mixed Reality

4. Augmented Reality (Android/iOS)

5. WebGL

6. Universal macOS applications

Unity Version

Skills and Knowledge

Before integrating the Convai SDK, you should be comfortable with the following:

1. Importing Packages: Know how to import external packages into a Unity project.

2. Unity Editor: Be proficient in navigating the Unity Editor interface.

3. Animations: Understand how to add and handle animations for assets.

4. Programming in C#: Have a basic experience programming Unity scripts in C#.

5. Script Integration: Be capable of adding scripts to a game object.

6. Building and Deployment: Know how to build and deploy an application to your chosen platform.

Having these skills will ensure a smooth integration and optimal use of the Convai Unity SDK in your projects.

Size Constraints

iOS browsers impose strict limitations on the size of WebGL builds. These constraints are primarily due to:

• Memory Limits: iOS devices have limited available memory for web applications, which can affect the performance and feasibility of running large WebGL builds.

• Browser Storage Quotas: Safari and other iOS browsers restrict the amount of data that can be stored locally. This includes caching and Indexed DB, which are often used to store assets for WebGL builds.

Key Limitations

• Maximum Downloadable Asset Size: iOS browsers may restrict the size of individual downloadable assets. Large assets might fail to load, causing the application to break.

• Total Build Size: The total size of all assets combined should ideally be kept under 50-100 MB for smooth performance. Exceeding this limit can lead to crashes or extremely slow loading times.

• Memory Usage: iOS devices typically have less RAM available compared to desktop environments. High memory usage by WebGL builds can result in frequent browser crashes.

Browser Compatibility

• Safari: The default browser on iOS, Safari, is generally the best option for WebGL builds, but it still has significant limitations compared to other desktop browsers.

In a typical application integrating with the Convai API, maintaining a consistent session ID across different sessions is crucial for providing a seamless user experience. This documentation outlines the best practices for storing and retrieving session IDs using Unity's PlayerPrefs, including detailed steps and example scripts.

Importance of Session IDs

A session ID uniquely identifies a session between the client and the Convai server. Storing the session ID locally ensures that the same session ID is used across different sessions, which helps in maintaining context and continuity in interactions.

Storing Session IDs

When initializing a session, if a session ID is not available locally, it should be fetched from the server and then stored locally for future use. Here's how you can achieve this:

1. Fetch and Store Session ID: When initializing a session, check if a session ID is stored locally. If not, fetch a new session ID from the server and store it using PlayerPrefs.

public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
{
    // Retrieve stored session ID if it exists
    string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

    // If no session ID is stored, initialize a new one
    if (string.IsNullOrEmpty(sessionID))
    {
        sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

        // Store the new session ID locally
        if (!string.IsNullOrEmpty(sessionID))
        {
            PlayerPrefs.SetString(characterID, sessionID);
            PlayerPrefs.Save();
        }
    }

    return sessionID;
}

Retrieving Session IDs

When initializing your application, retrieve the stored session ID to ensure continuity in user interactions.

private async void Start()
{
    // Initialize session ID on start
    string characterID = "YourCharacterID"; // Replace with your actual character ID
    string sessionID = await InitializeSessionIDAsync("CharacterName", grpcClient, characterID);

    if (!string.IsNullOrEmpty(sessionID))
    {
        Debug.Log("Session ID initialized and stored: " + sessionID);
    }
    else
    {
        Debug.LogError("Failed to initialize session ID.");
    }
}

Example Class for Session Management

The following example class demonstrates how to manage session IDs using PlayerPrefs in a Unity project:

using System;
using System.Threading.Tasks;
using Convai.Scripts.Utils;
using Google.Protobuf;
using Grpc.Core;
using Service;
using UnityEngine;
using static Service.GetResponseRequest.Types;

public class SessionManager : MonoBehaviour
{
    public ConvaiService.ConvaiServiceClient grpcClient;

    private void Start()
    {
        // Initialize session ID on start
        InitializeSession("CharacterName", grpcClient, "YourCharacterID");
    }

    private async void InitializeSession(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = await InitializeSessionIDAsync(characterName, client, characterID);

        if (!string.IsNullOrEmpty(sessionID))
        {
            Debug.Log("Session ID initialized and stored: " + sessionID);
        }
        else
        {
            Debug.LogError("Failed to initialize session ID.");
        }
    }

    public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

        if (string.IsNullOrEmpty(sessionID))
        {
            sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

            if (!string.IsNullOrEmpty(sessionID))
            {
                PlayerPrefs.SetString(characterID, sessionID);
                PlayerPrefs.Save();
            }
        }

        return sessionID;
    }
}

Detailed Steps for Session Management

1. Initialize Session: Call InitializeSessionIDAsync to check if a session ID is stored. If not, fetch and store it.

2. Store Session ID: Use PlayerPrefs.SetString(characterID, sessionID) to store the session ID locally.

3. Retrieve Session ID: Use PlayerPrefs.GetString(characterID, string.Empty) to retrieve the stored session ID.

4. Use Session ID: Pass the session ID to your Convai API calls to maintain session continuity.

Best Practices

• Error Handling: Ensure proper error handling when fetching and storing session IDs.

• Security: Consider encrypting sensitive information stored in PlayerPrefs.

• Performance: Use asynchronous methods to avoid blocking the main thread when fetching session IDs.

Version 3.2.0 (Current)

Released: October 31, 2024

New Features

• Implemented Dynamic Config Feature:

  • This feature allows you to dynamically pass variables to NPCs. For example, you can update NPCs with the player’s current health, inventory items, or information about the world, enhancing interactivity and immersion.

• Implemented Narrative Design Keys:

  • This feature enables dynamic variable passing within the Narrative Design section and triggers. For instance, you can use placeholders like {TimeOfDay} to create personalized dialogues, such as "Welcome, player! How is your {TimeOfDay} going?"

• Added MR Demo Scene

• Added MR Automatic Installation and Manual Installation

• Added Convai XR Package (compatibility with Meta SDK and other XR SDKs provided)

Improvements

• Added Long Term Memory API(s) to View and Delete Speaker ID(s)

• Improved VR Manual Installation

• Improved Custom Package Installation

Hotfix

• Minor Bug Fixes

Version 3.1.2

Released: September 16, 2024

Hotfix

Minor Bug Fixes

Version 3.1.1

Released: September 12, 2024

Hotfix

Fixed NPC2NPC response delay

Version 3.1.0

Released: Aug 28, 2024

What's Changed

• New Convai Setup Editor Window

• Long term memory (beta) integration in Unity SDK

• ChatBox UI revamp with RTL support

• Input system revamp and Mobile UI improvements

Improvements

• UI Improvements

  • Revamped ChatBox UI

  • Improved mobile UI

  • Implemented chat disabling feature

  • Added usage limit exceeded notification

  • Dialog box added for no API Key scenario

• API and Backend

  • Refactored ProcessUserQuery for better transcript handling

  • Implemented fuzzy matching for Action System

  • Ready Player Me and CC_Tools automatic import process

• Character and Animation

  • Updated Character Importer Pipeline

  • Added OVR effectors for RPM characters

  • Fixed animators for all characters

  • Provided Weight Multiplier for LipSync user preference

  • RPM Characters will have Lipsync added when imported

Bug Fixes and Improvements

• Bug Fixes

  • Fixed character resuming dialogue after toggle

  • Fixed section deletion issue in NarrativeDesignManager

  • Fixed layer issues

  • Optimised Convai LipSync

• Developer Tools and Workflow

  • Improved Convai Logger System

  • Updated namespace and formatting for all editor scripts

  • Removed CC Tools Folder and other temporary/junk files

  • Added "Update Triggers" button to NarrativeDesignTrigger inspector

  • Implemented approximate string matching for actions system

• Miscellaneous

  • Added PlayerDataHandler and PlayerDataSO

  • Updated NPC positions and topics in demo scenes

  • Fixed Convai logo in Convai Setup window

Version 3.0.1

Released: Jun 21, 2024

Bug Fixes and Improvements

• Fix macOS TMP UGUI render issue in demo scene

• Prefab missing animator

• Updated ActiveNPC layer check logic

Version 3.0.0

Released: Jun 13, 2024

What's Changed

NPC2NPC System

• Implemented NPC2NPC conversation flow system.

• Added handling for conversation interruptions and restarts.

• Enhanced conversation history tracking and flow management.

Narrative Design

• Added Narrative Design-related files and trigger narrative section function.

• Refactored Narrative Design API, created new behavior trees for movement and added section change events.

• Folder Restructuring

  • Complete folder and scripts folder restructure

• Gender-Based Animator: Added gender-based Animator controller

• Feedback System

  • Implemented a feedback system with thumb icons and animations

  • Updated Transcript UI Prefabs with feedback buttons

• Convai Custom Packages: Updated Convai Custom Package Installer and added iOS DLL Downloader

• Scene Perception: Added feature to allow players to point at game objects and talk about them

Improvements

• Texture and Material Compression

  • Compressed Amelia and other images (POT and Crunch)

  • Updated image names and removed unused image assets

• UI Updates

  • Updated UI prefabs, including Transcript UI, Mobile UI, and Mobile QA UI

  • Transcript UIs and text updated

  • Updated logos and logo paths

• System Improvements

  • Refactored Lipsync system with added teeth support and implemented facial expression proto files

  • Updated ConvaiURPConverterPackage, burst and TMPro packages; Convai Custom Package Installer/Exporter

  • Updated NavMesh and NPC2NPC character rotation

  • Added new demo scene and RPM characters

  • Updated various demo scenes for consistency

• Microphone Manager: Updated Microphone Manager to a singleton class

• API Key Access: Simplified API Key access

• Convai Scene Template: Created new scene template and dynamic input system assigner

• Demo Scenes

  • Added NPC2NPC demo scene

  • Added Narrative Design demo scene

  • Added new demo scene with all features encompassed

  • "Convai Essentials" prefab for desktop and mobile

• Lipsync Overhaul

  • Overhauled lipsync system, added AR-Kit and Reallusion character support

  • Updated version and added various improvements to frame processing

• Input System

  • Added new input system pragma checks and Convai Character Layer

  • Simplified Input Manager and ensured future-proofing

Bug Fixes

1. Transcript UI Bug Fixes: Fixed bugs and improved system for Transcript UI character list

2. Microphone Permission: Fixed Android and iOS microphone permission issues

Version 2.1.0

What's Changed

• VR Support: Implement Virtual Reality features to create a fully immersive experience with the press of a button.

• AR Support: Integrate Augmented Reality capabilities, allowing characters and environments to interact with you in the real world with the press of a button.

• Settings Panel: Introduce a comprehensive settings panel that allows users to customize their experience.

• Microphone Test System: Incorporate a microphone testing feature to ensure optimal audio input quality.

• Notification System: Implement a robust notification system to inform users of in-game events - specifically microphone-based issues.

• Input Manager: Develop a custom input management system that supports various input devices such as keyboards, gamepads, and touchscreens using Unity's new Input System.

Bug Fixes and Improvements

• Fixed: Head Tracking Doesn’t Work Without Action Component issue fixed.

• Improvement: Added support for a customizable and dynamic Chatbox.

• Improvement: Improved Lip-Sync Smoothing and audio-visual synchronization.

• Improvement: Implement Action Events and Event Callbacks.

• Improvement: Improved Logging System.

• Improvement: Added ability to interrupt Character Response with Voice Interruption.

• Improvement: Improved mobile platform transcription UI.

Version 2.0.0

What's Changed

• Lip-sync: Integrate off-the-shelf Lip-sync for Reallusion and Oculus-based Characters.

• Text-in Voice-out: Chat with the character using text.

• Character Importer: Import Ready Player Me characters created on the Convai Playground.

• Feature Control System: Enable Convai features as needed through the Convai NPC component.

• Logging System: Have better control over what Convai information you see on the debug console.

• Enhanced player controller: Automatically triggers the characters when you focus on them and then deactivates them when your focus has shifted.

• URP Upgrader: Upgrade the Render Pipeline to Universal Render Pipeline with the URP Upgrader package (present in the Convai Folder).

• UI Improvements: Improved user experience with automatically fading UI canvas.

Bugs and Improvement

• Fixed: Unlocking the cursor will still cause the first-person camera to move around.

• Fixed: Exiting play mode before the character is done speaking will cause Unity to crash or not complete compilation.

• Fixed: Extra space between multiple chunks of text in the UI Text Fields.

• Fixed: Actions crashing the Android scene.

• Fixed: Empty responses from the server will not crash the game but only throw an error.

• Improvement: Smoothened Blinking.

• Improvement: Smoothened Gaze-Following-based Neck movement.

• Improvement: Plugin structure reorganization.

Common Issues (FAQ)

Q. I cannot see the Convai menu.

A. Please check if there are any errors in the console. Unity needs to be able to compile all the scripts to be able to display any custom editor menu options. Resolving all the console errors will fix this issue.

Q. There are a lot of errors on my console.

A. Primarily, two issues cause errors in the console that can stem from the Convai Unity Plugin. You can use the links below to fix them quickly.

1. Disable Assembly Validation

2. Missing Newtonsoft Json

Q. I am talking to the character, but I cannot see the user transcript and the character does not seem to be coherently responding to what I am saying.

A. This may indicate issues with the microphone. Please ensure that the microphone is connected correctly. You also need to ensure that the applications have permission to access the menu.

Microphone Permission Issues

Q. The animations for my characters are looking very weird.

A. The animation avatar that we are using might be incompatible with the character mesh. Fixing that can solve the issue.

Default Animations Incompatibility

Q. There are two Settings Panel Buttons in Mobile Transcript UI.

A. If you are using Unity 2021, unexpected prefab variant issues may arise. This is because Unity Mobile Transcript UIs are variants of the main transcript UI prefab. With changes in the Prefab system in Unity 2022, it works correctly in Unity 2022. If you are using Unity 2021, you may encounter issues with prefabs. You can remove the redundant Settings Panel Button to address this problem.

Q: The lipsync is very faint or not visible.

A: The animations that we are using may be modifying facial animations. Editing the animations to remove facial animations should fix any issues related to lipsync.

Animations have Facial Blendshapes

A: The script also needs the avatar to not be mapped to the jaw bone to be manipulate the jaw bones itself.

Jaw Bone in Avatar is not Free

Q: I'm facing security permission issues using the grpc_csharp_ext.bundle DLL inside the Unity Editor on MacOS

A: macOS's strict security measures can block certain external unsigned DLLs. To address this, you can manually allow the DLL in "Security & Privacy" settings, modify Gatekeeper's settings through Terminal, ensure correct file permissions for the DLL, check its settings in Unity, and update the Mac Configuration in Unity's Player Settings

macOS Permission Issues

Q: I'm not able to talk to my character after building my Unity project for macOS (Intel64+Apple Silicon builds), especially on Intel Macs

A: The issue is rooted in the grpc_csharp_ext.bundle used in Unity for networking. This DLL has separate versions optimized for Intel and Apple Silicon architectures. When trying to create a Universal build that serves both, compatibility problems arise, especially on Intel Macs. Presently, the best solution is to use Standalone build settings specific to each architecture.

Building for macOS Universal apps

Error Index

Follow this Table to navigate to our most common errors.

For any other issues, feel free to contact us on the Convai Developer Forum.

Convai Character emit character emotions when they interact with the player and these emotions help in making the character more human-like, we are starting to implement a system which you as a developer can use to make your game more interactive using the character emotions.

Whenever the character responds to the user, we send back a list of emotions to the SDK, which look something like this

For v0 of this system, we will only be sending the emotions, in future we will apply the facial expressions corresponding to each emotion which will make the character more interactive.

This Unity tutorial shows you how to add a Convai-powered Tour Guide character for your 3D virtual environments. Convai’s Narrative Design Feature lets creators add spatial anchors in the 3D environment which guides the 3D AI character to navigate the world while following step-by-step instructions. The character follows instruction prompts combined with the spatial anchors while conversing with the user in a contextual open-ended manner. This enables a whole set of use cases from onboarding, tour guides, companion characters, tutor characters, and many more. Check out this demo and tutorial on how to build this in Unity. We are also releasing the sample project source code along with the tutorial to get you started.

Download the free Sample project files:

Unity Asset Store plugin link:

Character Details

Name: Christina Smith ID: 84434252-3776-11ef-a746-42010a7be00e

Resources and References for your help

Narrative Design Walkthrough:

Unity Factory Scene HDRP:

Sign up at Convai to get started. For any questions or bug reports, please visit the Convai Developer Forum where our community and support team will be happy to assist you.

Integrating Convai with MR: Quick and Flexible Setup Options

Setting up Convai for Mixed Reality (MR) is easier than you might think. With just a few clicks, you can get Convai integrated directly into your MR projects.

To give users flexibility and customization options, we offer two installation methods:

• Automatic Installation

• Manual Installation

Automatic Installation

The Automatic Installation method is ideal if you’re starting a new project and want to avoid dealing with MR packages or additional setup requirements. This method provides a fully Convai-integrated MR project right from the start, including the Convai MR Demo Scene for quick testing and exploration. This approach is particularly suitable for beginners.

Manual Installation

The Manual Installation method is designed for existing MR projects. With this method, you have full control over the setup, as no project settings are modified, no demo scenes are added, and no customizations are applied. Only Convai’s XR Unity Package will be integrated, giving you the freedom to use any MR SDKs without restrictions.

Choose Your Installation Method

Select one of the two installation methods and click on the corresponding setup section below to follow the instructions.

Convai Controller Scheme

If you ever get an error that looks like this, disable the Assemble Version Validation in Project Settings > Player > Other Settings.

Assembly 'Assets/Convai/Plugins/Grpc.Core.Api/lib/net45/Grpc.Core.Api.dll' will not be loaded due to errors: 
Grpc.Core.Api references strong named System.Memory Assembly references: 4.0.1.1 Found in project: 4.0.1.2.

Ensure that Assembly Validation is disabled in Project Settings > Player > Other Settings.

Restart the Unity project after unchecking the box should fix the issue.

Manual Installation Steps

Step 1

In the top menu, click on Convai. Then, select Custom Package Installer.

Step 2

In the Convai Panel that appears, click on Package Management and then select Install VR Package.

Step 3

A new window will appear prompting you to select your installation type. For this documentation, we will proceed with Manual Installation.

Step 4

The installation will start. This process will be completed quickly as only the ConvaiXR Package will be installed.

Step 5

Once installation is complete, the Convai Essentials - XR Prefab will be added to your scene.

This is the only GameObject required for Convai to run in your scene.

Step 6

The imported files can be found under Assets > Convai > ConvaiXR.

Step 7

The final step is to import your character into the scene.

If you need guidance on this, please refer to the relevant documentation here.

Manual Installation Steps

Step 1

In the top menu, click on Convai. Then, select Custom Package Installer.

Step 2

In the Convai Panel that appears, click on Package Management and then select Install MR Package.

Step 3

A new window will appear prompting you to select your installation type. For this documentation, we will proceed with Manual Installation.

Step 4

The installation will start. This process will be completed quickly as only the ConvaiXR Package will be installed.

Step 5

Once installation is complete, the Convai Essentials - XR Prefab will be added to your scene.

This is the only GameObject required for Convai to run in your scene.

Step 6

The imported files can be found under Assets > Convai > ConvaiXR.

Step 7

The final step is to import your character into the scene.

If you need guidance on this, please refer to the relevant documentation here.

Interacting with the UI may vary depending on the SDK you are using. For popular frameworks like XR Interaction Toolkit and Meta SDK, you can follow the documentation below. We recommend checking the relevant documentation for other SDKs.

To make the Settings Panel interactable in XR environments, where you can test your microphone at runtime or change the appearance of the Chat UI in Convai, follow these steps:

Meta SDK

Step 1

Right-click on the Convai Settings Panel.

Step 2

Select Interaction SDK and then click on Add Ray Interaction to Canvas.

Step 3

If a warning appears in the new window, click the Fix button.

Step 4

In the Settings section, choose Everything. Then click Create.

Step 5

To prevent the ISDK_RayInteraction from running while the Settings Panel is closed, drag it onto the Panel GameObject.

XR Interaction Toolkit

Step 1

Click on the Convai Settings Panel.

Step 2

In the Inspector, click the Add Component button.

Add the Tracked Device Graphic Raycaster component.

If the default animations that ship with the animator look bugged such that the hand seems to intersect with the body, it could indicate an issue with the wrong animation avatar being selected.

You can easily fix that by heading to the character's animator component and assigning the correct animator to the Avatar field.

The correct animation will look something like this. The hands should not intersect the body.

We provide several UI options to display character and user's transcript out of the box that players can use with the Convai Plugin. You can use and customize these prefabs.

The ConvaiNPC and ConvaiGRPCAPI scripts look for GameObjects with Convai Chat UI Handler as a component, and send any transcripts to the script so that it can be displayed on screen.

Types of UI

ChatBox

Prefab Name: Convai Transcript Canvas - Chat

Both the user's and the character's transcripts are displayed one after other in a scrollable chat box.

Subtitle

Prefab Name: Convai Transcript Canvas - Subtitle

The user and character transcripts are displayed in the bottom like subtitles.

Question-Answer

Prefab Name: Convai Transcript Canvas - QA

The user's transcript is displayed in the top where as the character's transcript is displayed in the bottom.

Mobile Optimised UI Styles

Prefab Name: Convai Transcript Canvas - Mobile Subtitle

Identical to #subtitle UI. Includes a button that can be pressed and held for the user to speak. Ideal for portrait orientation of screen.

Prefab Name: Convai Transcript Canvas - Mobile QA

Prefab Name: Convai Transcript Canvas - Mobile Chat

Convai Chat UI Handler Component

Functions to Know

This is a scriptable object which is made automatically after you hit play in the editor with Convai SDK installed and in a Scene where Convai Base Scene Essentials Prefab is present

Default Player Name

You can provide a default name of your players.

Player Name

Current name of your player, out of the box if you use our settings panel, we keep it updated automatically, if you are using some custom logic, it will be your responsibility to keep it updated, as our transcript UI use this name to show it in UI

Speaker ID

Speaker ID for the player. Please note that Speaker ID is directly linked with your API key, so for each API key there should be a unique speaker ID associated with it. We handle the creation of the Speaker ID when it's not found in the Player Prefs if the Boolean is set to true.

Create Speaker ID If Not Found

This Boolean lets the SDK if it should create a unique Speaker ID for that Player Name if it is not found in the Player Prefs.

Buttons

Reset Data

It just makes the Player Name and Speaker ID fields empty.

Copy Data

Copies the data into system buffer so you can paste it anywhere for debugging purpose

Player Pref Settings Button

1. Load: Loads the Player Name and associated Speaker ID from the player Prefs

2. Save: Saves the Player Name and associated Speaker ID from the player Prefs

3. Delete: Deletes the Player Name and associated Speaker ID from the player Prefs

How to maintain the Player Data

Convai provides a pre-made component which you can add to any GameObject to make the PlayerDataContainer work out of the box.

Choose an existing GameObject or create a new GameObject in the scene and add the ConvaiPlayerDataHandler component to your chosen GameObject and it should start working

Optional Step

You can also create the required Scriptable Object by going to Assets > Convai > Resources and right clicking in the project panel and navigating to Create > Convai > Player Data and name it ConvaiPlayerDataSO

Overview

When building Unity projects for macOS, developers may encounter issues with microphone permissions, particularly when targeting both Intel and Apple Silicon Macs. This document outlines the problem, symptoms, causes, and solutions to help ensure successful access to the microphone across different Mac architectures.

Problem Description

Some users have reported that while building macOS universal apps, Apple Silicon Macs handle microphone permissions without issue, while Intel Macs may fail to access the microphone due to differences in architecture. This can result in a lack of microphone response, DLL not found Exceptions, error messages, potential application crashes, or no audio input being detected.

Cause

The issue stems from the grpc_csharp_ext.bundle, which is crucial for networking in Unity projects. There are separate versions of this DLL for Intel and Apple Silicon architectures, and they cannot be easily merged or applied universally. The grpc library currently lacks dedicated support for resolving these dll issues in Unity.

Solutions

Standalone Build Settings

• For Intel Macs: Use Standalone builds targeted specifically for the Intel architecture to ensure compatibility.

• For Apple Silicon Macs: Prefer Standalone builds for the ARM64 framework for optimal performance, although Universal builds are also an option.

New Procedure for Universal Builds from Intel Macs

After completing a Universal build on an Intel Mac, you must manually update the grpc_csharp_ext.bundle to ensure proper functionality. Follow these steps:

1. Locate the .app file generated by the build process.
2. Right-click on the .app file and select "Show Package Contents."
3. Navigate to the Contents/Plugins folder within the package.
4. Replace the contents of this folder with the components from the provided plugin folder (link to plugin folder).

Important: The grpc_csharp_ext.bundle may not be included correctly in the final build when built from an Intel Mac. Always verify that the Plugins folder in the build contains the correct DLLs. If there is any confusion or the DLLs are missing, replace or add the contents of the Plugins folder with the one provided by us.

Conclusion

Building for macOS requires careful consideration of the distinct Intel and Apple Silicon architectures. The current best practice is to use Standalone build settings tailored to the specific architecture of the target Mac. As we progress, we will have a more integrated solution for managing DLLs that will simplify the development process for universal macOS applications.

Integrating Convai with VR: Quick and Flexible Setup Options

Setting up Convai for Virtual Reality (VR) is easier than you might think. With just a few clicks, you can get Convai integrated directly into your VR projects.

To give users flexibility and customization options, we offer two installation methods:

• Automatic Installation

• Manual Installation

Automatic Installation

The Automatic Installation method is ideal if you’re starting a new project and want to avoid dealing with VR packages or additional setup requirements. This method provides a fully Convai-integrated VR project right from the start, including the Convai VR Demo Scene for quick testing and exploration. This approach is particularly suitable for beginners.

Manual Installation

The Manual Installation method is designed for existing VR projects. With this method, you have full control over the setup, as no project settings are modified, no demo scenes are added, and no customizations are applied. Only Convai’s XR Unity Package will be integrated, giving you the freedom to use any VR SDKs without restrictions.

Choose Your Installation Method

Select one of the two installation methods and click on the corresponding setup section below to follow the instructions.

Convai Controller Scheme

Automatic Installation Steps

Step 1

In the top menu, click on Convai. Then, select Custom Package Installer.

Step 2

In the Convai Panel that appears, click on Package Management and then select Install VR Package.

Step 3

A new window will appear prompting you to select your installation type. For this documentation, we will proceed with Automatic Installation.

Step 4

In the next window, carefully read the setup instructions, warnings, and details of the changes that will be applied.

Step 5

When ready to proceed, click Yes, proceed.

Step 6

The installation process will begin, taking approximately 5 minutes. The duration may vary depending on your computer's performance.

Step 7

Once the installation is complete, it’s time to open the demo scene.

• Navigate to Assets > Convai > ConvaiXR > ConvaiVR > Scenes and open Convai Demo - VR.

Step 8

If you don’t have the TextMeshPro package installed, you’ll need it to display text properly in the demo scene. To install it:

• Go to Window > TextMeshPro > Import TMP Essential Resources.

• In the Unity Package Import window that appears, click Import.

Step 9

You can build your project by going to File > Build Settings.

Step 1: Setting up Convai NPC

1. Go to your Convai NPCs:

   • Select the NPCs you want to include in the conversation.

2. Enable Group NPC Controller:

   • Click on the Group NPC Controller checkbox in the inspector panel.

   • Click Apply Changes to add the group NPC controller script.
3. Create or Find the Speech Bubble Prefab:

   • Create a new speech bubble prefab or use the one provided in the Prefabs folder.
4. Attach Required Components:

   • Add the speech bubble prefab and the player transform (optional, defaults to the main camera if not provided).

   • Set the conversation distance threshold variable (set it to zero to disable this feature, meaning NPC to NPC conversations will always happen regardless of the player’s distance).
5. Add Relevant Components:

   • Add components like lip sync, eye and head tracking, character blinking, etc., to the Convai NPC.

Step 2: Setting up NPC Manager

1. Create an NPC To NPC Manager GameObject:

   • Add an empty GameObject and rename it to NPC to NPC Manager (optional).

2. Add the NPC2NPC Conversation Manager Script:

   • Attach the NPC2NPCConversationManager script to the GameObject.
3. Configure the NPC Group List:

   • In the NPC Group List, click on the + icon to add a new list element.

   • Add the NPCs you want to include in the group conversation.

   • Set the group discussion topic.
4. Post configuration of NPCs

   • Bring the NPCs close together

   • Play the to make sure everything is working as intended.

By following these steps you can set up and manage NPC to NPC conversations in your Convai-powered application. For further customization and integration, refer to the complete implementation code and adjust it as needed for your specific use case.

The Dynamic Config feature enables you to pass variables to NPCs in real time, allowing them to react dynamically to changes in the game environment. This can include the player’s current health, inventory items, or contextual world information, greatly enhancing interactivity and immersion.

Step-by-Step Guide to Setting Up Dynamic Config

First, Add the Dynamic Info Controller Component to your Convai NPC.

Create a new script or use an existing script to define a variable that will store a reference to the Dynamic Info Controller Component you added to your NPC.

Example: Passing Player Health to the NPC

• Initialize the Dynamic Info: In the script’s Start method, call the SetDynamicInfo method on the Dynamic Info Controller reference. This will set the dynamic information that the NPC will use. In this example, we’ll initialize the Player’s health as a dynamic variable.

• Updating the Dynamic Info: Whenever you need to update the NPC with new information (such as a change in Player Health), call the SetDynamicInfo method on the Dynamic Info Controller.

Sample Scenario

• At the start of the game, we set the Player’s health to 100 and send this information to the NPC as the initial value.

• Then, when the player takes damage (simulated here by pressing the "K" key), we reduce the Player’s health and update the Dynamic Info in real time so that the NPC remains aware of the Player's current health status.

Example Conversation

Below, we provide a sample conversation showcasing how the NPC can react based on the dynamic health information of the Player. By dynamically updating the Player's health, NPCs can deliver responses that feel personalized and relevant to the current gameplay.

In summary

Add the Dynamic Info Controller to your NPC. Use SetDynamicInfo to initialize the dynamic variable at the start, and call SetDynamicInfo again whenever updates are needed.

Description

Convai's Unity WebGL SDK is designed to complement the standalone application capabilities of our Unity Asset Store version. With this specialized SDK, you can build and deploy interactive WebGL applications that leverage Convai's advanced conversation pipeline. Please see the instructions below or check out our latest tutorial video on YouTube.

Getting Started

Download the SDK and Demos

Prerequisites

Import and Setup Instructions

Follow the Import and Setup Instructions from Import and Setup (nightly) and Setting Up Unity Plugin.

WebGL Incompatibility with Unity Editor

When attempting to play the scene in the Unity Editor, you may encounter the following error:

WebGL SDK does not run in Unity Editor. Please build and run in WebGL.

This error occurs because the WebGL SDK cannot be tested directly within the Unity Editor. To test your WebGL application, you must create a development build.

Switching to WebGL

Now, your Unity setup is done, let's setup WebGL

Head on over to File → Build Settings, then:

1. Click on WebGL.

2. Check the Development Build box.

3. Select Switch Platform.

Patience, remember? This shift takes a bit.

After the platform is switched to WebGL, click on Player Settings.This is where the fun begins:

Configuring Player Settings

Once the platform conversion is complete,

1. Open the filePlayer Settings.

2. Navigate to the pageWebGL settings.

3. Under theResolution and Presentation tab, select theConvai PWA Template.

Importing Characters and Building the Scene

After the reloads is completed, check if the settings have changed. then, close all the open menus and follow these steps :

1. Double-click the Convai folder and go to scenes.

2. Open the Convey Demo WebGL scene.

3. Head to Convai's website, grab your API key, and input it back in Unity via Convai → Convai Setup.

4. Now, head again to the Convai’s website and grab your favourite character’s id and paste it to Convai → Character Importer.

Remember, Unity's editor won't let us test WebGL directly. But fear not, there's a Build and Run option:

1. Go back to File → Build Settings.

2. Click Add Open Scenes and then Build and Run.

Choose a folder for the build output, make a new one if needed, and name it "WebGL."

The first build is the longest, so feel free to stretch a bit – but don't venture too far. Soon, you'll greet our demo character, Amelia, or any other character you brought into your digital oasis. Just give your Microphone permissions and here you go!

Engaging with your Convai AI NPCs

Now the magic happens. Press and hold 'T' to chat with your carefully cultivated character. Or click on the text box to type out a question. And for the attention to detail – press F10 to access the settings panel where you can change your name and the UI style to your liking.

Feeling accomplished? You should! You now have a successfully working WebGL build in your browser. Curious developers can take a step further by downloading the project files from GitHub, available for all who desire to peek behind the curtain.

Prerequisites

Before you begin, make sure you have the following:

• Unity 2022.3 or later

• Xcode (latest version recommended)

• Apple Developer account

• Project with Convai's Unity SDK integrated and running properly

• MacBook for building and deploying to iOS/iPadOS

Step 1: Prepare Your Unity Project

1. Open your Convai-powered Unity project.

2. Ensure you have the latest version of the Convai Unity SDK imported and setup into your project.

Step 2: Configure Build Settings

1. In Unity, go to File → Build Settings.

2. Select iOS as the target platform.

3. Click Switch Platform if it's not already selected.

4. Check the Development Build option for testing purposes.

Step 3: Manually add Required Files

Add link.xml

1. Create a new file named link.xml in your project's Assets folder.

2. Add the following content to the file:

<linker>
  <assembly fullname="UnityEngine">
    <type fullname="UnityEngine.Application" preserve="fields">
      <property name="platform"/>
    </type>
  </assembly>
</linker>

This file prevents potential FileNotFoundException errors related to the libgrpc_csharp_ext.x64.dylib file.

Add BuildIos.cs Script

1. Create a new C# script in Assets/Convai/Scripts named iOSBuild.cs.

2. Add the following content to the script:

#if UNITY_EDITOR && UNITY_IOS
using System.IO;
using UnityEditor;
using UnityEditor.Callbacks;
using UnityEditor.iOS.Xcode;
using UnityEngine;

public class iOSBuild : MonoBehaviour
{
    [PostProcessBuild]
    public static void OnPostProcessBuild(BuildTarget target, string path)
    {
        string projectPath = PBXProject.GetPBXProjectPath(path);
        PBXProject project = new PBXProject();
        project.ReadFromString(File.ReadAllText(projectPath));
#if UNITY_2019_3_OR_NEWER
        string targetGuid = project.GetUnityFrameworkTargetGuid();
#else
        string targetGuid = project.TargetGuidByName(PBXProject.GetUnityTargetName());
#endif
        project.AddFrameworkToProject(targetGuid, "libz.tbd", false);
        project.SetBuildProperty(targetGuid, "ENABLE_BITCODE", "NO");
        File.WriteAllText(projectPath, project.WriteToString());
    }
}
#endif

Step 4: Install required gRPC dlls for iOS:

1. Go to Convai -> Custom Package Installer

2. Click on Install iOS Build Package

3. Attach the script iOSBuild.cs to any GameObject in your scene.

Step 5: Build the Xcode Project

1. In Unity, go to File → Build Settings.

2. Click Build and choose a location to save your Xcode project.

3. Wait for Unity to generate the Xcode project.

Step 6: Configure and Build in Xcode

1. Open the generated Xcode project.

2. In Xcode, select your project in the navigator.

3. Select your target under the "TARGETS" section.

4. Go to the "Signing & Capabilities" tab.

5. Ensure that "Automatically manage signing" is checked.

6. Select your Team from the dropdown (you need an Apple Developer account for this).

7. If needed, change the Bundle Identifier to a unique string.

Step 7: Build and Run

1. Connect your iOS device to your Mac.

2. In Xcode, select your connected device as the build target.

3. Click the "Play" button or press Cmd + R to build and run the app on your device.

Troubleshooting

• If you encounter any build errors, ensure all the steps above have been followed correctly.

• Check that your Apple Developer account has the necessary provisioning profiles and certificates.

• If you face any GRPC-related issues, verify that the libgrpc_csharp_ext.a and libgrpc.a files are correctly placed in the Assets/Convai/Plugins/gRPC/Grpc.Core/runtime/ios folder.

Long-Term Memory (LTM) enables the persistent storage of conversational history with NPCs, allowing players to seamlessly continue interactions from where they previously left off, even across multiple sessions. This feature significantly enhances the realism of NPCs, aligning with our goal of creating more immersive and lifelike characters within your game.

Steps to get LTM working

1. Select your Convai Character
2. Add the Long-Term Memory Component onto your character
3. Make sure that Long Term Memory is enabled for that character

Long Term Memory should now be working for your character.

Components of the LTM System

Convai Long Term Memory Component

This component will enable or disable LTM right from the unity editor

Toggling Long Term Memory

1) Click the button provided in the component

2) It will take some time to update, and after that the new status of the LTM should be visible in the inspector.

Troubleshooting

If you encounter this error, ensure that the SpeakerID was created using the same API key currently in use. If you're uncertain about the API key used, you can reset the SpeakerID and PlayerName by accessing the ConvaiPlayerDataSO file located in Assets > Convai > Resources, allowing you to start the process anew.

Management of Speaker ID(s)

It is essential for developers to efficiently manage the Speaker ID(s) generated using their API key, as the number of IDs that can be created is limited and dependent on the subscription tier. Proper management ensures optimal usage of resources and prevents potential disruptions in the application's functionality.

Speaker ID limit per API key are as follows

You can view all the Speaker ID(s) associated with a specific API key by accessing the Convai Window within your Unity project. This feature provides a comprehensive list of IDs, allowing for easier management and monitoring.

Head over to Long Term Memory Section

If the message "No Speaker ID(s) Found" appears, there is no need to proceed with this guide. However, if a Speaker ID list is displayed, it's advisable to delete any ID(s) that are no longer in use or needed to optimize your available resources.

Automatic Installation Steps

Step 1

In the top menu, click on Convai. Then, select Custom Package Installer.

Step 2

In the Convai Panel that appears, click on Package Management and then select Install MR Package.

Step 3

A new window will appear prompting you to select your installation type. For this documentation, we will proceed with Automatic Installation.

Step 4

In the next window, carefully read the setup instructions, warnings, and details of the changes that will be applied.

Step 5

When ready to proceed, click Yes, proceed.

Step 6

The installation process will begin, taking approximately 5 minutes. The duration may vary depending on your computer's performance.

Step 7

If a prompt titled OVRPlugin Detected from MetaSDK appears, click Restart Editor to continue.

Step 8

Once the installation is complete, it’s time to open the demo scene.

• Navigate to Assets > Convai > ConvaiXR > ConvaiMR > Scenes and open Convai Demo - MR.

Step 9

If you don’t have the TextMeshPro package installed, you’ll need it to display text properly in the demo scene. To install it:

• Go to Window > TextMeshPro > Import TMP Essential Resources.

• In the Unity Package Import window that appears, click Import.

Step 10

You can build your project by going to File > Build Settings.

Replacing the Spawned Character

Step 1

In the demo scene, select [BuildingBlock] Find Spawn Positions in the hierarchy.

Step 2

In the Spawn Object field, drag and drop the character you imported into the project.

• If you’re unsure how to import your character, please refer to the relevant documentation here.

Allowing the grpc_csharp_ext.bundle dll file in macOS

Using external DLLs in Unity on MacOS can lead to security permission issues due to Apple's strict security measures. Here's a step-by-step guide to resolving this common problem.

1. Verify the Problem:
2. Manually Allow Blocked DLLs:

   • Open System Preferences on your Mac.

   • Navigate to "Security & Privacy".

   • Under the "Security" tab, you might see a message at the bottom about the DLL being blocked. Click "Allow Anyway" or "Open Anyway" and enter password if asked.

3. Modify Gatekeeper settings: MacOS's Gatekeeper can prevent unidentified developers' software from running. To allow the DLL:

   • Open the Terminal (found in Applications > Utilities).

   • Type sudo spctl --master-disable and press Enter.

   • This command will allow apps to be downloaded from anywhere.

   • Now, try running the Unity project again.

   • After you're done, you should re-enable Gatekeeper with sudo spctl --master-enable to avoid any malware.

4. Check File Permissions: Ensure the DLL has the correct file permissions.

   • In Finder, right-click (or control-click) on the DLL file and choose "Get Info".

   • Under “Sharing & Permissions”, ensure that your user account has "Read & Write" permissions.

5. Review Unity's Plugin Settings:

   • In the Unity editor, select the DLL in the Project view.

   • In the Inspector window, make sure the appropriate platform (in this case, Mac OS X) and architecture (Apple Silicon, Intel-64) is selected for the DLL.

   • Ensure that the "Load on Startup" and other pertinent options are checked (should be enabled by default)

Mac Configuration in Player Settings during build

• Update Mac Configuration:

  • In Unity, navigate to Edit > Project Settings > Player.

  • Scroll down and click on Other Settings

  • Scroll down again to find Mac Configuration section

  • Update the Mac Configuration section (follow the below Screenshot)

AR Installation

If you want to make your Convai Plugin compatible with AR, you can do so in two ways. Please see the instructions below or check out our latest tutorial video on YouTube.

Method 1 : Automatic Setup

1. Click on " Convai / Convai Custom Package Installer / Install AR Package "

1. Confirm the changes and processes to be made. If you agree, the process will start. Click " Yes, Proceed " and the process will begin. You'll see logs in the console.

1. If you encounter an error like "Failed to Resolve Packages," don't worry. The process will continue, and the error will be resolved automatically after the package installations are complete.

1. Open the " Convai / Scenes / Convai Demo - AR " demo scene. If the TMP Importer window appears ( It will appear if TMP Essentials is not installed in your project ), click " Import TMP Essentials " to install TextMeshPro Essentials for UI text objects.

Alternatively, you can use the " Window / TextMeshPro / Import TMP Essential Resources " to install it.

1. After importing TMP Essentials, you can remove the empty GameObject in your scene that triggers the Prompt window to appear.

1. Build your project by going to " File / Build Settings / Build " Ensure that the " Convai Demo - AR " scene is included in the Scenes in Build section.

Now everything is ready for testing. 🙂✅

Method 2 : Manual Setup

1. Double-click on " Convai / Convai Custom Unity Packages / ConvaiVRUpgrader.unitypackage "

1. You'll see a warning that the settings will overwrite your project settings. You can either allow it by clicking " Import " or create a temporary project by clicking " Switch Project "

1. In the Import Unity Package window, review the assets to be imported and click " Next "

1. Select all settings to be changed in the Project Settings and complete the installation by clicking " Import "

1. Open the " Convai / Scenes / Convai Demo - AR " demo scene. If the TMP Importer window appears ( It will appear if TMP Essentials is not installed in your project ), click " Import TMP Essentials " to install TextMeshPro Essentials for UI text objects.

Alternatively, you can use the " Window / TextMeshPro / Import TMP Essential Resources " to install it.

1. After importing TMP Essentials, you can remove the empty GameObject in your scene that triggers the Prompt window to appear.

1. If you see 3D objects in pink, it's a shader issue. If you're using URP, convert the materials to URP by double-clicking on " Convai / Convai Custom Unity Packages / ConvaiURPConverter " and importing all assets in the window that appears.

1. Ensure you've set up your API Key ( Convai / Convai Setup ).

2. Build your project by going to " File / Build Settings / Build " Ensure that the " Convai Demo - AR " scene is included in the Scenes in Build section.

Now everything is ready for testing. 🙂✅

How to Add and Adjust Size Of My Own Character?

If you've created a Ready Player Me character on convai.com playground and want to add it to your AR project, follow these steps:

1. Right-click on the " Convai / ConvaiAR / Prefabs / Convai NPC AR Base Empty Character " prefab.

2. Click on " Create / Prefab Variant "

1. You'll see a prefab variant created for " Convai NPC AR Base Empty Character "

2. Double-click on this prefab variant.

1. In the Hierarchy section, add your imported character as a child to this prefab variant.

1. After adding your character, click on your character.

2. In the Inspector, adjust the Scale settings as needed. To prevent your character from moving with animation while talking, disable the " Apply Root Motion " option in the Animator.

1. After these steps, save your prefab variant by pressing CTRL + S.

2. Open the " Convai / Scenes / Convai Demo - AR " scene.

3. Click on the " Convai AR Player " object under " ConvaiAR Base Scene "

1. In the Inspector, under the " Convai Character Spawner " component, add your prefab variant to the Character Prefab field.

Now, everything is ready to test your character in the AR environment!🙂✅

Overview

The Dynamic UI system is a feature within the Convai Unity SDK that provides developers a robust system for in-game communication. This feature allows for displaying messages from characters and players and supports various UI components for chat, Q&A sessions, subtitles, and custom UI types. This document will guide you through the integration, usage, and creation of custom UI types of the Dynamic UI feature in your Unity project.

Usage

Accessing the Chat UI Handler

To interact with the chat system, you need to reference the ConvaiChatUIHandler in your scripts. You can find the Transcript UI prefab in the Prefabs folder.

Here's an example of how to find and assign the handler:

private ConvaiChatUIHandler _convaiChatUIHandler;

private void OnEnable()
{
    // Find and assign the ConvaiChatUIHandler component in the scene
    _convaiChatUIHandler = ConvaiChatUIHandler.Instance;
    if (_convaiChatUIHandler != null) _convaiChatUIHandler.UpdateCharacterList();
}

Sending Messages

Once you have a reference to the ConvaiChatUIHandler, you can send messages using the following methods:

Sending Player Text

To send text as the player:

_convaiChatUIHandler.SendPlayerText(input);

• input: The string containing the player's message.

Sending Character Text

To send text as a character:

_convaiChatUIHandler.SendCharacterText(characterName, currentResponseAudio.AudioTranscript.Trim());

• characterName: The name of the character sending the message.

• currentResponseAudio.AudioTranscript: The transcript of the audio response from the character, trimmed of any leading or trailing whitespace.

Adding Custom UI Types to the Dynamic Chatbox

While the Dynamic UI system within the SDK provides several pre-built UI types, you may want to create a custom UI that better fits the style and needs of your game and it designed to be extensible, allowing developers to add their custom UI types. This is achieved by inheriting from the ChatUIBase class and implementing the required methods. The ConvaiChatUIHandler manages the different UI types and provides a system to switch between them.

Creating a Custom UI Class

To create a custom UI type, follow these steps:

Step 1: Define Your Custom Class

Create a new C# script in your Unity project and define your class to inherit from ChatUIBase. For example:

using Convai.Scripts.Utils;
using UnityEngine;

public class CustomChatUI : ChatUIBase
{
    // Implement the required methods from ChatUIBase here.
}

Step 2: Implement Required Methods

Implement the abstract methods from ChatUIBase. You must provide implementations for Initialize, SendCharacterText, and SendPlayerText:

public override void Initialize(GameObject uiPrefab)
{
    // Instantiate and set up your custom UI prefab here.
}

public override void SendCharacterText(string charName, string text, Color characterTextColor)
{
    // Handle sending character text to your custom UI here.
}

public override void SendPlayerText(string playerName, string text, Color playerTextColor)
{
    Handle sending player text to your custom UI here.
}

Step 3: Add Custom Functionality

Add any additional functionality or customization options that your custom UI may require.

Step 4: Assign and Use Your Custom UI

To use your custom UI class within the dictionaryConvaiChatUIHandler, you need to add it to the GetUIAppearances dictionary. This involves creating a prefab for your custom UI and assigning it in the ConvaiChatUIHandler.

Here's an example of how to do this:

1. Create a prefab for your custom UI and add your CustomChatUI component to it.

2. Assign the prefab to a public variable in the ConvaiChatUIHandler script.

3. Modify the InitializeUIStrategies method in the ConvaiChatUIHandler script to include your custom UI type.

[Tooltip (Prefab for the customChatUI.")]
public GameObject customChatUIPrefab;

private void InitializeUIStrategies()
{
    Existing UI types
    InitializeUI(chatBoxPrefab, UIType.ChatBox);
    InitializeUI(questionAnswerPrefab, UIType.QuestionAnswer);
    InitializeUI(subtitlePrefab, UIType.Subtitle);

    // Custom UI type
    InitializeUI(customChatUIPrefab, UIType.Custom); // Make sure to define UIType.Custom in the UIType enum
}

private void InitializeUI (GameObject uiPrefab, UIType uiType)
{
    // existing code...

    Add your custom UI initialization here
    if (uiType == UIType.Custom)
    {
        CustomChatUI customUIComponent = uiPrefab.GetComponent<CustomChatUI>();
        if (customUIComponent == null)
        {
            Debug.LogError("CustomChatUI component not found on prefab.");
            return;
        }

        customUIComponent.Initialize(uiPrefab);
        GetUIAppearances[uiType] = customUIComponent;
    }
}