This is a scriptable object which is made automatically after you hit play in the editor with Convai SDK installed and in a Scene where Convai Base Scene Essentials Prefab is present

Default Player Name

You can provide a default name of your players.

Player Name

Current name of your player, out of the box if you use our settings panel, we keep it updated automatically, if you are using some custom logic, it will be your responsibility to keep it updated, as our transcript UI use this name to show it in UI

Speaker ID

Speaker ID for the player. Please note that Speaker ID is directly linked with your API key, so for each API key there should be a unique speaker ID associated with it. We handle the creation of the Speaker ID when it's not found in the Player Prefs if the Boolean is set to true.

Create Speaker ID If Not Found

This Boolean lets the SDK if it should create a unique Speaker ID for that Player Name if it is not found in the Player Prefs.

Buttons

Reset Data

It just makes the Player Name and Speaker ID fields empty.

Copy Data

Copies the data into system buffer so you can paste it anywhere for debugging purpose

Player Pref Settings Button

1. Load: Loads the Player Name and associated Speaker ID from the player Prefs

2. Save: Saves the Player Name and associated Speaker ID from the player Prefs

3. Delete: Deletes the Player Name and associated Speaker ID from the player Prefs

How to maintain the Player Data

Convai provides a pre-made component which you can add to any GameObject to make the PlayerDataContainer work out of the box.

Choose an existing GameObject or create a new GameObject in the scene and add the ConvaiPlayerDataHandler component to your chosen GameObject and it should start working

Optional Step

You can also create the required Scriptable Object by going to Assets > Convai > Resources and right clicking in the project panel and navigating to Create > Convai > Player Data and name it ConvaiPlayerDataSO

Convai Character emit character emotions when they interact with the player and these emotions help in making the character more human-like, we are starting to implement a system which you as a developer can use to make your game more interactive using the character emotions.

Whenever the character responds to the user, we send back a list of emotions to the SDK, which look something like this

For v0 of this system, we will only be sending the emotions, in future we will apply the facial expressions corresponding to each emotion which will make the character more interactive.

Convai offers comprehensive transcript and voice support for a wide range of languages. To facilitate seamless integration, our Unity plugin comes with a custom TextMeshPro (TMP) package, which includes essential fonts and required settings for major languages.

Setup

To implement these language-specific features in your project:

1. Navigate to the Convai Setup Window within Unity.

2. Locate the Package Management section.

3. Click on the "Convai Custom TMP Package" button.

Once installed, just import the character for which you require the language support, talk with it and the font will automatically render in the transcript.

For now, we provide fonts for these languages:

• Arabic

• Japanese

• Korean

• Chinese

RTL Support

We also provide support for Right-to-Left languages, like Urdu, Persian and Arabic through our Chat UIs. So, for example, if you talk with an Arabic character or if the character's name is in Arabic, the text will automatically enable the RTL feature provided by unity to reflect proper transcripts.

In a typical application integrating with the Convai API, maintaining a consistent session ID across different sessions is crucial for providing a seamless user experience. This documentation outlines the best practices for storing and retrieving session IDs using Unity's PlayerPrefs, including detailed steps and example scripts.

Importance of Session IDs

A session ID uniquely identifies a session between the client and the Convai server. Storing the session ID locally ensures that the same session ID is used across different sessions, which helps in maintaining context and continuity in interactions.

Storing Session IDs

When initializing a session, if a session ID is not available locally, it should be fetched from the server and then stored locally for future use. Here's how you can achieve this:

1. Fetch and Store Session ID: When initializing a session, check if a session ID is stored locally. If not, fetch a new session ID from the server and store it using PlayerPrefs.

public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
{
    // Retrieve stored session ID if it exists
    string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

    // If no session ID is stored, initialize a new one
    if (string.IsNullOrEmpty(sessionID))
    {
        sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

        // Store the new session ID locally
        if (!string.IsNullOrEmpty(sessionID))
        {
            PlayerPrefs.SetString(characterID, sessionID);
            PlayerPrefs.Save();
        }
    }

    return sessionID;
}

Retrieving Session IDs

When initializing your application, retrieve the stored session ID to ensure continuity in user interactions.

private async void Start()
{
    // Initialize session ID on start
    string characterID = "YourCharacterID"; // Replace with your actual character ID
    string sessionID = await InitializeSessionIDAsync("CharacterName", grpcClient, characterID);

    if (!string.IsNullOrEmpty(sessionID))
    {
        Debug.Log("Session ID initialized and stored: " + sessionID);
    }
    else
    {
        Debug.LogError("Failed to initialize session ID.");
    }
}

Example Class for Session Management

The following example class demonstrates how to manage session IDs using PlayerPrefs in a Unity project:

using System;
using System.Threading.Tasks;
using Convai.Scripts.Utils;
using Google.Protobuf;
using Grpc.Core;
using Service;
using UnityEngine;
using static Service.GetResponseRequest.Types;

public class SessionManager : MonoBehaviour
{
    public ConvaiService.ConvaiServiceClient grpcClient;

    private void Start()
    {
        // Initialize session ID on start
        InitializeSession("CharacterName", grpcClient, "YourCharacterID");
    }

    private async void InitializeSession(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = await InitializeSessionIDAsync(characterName, client, characterID);

        if (!string.IsNullOrEmpty(sessionID))
        {
            Debug.Log("Session ID initialized and stored: " + sessionID);
        }
        else
        {
            Debug.LogError("Failed to initialize session ID.");
        }
    }

    public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

        if (string.IsNullOrEmpty(sessionID))
        {
            sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

            if (!string.IsNullOrEmpty(sessionID))
            {
                PlayerPrefs.SetString(characterID, sessionID);
                PlayerPrefs.Save();
            }
        }

        return sessionID;
    }
}

Detailed Steps for Session Management

1. Initialize Session: Call InitializeSessionIDAsync to check if a session ID is stored. If not, fetch and store it.

2. Store Session ID: Use PlayerPrefs.SetString(characterID, sessionID) to store the session ID locally.

3. Retrieve Session ID: Use PlayerPrefs.GetString(characterID, string.Empty) to retrieve the stored session ID.

4. Use Session ID: Pass the session ID to your Convai API calls to maintain session continuity.

Best Practices

• Error Handling: Ensure proper error handling when fetching and storing session IDs.

• Security: Consider encrypting sensitive information stored in PlayerPrefs.

• Performance: Use asynchronous methods to avoid blocking the main thread when fetching session IDs.

The Convai plugin comes with default notifications, totaling four. Here they are:

Notifications

Not Close Enough to the Character

Appears when you press the talk button but there is no active NPC nearby.

Talk Button Released Early

Appears if you release the talk button in less than 0.5 seconds.

Microphone Issue Detected

Appears when the recorded audio input level is below the threshold.

Connection Problem

Appears when there is no internet connection upon launching the application.

How to Add Your Own Notification?

Adding your custom notification is straightforward.

Let's go through the steps to add a " CharacterStartedListening" notification as an example.

1. Open the script "Convai/Scripts/Notification System/Notification Type.cs." This script stores Notification Types as enums. Give a name to your desired Notification type and add it here.

1. Right-click on "Convai / Scripts / Notification System / Scriptable Objects" and select "Create > Convai > Notification System > Notification" then create a "Notification Scriptable Object".

1. Name the created Notification Scriptable Object. Click on it, and fill in the fields in the Inspector as desired.

1. Add the created Notification Scriptable Object to "Convai/Scripts/Notification System/Scriptable Objects" under "Convai Default Notification Group" (details of Notification Group here****).

1. Your Notification is now ready. The last step is to call this Notification from where you need it. For example, if you created the " CharacterStartedTalking " Notification, find the location where your character listens and write the code.

NotificationSystemHandler.Instance.NotificationRequest(NotificationType.CharacterStartedListening);

1. Replace the parameter with the NotificationType you created. (For our example, NotificationType.CharacterStartedListening)

2. Ensure that the Convai Notification System is present in your scene. (accessible from "Convai/Prefabs/ Notification System")

All steps are complete, and you're ready to test!🙂✅

Notification Scriptable Object

This Scriptable Object stores information about a Notification

• Notification Type

• Notification Icon

• Notification Title

• Notification Message

To create a new Notification Scriptable Object, right-click anywhere in the Project Window and select "Create > Convai > Notification System > Notification"

Notification Group Scriptable Object

This Scriptable Object stores Notification Scriptable Objects as groups. When a Notification is requested, it searches for the Notification using the specified Notification Group in the Convai Notification System prefab's Notification System Handler script.

You can create different Notification groups based on your needs. Note: If your referenced Notification Group does not have the Notification you want, that Notification won't be called.

The Convai Default Notification Group has four Notifications, but you can add more or create a new group with additional notifications.

Long-Term Memory (LTM) enables the persistent storage of conversational history with NPCs, allowing players to seamlessly continue interactions from where they previously left off, even across multiple sessions. This feature significantly enhances the realism of NPCs, aligning with our goal of creating more immersive and lifelike characters within your game.

Steps to get LTM working

1. Select your Convai Character
2. Add the Long-Term Memory Component onto your character
3. Make sure that Long Term Memory is enabled for that character

Long Term Memory should now be working for your character.

Components of the LTM System

Convai Long Term Memory Component

This component will enable or disable LTM right from the unity editor

Toggling Long Term Memory

1) Click the button provided in the component

2) It will take some time to update, and after that the new status of the LTM should be visible in the inspector.

Troubleshooting

If you encounter this error, ensure that the SpeakerID was created using the same API key currently in use. If you're uncertain about the API key used, you can reset the SpeakerID and PlayerName by accessing the ConvaiPlayerDataSO file located in Assets > Convai > Resources, allowing you to start the process anew.

Management of Speaker ID(s)

It is essential for developers to efficiently manage the Speaker ID(s) generated using their API key, as the number of IDs that can be created is limited and dependent on the subscription tier. Proper management ensures optimal usage of resources and prevents potential disruptions in the application's functionality.

Speaker ID limit per API key are as follows

You can view all the Speaker ID(s) associated with a specific API key by accessing the Convai Window within your Unity project. This feature provides a comprehensive list of IDs, allowing for easier management and monitoring.

Head over to Long Term Memory Section

If the message "No Speaker ID(s) Found" appears, there is no need to proceed with this guide. However, if a Speaker ID list is displayed, it's advisable to delete any ID(s) that are no longer in use or needed to optimize your available resources.

We provide several UI options to display character and user's transcript out of the box that players can use with the Convai Plugin. You can use and customize these prefabs.

The ConvaiNPC and ConvaiGRPCAPI scripts look for GameObjects with Convai Chat UI Handler as a component, and send any transcripts to the script so that it can be displayed on screen.

Types of UI

ChatBox

Prefab Name: Convai Transcript Canvas - Chat

Both the user's and the character's transcripts are displayed one after other in a scrollable chat box.

Subtitle

Prefab Name: Convai Transcript Canvas - Subtitle

The user and character transcripts are displayed in the bottom like subtitles.

Question-Answer

Prefab Name: Convai Transcript Canvas - QA

The user's transcript is displayed in the top where as the character's transcript is displayed in the bottom.

Mobile Optimised UI Styles

Prefab Name: Convai Transcript Canvas - Mobile Subtitle

Identical to #subtitle UI. Includes a button that can be pressed and held for the user to speak. Ideal for portrait orientation of screen.

Prefab Name: Convai Transcript Canvas - Mobile QA

Prefab Name: Convai Transcript Canvas - Mobile Chat

Convai Chat UI Handler Component

Functions to Know

The Dynamic Config feature enables you to pass variables to NPCs in real time, allowing them to react dynamically to changes in the game environment. This can include the player’s current health, inventory items, or contextual world information, greatly enhancing interactivity and immersion.

Step-by-Step Guide to Setting Up Dynamic Config

First, Add the Dynamic Info Controller Component to your Convai NPC.

Create a new script or use an existing script to define a variable that will store a reference to the Dynamic Info Controller Component you added to your NPC.

Example: Passing Player Health to the NPC

• Initialize the Dynamic Info: In the script’s Start method, call the SetDynamicInfo method on the Dynamic Info Controller reference. This will set the dynamic information that the NPC will use. In this example, we’ll initialize the Player’s health as a dynamic variable.

• Updating the Dynamic Info: Whenever you need to update the NPC with new information (such as a change in Player Health), call the SetDynamicInfo method on the Dynamic Info Controller.

Sample Scenario

• At the start of the game, we set the Player’s health to 100 and send this information to the NPC as the initial value.

• Then, when the player takes damage (simulated here by pressing the "K" key), we reduce the Player’s health and update the Dynamic Info in real time so that the NPC remains aware of the Player's current health status.

Example Conversation

Below, we provide a sample conversation showcasing how the NPC can react based on the dynamic health information of the Player. By dynamically updating the Player's health, NPCs can deliver responses that feel personalized and relevant to the current gameplay.

In summary

Add the Dynamic Info Controller to your NPC. Use SetDynamicInfo to initialize the dynamic variable at the start, and call SetDynamicInfo again whenever updates are needed.

Settings Panel consists of two main sections.

• Audio Settings

• Interface Settings

Audio Settings

Microphone Settings

The Microphone Settings section is primarily for troubleshooting and testing the microphone when using the Convai plugin.

• In the Input section, you can view the microphones connected to your computer and select the desired one.

• In the Test Input field, you can record your voice using the selected microphone in the Input section. After clicking Stop, you can listen to the recorded voice and observe the sound levels.

Interface Settings

Appearance

The first setting that greets us here is the Appearance setting.

In the Appearance section, you can switch between Transcript UI designs.

There are three Transcript UI options:

• ChatBox

• QuestionAnswer

• Subtitle

Display Name

The second section in Interface Settings is the Display Name section. This section allows you to change how the user's name appears in the Transcript UI.

Notifications Checkmark

The last section in Interface Settings is the Notifications Checkmark.

Convai sometimes displays notifications on the screen to inform the user. If you want to disable these notifications, you can click the checkbox here. ( If the box is green, it's active. If empty, it's inactive )

Overview

The Dynamic UI system is a feature within the Convai Unity SDK that provides developers a robust system for in-game communication. This feature allows for displaying messages from characters and players and supports various UI components for chat, Q&A sessions, subtitles, and custom UI types. This document will guide you through the integration, usage, and creation of custom UI types of the Dynamic UI feature in your Unity project.

Usage

Accessing the Chat UI Handler

To interact with the chat system, you need to reference the ConvaiChatUIHandler in your scripts. You can find the Transcript UI prefab in the Prefabs folder.

Here's an example of how to find and assign the handler:

Sending Messages

Once you have a reference to the ConvaiChatUIHandler, you can send messages using the following methods:

Sending Player Text

To send text as the player:

• input: The string containing the player's message.

Sending Character Text

To send text as a character:

• characterName: The name of the character sending the message.

• currentResponseAudio.AudioTranscript: The transcript of the audio response from the character, trimmed of any leading or trailing whitespace.

Adding Custom UI Types to the Dynamic Chatbox

Creating a Custom UI Class

To create a custom UI type, follow these steps:

Step 1: Define Your Custom Class

Create a new C# script in your Unity project and define your class to inherit from ChatUIBase. For example:

Step 2: Implement Required Methods

Implement the abstract methods from ChatUIBase. You must provide implementations for Initialize, SendCharacterText, and SendPlayerText:

Step 3: Add Custom Functionality

Add any additional functionality or customization options that your custom UI may require.

Step 4: Assign and Use Your Custom UI

To use your custom UI class within the dictionaryConvaiChatUIHandler, you need to add it to the GetUIAppearances dictionary. This involves creating a prefab for your custom UI and assigning it in the ConvaiChatUIHandler.

Here's an example of how to do this:

1. Create a prefab for your custom UI and add your CustomChatUI component to it.

2. Assign the prefab to a public variable in the ConvaiChatUIHandler script.

3. Modify the InitializeUIStrategies method in the ConvaiChatUIHandler script to include your custom UI type.

1. Ensure that your custom UI type is added to the UIType enum:

By following these steps, you can integrate your custom UI type into the Dynamic Chatbox system and switch between different UI types at runtime.

How to Change the Talk Button or Any Input?

1. Double click on the "Controls" asset in your project tab.
2. You can setup multiple control schemes for different devices here, currently we have it for PC (Keyboard & Mouse) and Gamepad. For mobile, we have provided joystick and buttons, which are mapped to Gamepad controls for functionality, but you can directly add touchscreen and use its different features to trigger an Input Action. You can also add your own control scheme if you want support for a different device by clicking on "Add Control Scheme".

1. Find the Input Action you want to change in the above window. If you want to add a new Input Action, refer to the other section in documentation. In this case, we selected "Talk Key Action" to change the talk button. Click on "T [Keyboard]". In the Binding Properties window, click on the " T [Keyboard] " button in the Path field.

2. Press the " Listen " button in the top left of the opened window. If you prefer, you can choose your desired input from the categories below.

1. Press the key you want to assign and it will be reflected in the control asset.

How to Add a New Input Action?

1. First, go to the controls asset mentioned above and use the add button to create an input action. For this example, we will call it interact and provide it the binding with [E] button.

2. Then, click on the <No Binding> item to setup binding for this action. As before, you can use the listen button (has a UI bug for Windows but works) or you can select the key from dropdown. After selecting the binding (we will select [E] key for this), don't forget to press on the Save Asset option on the top menu.
3. You will now get an error saying ConvaiInputManager does not implement OnInteract. We need to implement this. Open the " ConvaiInputManager.cs " script to do so. ( " Convai / Scripts / Runtime / Core / ConvaiInputManager.cs " )
4. Your IDE might suggest you to implement missing members. If it doesn't we can manually write the OnInteract function like in the last figure shown. You receive a callback context which shows which frame input started, performed or got cancelled which you can use for different purposes. And that's it the error should be gone and you are good to go!