Get your unique API key

Your API requests are authenticated using API keys. Any request that doesn't include an API key will return an error.

As soon as you sign up to Convai, an API key is generated for you. You can always visit your profile section on convai.com and reveal your API key for use, which will be present at the top right section of the page. You can always generate new ones in case the other is compromised.

Unleash Your Creativity with Convai Playground

Step into the Convai Playground, where you let your creativity shine. Here's a concise guide to bringing your character, let's call her Sabrina, to life.

Begin with Basics

Start by naming your character and selecting a voice from an array of options. This choice sets the stage for Sabrina's digital persona.

Craft a Unique Backstory

Next, dive into Sabrina's backstory. Is she energized by city life, or does she aspire to make people laugh as a stand-up comedian? This backstory isn't just filler; it's the soul of your character, making her relatable and dynamic. if you are out of ideas just click on Generate Backstory and relax.

Custom Actions, Personality and more

The Convai Playground offers extensive customisation options. From Sabrina's actions and language abilities to her knowledge and emotional state, every detail you choose adds depth to her character. Imagine adjusting her mood to react dynamically within a game, adding a layer of realism to the interaction.

Create an 3D Avatar

You can create your 3d character from scratch picking up hairstyles, body type, clothes and accessories and give your character a unique appearance according to the personality.

Interact with Your Creation

Finally, engage with Sabrina in a 3D web experience. This interaction is the result of your creativity, chat with the character on various topics, ask her about previous experiences from the memory or do some actions.

Your Creative Playground Awaits

The Convai Playground is more than just a tool; it's a gateway to endless possibilities. Whether you're a developer, storyteller, or simply curious, it offers a unique opportunity to explore the potential of AI in character creation. Dive in and let your imagination lead the way.

The Character Creator tool enables you to create and update characters with human-like capabilities for use in your applications. Use this tool to configure, test, and even share your characters with others online. Your characters will be available for use in any Convai plugin or SDK, and any updates will be reflected immediately in your applications.

Let's take a closer look into the Character Creator tool features.

Playground Dashboard

The dashboard view shows all your characters and the Convai sample characters. When you first log in to Convai and visit the Playground, we recommend that you first explore the sample characters.

Click on one of the sample characters to open the Character Editor section. Try interacting with the character using either text or speech input. Be sure to allow the use of your microphone when prompted by the browser. The Sample Characters all have different backstories that guide their responses. Feel free to try them all and explore how their backstories affects how they respond. We'll go over all the features and how to set up a new character in the next section.

In the next section, we'll show you how to create your first character.

We'll be updating this page with the most frequently asked questions from our users! Our Convai Developer Forum and YouTube Channels are also great resources if you have further questions or want to learn more!

1. What is "Character Concurrency"?

   • Character Concurrency is the number of users who can interact with the NPC at the same time.

     • For example, there are 5 people using separate instances of your application. In order for everyone to be able to interact with the NPC at the same time, the character concurrency limit for your account needs to be at least 5 or more.

   • Character Concurrency Example If there are two NPCs in the game, will the system allow three users to interact with NPC A and another three users to interact with NPC B at the same time?

     • Only if you have a character concurrency limit of at least 6.

   • Does the "Character Concurrency" limit apply individually to each NPC?

     • No, the limit applies to the API key being deployed. You can think of them as a limit on actively connected sessions. With a character concurrency limit of 1, you can have 10 AI NPC's or more, but you will only be able to have one user speaking to any of the NPCs at the same time.

2. How do I get my character to stick to their knowledge bank and backstory?

   • Utilize the Moderation & Guardrails toggle and minimize external information. We continue to work on improving character cohesion and prevent hallucinations in conversation responses.

3. What LLM's are powering Convai?

   • Convai utilizes a variety of models across the text-to-speech, input, and speech-to-text output, including fine-tuned open source, as well as commercial foundational models like ChatGPT. The best model for the current context (audio, text, video, etc) is used.

4. Is there an API to change my character's information?

   • Yes! You can see the list of API calls and their functions for characters here.

5. Do the SDK's support Custom Characters?

   • Yes, you can use your own custom characters! Note that your custom characters will require facial blendshapes and a rigged skeleton in order for the default animations and lipsync to function as similar to the other natively supported Avatar systems. Here is the Unreal custom character tutorial, and the Unity video will be shared soon.

Convai provides a variety of plugins and integrations to help integrate conversational AI and avatars into your projects.

Game Engines

Web Plugin

Modding Framework

When you create a new character, it loads with a default 3D ReadyPlayerMe avatar. You can easily configure and upload your own customized 3D avatar from ReadyPlayerMe (RPM) with a few simple steps.

Follow these steps to create your custom RPM avatar for your character:

Steps

1. Open convai.com and visit the Dashboard section of the playground. You have to log in to access this page.
2. Click on the character that you wish to edit. This will open the character creator tool.

3. You will notice that the page already has a 3D model that Convai randomly assigns when the character is first created. To add your own model, click on the Configure Avatar option from the left menu. This will open the RPM Avatar Creator section.
4. In the RPM Avatar Creator section, you can either sign in to RPM and access one of your existing characters, or you can create a completely new one. Here we will create one from scratch. As the character I have chosen is a female, I will select the Feminine option.

5. You can now upload a photo to create an avatar from or you can continue without one. I will select the Continue without a photo option.
6. Now select a facial structure to start with. We'll select the first one for our example. Click on Next.
7. Now you can configure your avatar just the way you want. RPM provides a list of configurable options starting with minute facial structure and details, to hair style, to dress, and many more. All of these are available on the right-hand side of the RPM Avatar Creator section. Once you are satisfied with the avatar, click on NEXT at the top right corner.
8. You should now have your own custom RPM avatar. Once the processing completes, you will be able to see the avatar on the right-hand side of the screen.

Now, every time you open your character details page, your newly created avatar will appear.

Welcome to Convai

Convai is a platform for developers and creators. It provides the most intuitive interface for designing characters with multimodal perception abilities in both virtual and real world environments. Creators, game designers, and developers can modify the NPC's backstory, personality, and knowledge at any given moment via the playground or programmatically through the API.

Want to deep dive?

Dive a little deeper and start exploring our API reference to get an idea of everything that's possible with Convai:

Set Language

This section enables you to choose languages and voice settings for your character.

Multilingual Support : Addressing global language diversity, this multilingual feature enables widespread adoption of human-like AI systems for games and digital human applications. Currently supporting 21 languages, Convai is committed to ensuring more inclusive and accessible AI interactions across different cultures and regions.

You can use the language field to set up-to four different languages that your character can speak. These choices define the languages that the character can speak and does not affect the input language from the user. Currently convey supports the following languages: Arabic, Chinese (Cantonese), Chinese (Mandarin), Dutch, Dutch (Belgium), English, Finnish, French, German, Hindi, Italian, Japanese, Korean, Polish, Portuguese (Brazil), Portuguese (Portugal), Russian, Spanish, Spanish (Mexico), Spanish (US), Swedish, Turkish, and Vietnamese. New languages are added on an ongoing basis.

Custom Pronunciations

Custom Pronunciations enhances the pronunciation capabilities of the character. It allows you to specify how certain words should be pronounced. This can be useful for words the character may struggle with or pronounce incorrectly.

To use this feature, add the word you want to specify in the "Word" column. Then, in the "Pronunciation" column, spell out how the word should sound using plain English letters and syllables.

For example, let's try the name "convai":

• Spelled As: convai

• Pronounced As: convey

New Word Recognition

The New Word Recognition feature enhances the character's speech recognition capabilities, allowing it to better understand unique or challenging words.

To use this feature, simply enter the word you want the system to learn in the "Spelled As" column. Then, in the corresponding "Pronounced As" column, spell out the word's pronunciation using easy-to-read syllables and sounds.

For example, let's try the name "Ankur":

• Spelled As: Ankur

• Pronounced As: Ahnkur

This section enables you to adjust the personality and style of your character. Personality and style settings affect how your character responds and the type of language it uses when speaking. Under the speaking style tab, you can choose from a number of preset styles including things like a cowboy, a pirate, or a jazz musician. You can add to these styles or create your own by adding examples and catchphrases in the field provided.

The personality trait tab in this section allows you to adjust five main personality traits to help you further shape how your character responds. Each trait, including openness, meticulousness, extroversion, agreeableness, and sensitivity can be adjusted from zero to four, with zero being the lowest setting and four the highest. You can try changing the settings to see how they affect the output from your character. There is also a drop-down list of some predefined personalities for you to explore.

This section displays a State of Mind graph, which relates to the emotional state of your character. The graph depicts a wheel of emotions and dynamically highlights the particular emotions that your character is experiencing during the conversation. Stay tuned for more about emotions in future updates.

The graph below shows the current emotional state based on the last chat message.

Memory section allows you to view your previous conversations with the character, organized by date and time. Each chat session is assigned a unique ID for easy reference. You can download, copy, or delete individual chat logs from the character's memory as needed.

Enabling Long-Term Memory

For an even more immersive experience, you can enable the Long-Term Memory feature in the Memory Settings. This allows the character to remember details about you, the player, fostering deeper, personalized relationships over time. With Long-Term Memory enabled, characters can recall your preferences and choices, adapting their responses to your individual playstyle for a truly unique experience.

Unity Version

Skills and Knowledge

Before integrating the Convai SDK, you should be comfortable with the following:

1. Importing Packages: Know how to import external packages into a Unity project.

2. Unity Editor: Be proficient in navigating the Unity Editor interface.

3. Animations: Understand how to add and handle animations for assets.

4. Programming in C#: Have a basic experience programming Unity scripts in C#.

5. Script Integration: Be capable of adding scripts to a game object.

6. Building and Deployment: Know how to build and deploy an application to your chosen platform.

Having these skills will ensure a smooth integration and optimal use of the Convai Unity SDK in your projects.

Size Constraints

iOS browsers impose strict limitations on the size of WebGL builds. These constraints are primarily due to:

• Memory Limits: iOS devices have limited available memory for web applications, which can affect the performance and feasibility of running large WebGL builds.

• Browser Storage Quotas: Safari and other iOS browsers restrict the amount of data that can be stored locally. This includes caching and Indexed DB, which are often used to store assets for WebGL builds.

Key Limitations

• Maximum Downloadable Asset Size: iOS browsers may restrict the size of individual downloadable assets. Large assets might fail to load, causing the application to break.

• Total Build Size: The total size of all assets combined should ideally be kept under 50-100 MB for smooth performance. Exceeding this limit can lead to crashes or extremely slow loading times.

• Memory Usage: iOS devices typically have less RAM available compared to desktop environments. High memory usage by WebGL builds can result in frequent browser crashes.

Browser Compatibility

• Safari: The default browser on iOS, Safari, is generally the best option for WebGL builds, but it still has significant limitations compared to other desktop browsers.

In a typical application integrating with the Convai API, maintaining a consistent session ID across different sessions is crucial for providing a seamless user experience. This documentation outlines the best practices for storing and retrieving session IDs using Unity's PlayerPrefs, including detailed steps and example scripts.

Importance of Session IDs

A session ID uniquely identifies a session between the client and the Convai server. Storing the session ID locally ensures that the same session ID is used across different sessions, which helps in maintaining context and continuity in interactions.

Storing Session IDs

When initializing a session, if a session ID is not available locally, it should be fetched from the server and then stored locally for future use. Here's how you can achieve this:

1. Fetch and Store Session ID: When initializing a session, check if a session ID is stored locally. If not, fetch a new session ID from the server and store it using PlayerPrefs.

public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
{
    // Retrieve stored session ID if it exists
    string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

    // If no session ID is stored, initialize a new one
    if (string.IsNullOrEmpty(sessionID))
    {
        sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

        // Store the new session ID locally
        if (!string.IsNullOrEmpty(sessionID))
        {
            PlayerPrefs.SetString(characterID, sessionID);
            PlayerPrefs.Save();
        }
    }

    return sessionID;
}

Retrieving Session IDs

When initializing your application, retrieve the stored session ID to ensure continuity in user interactions.

private async void Start()
{
    // Initialize session ID on start
    string characterID = "YourCharacterID"; // Replace with your actual character ID
    string sessionID = await InitializeSessionIDAsync("CharacterName", grpcClient, characterID);

    if (!string.IsNullOrEmpty(sessionID))
    {
        Debug.Log("Session ID initialized and stored: " + sessionID);
    }
    else
    {
        Debug.LogError("Failed to initialize session ID.");
    }
}

Example Class for Session Management

The following example class demonstrates how to manage session IDs using PlayerPrefs in a Unity project:

using System;
using System.Threading.Tasks;
using Convai.Scripts.Utils;
using Google.Protobuf;
using Grpc.Core;
using Service;
using UnityEngine;
using static Service.GetResponseRequest.Types;

public class SessionManager : MonoBehaviour
{
    public ConvaiService.ConvaiServiceClient grpcClient;

    private void Start()
    {
        // Initialize session ID on start
        InitializeSession("CharacterName", grpcClient, "YourCharacterID");
    }

    private async void InitializeSession(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = await InitializeSessionIDAsync(characterName, client, characterID);

        if (!string.IsNullOrEmpty(sessionID))
        {
            Debug.Log("Session ID initialized and stored: " + sessionID);
        }
        else
        {
            Debug.LogError("Failed to initialize session ID.");
        }
    }

    public static async Task<string> InitializeSessionIDAsync(string characterName, ConvaiService.ConvaiServiceClient client, string characterID)
    {
        string sessionID = PlayerPrefs.GetString(characterID, string.Empty);

        if (string.IsNullOrEmpty(sessionID))
        {
            sessionID = await ConvaiGRPCAPI.InitializeSessionIDAsync(characterName, client, characterID, sessionID);

            if (!string.IsNullOrEmpty(sessionID))
            {
                PlayerPrefs.SetString(characterID, sessionID);
                PlayerPrefs.Save();
            }
        }

        return sessionID;
    }
}

Detailed Steps for Session Management

1. Initialize Session: Call InitializeSessionIDAsync to check if a session ID is stored. If not, fetch and store it.

2. Store Session ID: Use PlayerPrefs.SetString(characterID, sessionID) to store the session ID locally.

3. Retrieve Session ID: Use PlayerPrefs.GetString(characterID, string.Empty) to retrieve the stored session ID.

4. Use Session ID: Pass the session ID to your Convai API calls to maintain session continuity.

Best Practices

• Error Handling: Ensure proper error handling when fetching and storing session IDs.

• Security: Consider encrypting sensitive information stored in PlayerPrefs.

• Performance: Use asynchronous methods to avoid blocking the main thread when fetching session IDs.

Common Issues (FAQ)

Q. I cannot see the Convai menu.

A. Please check if there are any errors in the console. Unity needs to be able to compile all the scripts to be able to display any custom editor menu options. Resolving all the console errors will fix this issue.

Q. There are a lot of errors on my console.

A. Primarily, two issues cause errors in the console that can stem from the Convai Unity Plugin. You can use the links below to fix them quickly.

1. Disable Assembly Validation

2. Missing Newtonsoft Json

Q. I am talking to the character, but I cannot see the user transcript and the character does not seem to be coherently responding to what I am saying.

A. This may indicate issues with the microphone. Please ensure that the microphone is connected correctly. You also need to ensure that the applications have permission to access the menu.

Microphone Permission Issues

Q. The animations for my characters are looking very weird.

A. The animation avatar that we are using might be incompatible with the character mesh. Fixing that can solve the issue.

Default Animations Incompatibility

Q. There are two Settings Panel Buttons in Mobile Transcript UI.

A. If you are using Unity 2021, unexpected prefab variant issues may arise. This is because Unity Mobile Transcript UIs are variants of the main transcript UI prefab. With changes in the Prefab system in Unity 2022, it works correctly in Unity 2022. If you are using Unity 2021, you may encounter issues with prefabs. You can remove the redundant Settings Panel Button to address this problem.

Q: The lipsync is very faint or not visible.

A: The animations that we are using may be modifying facial animations. Editing the animations to remove facial animations should fix any issues related to lipsync.

Animations have Facial Blendshapes

A: The script also needs the avatar to not be mapped to the jaw bone to be manipulate the jaw bones itself.

Jaw Bone in Avatar is not Free

Q: I'm facing security permission issues using the grpc_csharp_ext.bundle DLL inside the Unity Editor on MacOS

A: macOS's strict security measures can block certain external unsigned DLLs. To address this, you can manually allow the DLL in "Security & Privacy" settings, modify Gatekeeper's settings through Terminal, ensure correct file permissions for the DLL, check its settings in Unity, and update the Mac Configuration in Unity's Player Settings

macOS Permission Issues

Q: I'm not able to talk to my character after building my Unity project for macOS (Intel64+Apple Silicon builds), especially on Intel Macs

A: The issue is rooted in the grpc_csharp_ext.bundle used in Unity for networking. This DLL has separate versions optimized for Intel and Apple Silicon architectures. When trying to create a Universal build that serves both, compatibility problems arise, especially on Intel Macs. Presently, the best solution is to use Standalone build settings specific to each architecture.

Building for macOS Universal apps

Error Index

Follow this Table to navigate to our most common errors.

Assembly 'Assets/Convai/Plugins/Grpc.Core.Api/lib/net45/Grpc.Core.Api.dll' will not be loaded due to errors: 
Grpc.Core.Api references strong named System.Memory Assembly references: 4.0.1.1 Found in project: 4.0.1.2.

Assets\Convai\Plugins\GLTFUtility\Scripts\Spec\GLTFPrimitive.cs(8,4): error CS0246: The type or namespace name 'JsonPropertyAttribute' could not be found (are you missing a using directive or an assembly reference?)

Assets\Convai\Scripts\Utils\HeadMovement.cs (2,30): error CS0234: The type or namespace name 'Rigging' does not exist in the namespace 'UnityEngine.Animations' (are you missing an assembly reference?)

For any other issues, feel free to contact us on the Convai Developer Forum.

Introduction

Welcome to the dynamic realm of character actions! Just imagine: your meticulously crafted characters are no longer standing still, but rather dancing, waving, or even performing complex maneuvers at your command. Isn't it exhilarating to think about the endless possibilities? Whether you want to add a simple gesture or craft an intricate sequence of movements, this page will serve as your gateway. Dive in to learn how to enable your characters into action.

Overview

Let's examine the Actions interface, and then explore how to assign actions to your characters.

Character without Actions

Let us prompt the character, and observe their response before enabling any specific actions.

Adding Actions

• Add your preferred action
• Click the 'Update' button to refresh your character, then prompt it to execute the chosen action.

Complex Action Sequences

• Add Further Actions to Your Character and click 'Update'

• Prompt your complex action

Introduction

We know that language models are helpful for a variety of different tasks. But their capabilities are severely limited by the input length of these models. What this means for our character chatbot is that we now have access to a knowledge bank where you can store large amounts of text-based knowledge for your character.

Overview

Let us look at the knowledge bank interface and then drill down into each element.

We have two main ways to add a knowledge bank:

1. Use the Text Box

2. Upload files (Limited Right now to 1MB)

Let us go through these and see how to use each of these ways.

Text Box

Using File Upload

Right now, we support only uploading text files as a knowledge bank. You can add information for your character as text files. You can upload files simply by clicking on the “Upload” button. You can click on the highlighted button to upload files from your computer.

Once you have uploaded the file, it will require some time before it is available for use (we currently have an upper limit of around 10 minutes for this). Once done, your file will appear under the “Available files on your account”, and the “Connect” button will become green like in the image above. You can then choose to associate the file with the current character.

Once connected, you can ask questions which would be only present in the knowledge bank and get the relevant information from your character.

Using the Knowledge Bank

Using the knowledge bank is pretty simple; once you have uploaded the knowledge bank for your character, you can use the chat UI or the /getResponse API to ask about anything stored in the knowledge bank. You should receive accurate results. Let us look at an example. We have a file called "mb4.txt", which is a made-up story about a Moon Base and its commander, Samantha. We can be sure that it was not part of the training data for our models because this story was made up just for this tutorial. Let us see how the character chatbot responds when asked about our dashing commander Samantha without the correct document connected to this character.

That definitely does not look correct, it is just a generic response without any particulars. Moreover, there is no mention of the moon base at all. Let us use the UI to connect the relevant file.

Once we have connected the file, you will get a pop-up when the connection is complete. Let us ask our character once again about Samantha and see if we get the correct answer.

The correct information is present in the reply this time. It means our operation was successful.

We hope this page gives you enough information about how to use the knowledge bank to start utilising it for your own purposes. Feel free to contact us at support@convai.com if you have any questions.

Knowledge Bank Best Practices

This guide is designed to help effectively utilize the Knowledge Bank to create engaging and informative conversational AI experiences.

What is the Knowledge Bank?

The Convai Knowledge Bank is a powerful tool that allows you to provide your AI character with a wealth of information on various subjects. It uses a technique called RAG (Retrieval-Augmented Generation) to efficiently store and retrieve relevant information during conversations.

RAG works by automatically chunking the uploaded text, PDF, or other files into smaller segments based on the spaces between paragraphs. This enables the AI to quickly locate and access the most relevant information when responding to user queries.

Best Practices for Optimal Efficiency

To ensure that your AI character can effectively understand and utilize the information in the Knowledge Bank, follow these best practices when preparing your files:

1. Single File Format: Upload your information as a single file, such as a text document or PDF.

2. Paragraph Structure: Each paragraph in the file should focus on a single subject and be approximately 5 lines long. This allows the AI to clearly understand the topic and context of the information.

3. Q&A Format: Alternatively, you can structure your information in a question-and-answer format within the same paragraph. This helps the AI understand the topic and how to respond appropriately.

Enhancing the AI Character Experience

To create a more engaging and believable AI character, it's crucial to ensure that the information in the Knowledge Bank aligns with the character's way of thinking and speaking. Consider the following tips:

• Use language and terminology that fits the character's background, personality, and domain expertise.

• Incorporate the character's unique perspective, opinions, and experiences when crafting the knowledge bank content.

• Maintain consistency in tone, style, and information throughout the knowledge bank to reinforce the character's identity.

By following these best practices and tips, you'll be able to create a rich and immersive conversational AI experience that brings your character to life.

Happy creating with the ConvAI Knowledge Bank!

Introduction

Narrative Design enables game developers to outline high-level objectives for NPCs, thereby guiding the narrative flow without constraining it to traditionally rigid dialogue trees. This approach allows for similar behaviour as a state machine but with support for scripted as well as dynamic responses as the NPC progresses through the decision making process from interactions with the player or via triggers from the game state. You can read more about the considerations behind Narrative Design here.

Videos

Here is a helpful series of videos outlining how to create a Narrative Design Graph in the Convai Playground, as well as the demo use case we created with a Tour Guide that shows the steps involved in creating your own Narrative Design solution and implementation.

Narrative Graph

You can find this tool under the "Narrative Design" tab on the Convai Playground. There are three fundamental elements to the graph, Sections, Triggers and Decisions.

Sections

Sections consist of two components Objectives and Decisions; and each Section also has a unique ID for ease of reference and tracking.

Objectives

This defines the overarching goal that the character aims to fulfill. For example, the initial objective for a museum tour guide NPC, is to extend a warm welcome and inquire whether the player is interested in taking a guided tour.

Decisions

As the conversation unfolds, it becomes essential to adapt to the player's preferences and responses, adjusting the NPC's objectives accordingly. Decisions are critical in this context. Taking the tour guide example further, when the NPC poses the question about taking a tour, the player's affirmative or negative response will lead the NPC to pursue a different objective, tailored to the player's choice. Decisions lead to new sections and allow for the storyline or experience to progress.

Syntax Instructions

These special character can be utilized in the nodes to trigger specific outcomes.

Triggers

There are three types of triggers currently: Spatial, Time-Based, and Event-Based. These are essential mechanisms that enable NPCs to discern when certain conditions have been met or events have occurred before proceeding to the next Section. Each trigger has a unique ID for referencing and tracking.

Spatial

Spatial triggers are activated when characters or players are in the correct location in the experience. For example, standing in front of an information booth could be the Spatial trigger for the NPC to ask the player if they require assistance.

Time-Based

These triggers would occur after a set amount of time has elapsed. For example, if the player has not said anything or responded after a certain number of seconds, the character could repeat the question or inquire what was the delay. Adding more dimensions and natural engagement to the experience.

Event-Based

Event-based triggers correspond to events that occur in the experience or game engine that you would want the characters to respond to. For example, if there was an explosion event in the game, you could have that trigger responses and new Sections from the characters within the range of the explosion event.

Introduction

In this section, we look into Character Versioning, i.e., maintaining different states of the character. This enables the user to preserve a previous stable state before trying out more changes. You can now experiment without the fear of losing an older state of the character, and in case you want discard the current changes and return to a previous version, you now have the ability to restore the version and continue working from there. We conveniently call these saved states as Snapshots of the character.

We will go over the features and how to use them in this section.

Overview

The character versioning option is available at the top right-hand side in the character editor section beside the Update button

Once you click on it, you get to see the list of all your previous saved revisions ordered by date.

We will go over the steps of creating and maintaining snapshots from scratch in the next section

Create a Version

Let us start with a character that we already have saved. The data that we see when we open the details related to a character denotes the Current Snapshot of the character. When you interact with the character, you are essentially referring to all the date in this Current Snapshot of the character.

1. To create a new version, first open the Character Versioning section and click on the + Add Snapshot sign at the top.
2. A pop-up appears asking you to give your snapshot a name and some description. Please note that a Snapshot Name is a required field to create a new version. Once you have filled the details, click on the Submit button.
3. Now, you can see the new version in the list of snapshots. Now what does this version actually represent? This snapshot stores all the data related to the character at that point of time. Everything about the character ranging from character description, embodiment to knowledge bank files, narrative-design structure and other details.

Restoring a Version

Assuming you have gone ahead and worked on the character further, but you are unhappy with the results and want to go back and start from the previous version. This is where you have the ability to restore an old snapshot to the current state and work with them again. Here are the steps to follow:

1. To restore a version, open the Character Versioning section and select the snapshot you want to restore back. You will see the Restore Version button below come to life.
2. Once you click on the Restore Version button, a pop-up appears asking you if you want to save the current changes as a new snapshot or discard them. You have the option to store your current changes as some test version and refer back later on.
3. For now, we are happy to discard the changes, so we will click on Restore button. This brings the data from the selected version to the Current Snapshot of the character.

4. To save the changes, you can always Cancel and go back to creating a new snapshot with your progress and the restoring it.

Delete a Snapshot

You can also go ahead and delete a snapshot that you no longer require. To that you can click on the 3-dots by the corresponding snapshot in the list of Character Version and select Delete Version

Some important points to remember

At any given point you can interact with the Current Snapshot of the character. If you have any publicly available app that utilises the character, your users will only be able to interact with this current version.

Overview

Convai's Unity SDK provides you with all the tools you need to integrate conversational AI into your Unity projects. Convai offers specialized NLP-based services to build intelligent NPCs for your games and virtual worlds. Our platform is designed to seamlessly integrate with your game development workflow, enhancing the interactivity and depth of your virtual environments.

Key Features

• Conversational AI: Leverage advanced NLP capabilities to create NPCs that can understand and respond to player input in natural, engaging ways.

• Intelligent NPCs: Build characters with dynamic dialogue and behaviors that adapt to player actions and the game world.

• Easy Integration: Our SDK is designed for quick and simple integration with your Unity projects, allowing you to focus on creating compelling gameplay experiences.

• Cross-Engine Support: In addition to Unity, Convai supports other popular game engines, ensuring broad compatibility and flexibility for your development needs.

This is the Core version of the plugin. It has a sample scene for anyone to get started. This version of the plugin only contains the basic Convai scripts and Character Downloader.

Visit convai.com for more information and support.

Quick Setup Tutorial

Prerequisites

Installing Visual Studio for Windows

• Download Visual Studio from here.

• Ensure having the required C++ toolchains mentioned here. If you already have Visual Studio installed, you may open the installer and select 'Modify' to add the above mentioned toolchains.

Installing XCode for Mac

• Download XCode from the app store

• For UE 5.3 and UE 5.0, follow this guide to enable microphone permissions.

Installing the plugin

There are two methods to install the plugin depending on your requirements

1. Directly from the Marketplace Link: Recommended for easy installation and tracking updates to the plugin. (UE 5.1 - 5.3)

2. Building from Github Source: For source UE builds and for unsupported UE versions on the marketplace, but not guranteed as the marketplace approach. (UE 4.26 - 5.0)

Set up your project

1. From the top toolbar, go to Edit > Plugins.

2. Find the Convai plugin.

3. Click the checkbox to enable the plugin.
4. Restart Unreal Engine.

Set up the API key

1. Find your API key.

2. Go to Edit > Project Settings.

3. Choose Convai under the Plugins section on the left bar.
4. Paste the API key into the API Key field.

Overview

Dynamic Environment Info is a powerful feature that allows users to pass additional environmental data to characters without direct interaction. This enables more immersive and creative gameplay scenarios by enhancing how characters perceive their surroundings.

For example:

• The character can understand the time of day (e.g., "time of day is night").

• The character can access inventory details (e.g., "You currently have a gun and a healing potion").

• The feature supports structured data formats for richer information exchange.

Follow the steps below to integrate Dynamic Environment Info into your project.

1. Open the Character Blueprint in your project.

2. In the Begin Play event, locate the ConvaiChatbot component.

1. Set the Dynamic Environment Info variable with a string value of your choice.

• Example 1 (Simple):

  Copy

  time of day is night

• Example 2 (Structured Format):

  Copy

  inventory: {weapon: gun, tools: [flashlight, rope]}

Save and Play

1. Save your blueprint changes.

2. Hit Play to test the interaction.

3. Observe how the character dynamically responds based on the information passed.

By following these simple steps, you can unlock more engaging gameplay mechanics and enrich the interactions.

Guides

More on the Unreal Engine SDK

Here, we outline the steps to create your own AI character on Convai.com in a simple way. We will cover advanced configuration options in separate sections later.

Steps

1. Sign in to Convai and click on the Create Character button from your Dashboard.

2. The landing page provides intuitive fields for creating a character. Don't worry about configuring an avatar yet - we'll cover that later.

1. Enter a name for your character, select a voice from the provided options, and write a backstory. ready.

1. Click Create Character when you are satisfied with your character details.

2. You will receive a unique ID for your new character. This will allow you to access the character outside of Convai.com. A default avatar will be loaded that you can customize later.

1. That's it! You now have a basic character to work with. You can start talking to your character in the chat window on the right side of the page.

In the next few pages we'll go through more advanced options for enhancing your character.

Unity Version​

The minimum supported Unity version is 2022.3.x. Earlier versions may not be compatible.

Supported Platform​s

Disabling Assembly Validation

Setting up Unity Plugin

In the Menu Bar, go the Convai > API Key Setup.

Go to convai.com, and sign in to your Convai account. Signing in will redirect you to the Dashboard. From the dashboard, grab your API key.

Enter the API Key and click begin.

This will create an APIKey asset in the resources folder. This contains your API Key.

Open the demo scene by going to Convai > Demo > Scenes > Full Features

Click the Convai NPC Amelia and add the Character ID (or you can keep the default character ID). You can get the character ID for your custom character from this page Create Character. Now you can converse with the character. The script is set up so that you have to go near the character for them to hear you.

Now you can test out the Convai Demo Scene and talk to the character present there. Her name is Amelia and she loves hiking!

You can open the Convai NPC Script to replicate or build on the script to create new NPCs.

If you ever get an error that looks like this, disable the Assemble Version Validation in Project Settings > Player > Other Settings.

Assembly 'Assets/Convai/Plugins/Grpc.Core.Api/lib/net45/Grpc.Core.Api.dll' will not be loaded due to errors: 
Grpc.Core.Api references strong named System.Memory Assembly references: 4.0.1.1 Found in project: 4.0.1.2.

Ensure that Assembly Validation is disabled in Project Settings > Player > Other Settings.

Restart the Unity project after unchecking the box should fix the issue.

1. Go to your player blueprint.

2. Click on the gear icon and select Show Inherited Variables under My Blueprint tab.

3. Search for the variable MaxChatDistance.

4. Set it to a number of your choice. Note: Set it to 0 for infinite distance.

1. Open your AI character blueprint.

2. Select the Floating Pawn Movement component from the components list.

3. Set the Max Speed field under the details panel to your required speed.

If you see the microphone indicator turning on in the top left corner but no user transcript in the chat UI and the character's response doesn't seem coherent to what you said, then it is likely that the game or Unity is not accessing the correct microphone or does not have sufficient microphone privilege. To fix this, please follow along.

Check Notification System page to learn more about various issues related to microphone

If the Lip-sync from characters are either not visible or are very faint, if could be a result of character's animations overriding the blendshape changes made by the script. We recommend deleting the relevant components in the animation dopesheet.

If the default animations that ship with the animator look bugged such that the hand seems to intersect with the body, it could indicate an issue with the wrong animation avatar being selected.

You can easily fix that by heading to the character's animator component and assigning the correct animator to the Avatar field.

The correct animation will look something like this. The hands should not intersect the body.

Our plugin has various scripts and dependencies that use Newtonsoft Json. If Newtonsoft Json is missing from the plugin, it could lead to a large number of errors as shown below:

Ensure that NewtonSoft.Json is present in your packages. Go to your project folder.

Then navigate to Packages folder. In the Packages folder. Click on manifest.json. A json file containing the project dependacies should open up.

Add the Newtonsoft Json Package on top.

    "com.unity.nuget.newtonsoft-json": "3.2.1",

The final manifest.json should look like this.

{  
    "dependencies": {
        "com.unity.nuget.newtonsoft-json": "3.2.1", 
        "com.unity.animation.rigging": "1.1.1",
        "com.unity.ide.rider": "3.0.16",
        "com.unity.ide.visualstudio": "2.0.16",
        "com.unity.ide.vscode": "1.2.5",
        "com.unity.test-framework": "1.1.33",
        "com.unity.textmeshpro": "3.0.6",
        "com.unity.timeline": "1.6.4",
        .
        .
        .
    }
}

Allowing the grpc_csharp_ext.bundle dll file in macOS

Using external DLLs in Unity on MacOS can lead to security permission issues due to Apple's strict security measures. Here's a step-by-step guide to resolving this common problem.

1. Verify the Problem:
2. Manually Allow Blocked DLLs:

   • Open System Preferences on your Mac.

   • Navigate to "Security & Privacy".

   • Under the "Security" tab, you might see a message at the bottom about the DLL being blocked. Click "Allow Anyway" or "Open Anyway" and enter password if asked.

3. Modify Gatekeeper settings: MacOS's Gatekeeper can prevent unidentified developers' software from running. To allow the DLL:

   • Open the Terminal (found in Applications > Utilities).

   • Type sudo spctl --master-disable and press Enter.

   • This command will allow apps to be downloaded from anywhere.

   • Now, try running the Unity project again.

   • After you're done, you should re-enable Gatekeeper with sudo spctl --master-enable to avoid any malware.

4. Check File Permissions: Ensure the DLL has the correct file permissions.

   • In Finder, right-click (or control-click) on the DLL file and choose "Get Info".

   • Under “Sharing & Permissions”, ensure that your user account has "Read & Write" permissions.

5. Review Unity's Plugin Settings:

   • In the Unity editor, select the DLL in the Project view.

   • In the Inspector window, make sure the appropriate platform (in this case, Mac OS X) and architecture (Apple Silicon, Intel-64) is selected for the DLL.

   • Ensure that the "Load on Startup" and other pertinent options are checked (should be enabled by default)

Mac Configuration in Player Settings during build

• Update Mac Configuration:

  • In Unity, navigate to Edit > Project Settings > Player.

  • Scroll down and click on Other Settings

  • Scroll down again to find Mac Configuration section

  • Update the Mac Configuration section (follow the below Screenshot)

Use of Invoke Speech function

Our goal in this example is to have the character welcome the player whenever the player enters a certain area, this can be done by using the Invoke Speech node that basically invokes the AI character to talk and a simple collision box.

1. Open your AI character blueprint and select the Viewport tab. Note: the character blueprint can be a MetaHuman, ReadyPlayerMe, Reallusion or even a custom one you have created, just ensure that it has the Convai Chatbot component.

2. From the Components list add a Box Collision.

1. Switch back to the Event Graph tab.

2. Select the Box Collision you just added and scroll down in the Details panel. Under Events add the On Component Begin Overlap event to your event graph.

1. Setup the following blueprint schematic which uses the Invoke Speech node from the Convai Chatbot component.

1. Enter a Trigger Message that expresses what happened (i.e. "Player Approached") and you can add a simple instruction (i.e. Greet the player).

2. Setting the In Generate Actions and In Voice Response boolean to true will let the Convai Characters perform actions and generate audio responses respectively.

3. Hit Compile and Save then run the program.

4. On approaching a certain vicinity will trigger the event and the Convai Character will greet the character as mentioned in the Trigger Message.

If the Lip Sync does not seem to cause any facial animations, even after removing all blendshapes from animations, then the following steps should help resolve the issue.

Select the Character and head to the Animator component.

Click the Avatar Field once to select the character's avatar in the Project window.

Select the Avatar and click Configure Avatar.

Select the Head option in the Mapping tab.

Select the Jaw Mapping and set it to None.

Finally scroll down and click Apply.

This will free the avatar's jaw mapping and allow the script to manipulate the Jaw bones.

• Open your Convai Character blueprint and click on Class Settings and then on ConvaiChatbot component.

• Under the Details section scroll down to the Events section and add On Transcription Received event.

• Once we have the transcription of player input, we can perform a substring search on it.

Import RPM characters from the Convai Playground

This is how you can import characters from the Convai Playground into your Unity Project.

In the Menu Bar, go the Convai > Character Importer.

Enter the Character ID and click Import.

If you are unsure how to get the character ID, click the "How do I create a character?".

You can get the character ID from the Character Description.

The downloading will take a while. On successful download, you will see the character in the scene with the same GameObject as the character ID.

This character will automatically be set up with the basic Convai Setup including the ConvaiNPC Script and Out-Of-Box Animations.

If you are facing issues with the animations in your imported character, make sure to change the animation type of Ellen@IdleNew and Ellen@TalkingNew Animations in the Assets/Convai/Animations folder to Humanoid.

Now you are ready to set up the character with transcriptions.

Step 1) Open the New Scene window

You can open the new scene window by two ways, first by pressing Ctrl + N for windows or CMD + N for Mac on your keyboard, second way is to navigate to File -> New Scene

Step 2) Select Convai Scene Template

There will be many scene templates depending upon your project, but in this guide, we are interested in Convai Scene Template so select that and click on Create button.

Step 3) Save Created Scene

You can now save the newly created scene in the project at your desired location by either pressing Ctrl + S on Windows or CMD + S on Mac. Another method is to navigate to File -> Save Scene

This is open the Save Scene Window, choose your desired location, for this demo we will save it inside Demo folder, but you can save it anywhere in the assets directory.

Give your scene a name and then click on Save Scene button

Now you can import your Convai Character or your Custom Characters by following our complete guide on it

Lip Sync System

Convai sends Visemes or Blend Shape Frame from back-end depending upon the face model the developer chooses to use and when returned Convai SDK out of the box extracts and parses it and provides it to the Convai LipSync Component, after which the component relies on the SkinMeshRenderer's Blendshape Effectors and Bone Effectors to give Convai powered NPC's realistic lipsync.

Components of LipSync System

Viseme Effector List

This is where the developer will tell the Convai SDK, which index of Blendshape Array will be effector how much from which value. To better explain its working let's understand it with a diagram.

Here, it is saying that whatever value is coming from the server will affect Blendshape at the 116th index by 0.2 multipliers and Blendshape at the 114th index by 0.5 multipliers. The engine representation of this would look something like this.

So, you can make your own Effector list or use one of the many that we ship in the SDK.

How to Create your own Viseme Effector List

Right click inside project panel and head over to Create > Convai > Expression > Viseme Skin Effector which will create a Viseme Effector List Scriptable Object and now you can define your own values.

Viseme Bone Effector List

This is where developer will tell the Convai SDK, how much each value coming from the server will affect the rotation of the bone. To better explain its working let's understand it with a diagram.

Here, bone's rotation will be affected by the values coming from the server multiplied by the values in effects. For example, for TH the value will affect the bone's rotation by a 0.2 multiplier and etc. The engine representation of this would look something like this.

So, you can make your own Bone Effector list or use one of the many that we ship in the SDK.

We use this formula to calculate the rotation

UpdateJawBoneRotation(
new Vector3(
        0.0f, 
        0.0f, 
        -90.0f - CalculateBoneEffect(FacialExpressionData.JawBoneEffector) * 30f
    )
);
UpdateTongueBoneRotation(
new Vector3(
        0.0f,
        0.0f,
        CalculateBoneEffect(FacialExpressionData.TongueBoneEffector) * 80f - 5f
    )
);

How to Create Your Own Viseme Bone Effector List

Right click inside the project panel and head over to Create > Convai > Expression > Viseme Bone Effector which will create a Viseme Bone Effector List Scriptable Object and now you can define your own values.

Convai Lipsync Component

When you attach this component to your Convai Character, you will see something like this.

Let's learn what these learns are

1. Facial Expression Data

   1. Head | Teeth | Tongue

      1. Renderer: Skin Mesh Renderer which corresponds to that specified part of the body

      2. Viseme Effectors List: How the SkinMeshRenderer's Blendshape will be affected by values coming from server.

   2. Jaw | Tongue Bone Effector

      1. How much of Bone's rotation will be affected by values coming from server?

   3. Jaw | Tongue Bone

      1. Reference to the bone which controls jaw and tongue respectively

2. Weight Blending Power

   1. Percentage to interpolate between two frames in late update.

3. Character Emotions

   1. Learn More about Character Emotions here Character Emotion

Steps to add Lipsync to your Convai Character

Now you can configure the Component according to your custom configuration or use one of the many Presets Convai ships with the SDK

Now your lipsync component would be ready to use in your application.

We will create a simple scenario where the character welcomes the player and asks them about their evening or morning based on the player's time of day.

Step 1

Activate the Narrative Design for your character in the Playground. Then, create a new Section.

Step 2

In the Objective section of the new Section, add the following text:

The time of day currently is {TimeOfDay}. Welcome the player and ask him how his {TimeOfDay} is going.

Step 3

Now, let’s back to Unity and make the necessary adjustments. Click on your NPC.

Click the Add Component button and add the Narrative Design Key Controller Component.

Step 4

In the Name field, enter TimeOfDay. In the Value field, specify the corresponding value for that variable, which could be Morning, Evening, or anything else you choose.

Setting Up Action Configurations

1. Select the Convai NPC character from the hierarchy.

2. Scroll down to the ConvaiNPC script attached to your character.

3. Click the "Add Component" button.

1. Use the checkbox to add the action script to the NPC Actions.

2. Click "Apply Changes" to confirm.

Pre-defined Actions

Convai offers predefined actions for a quick start.

1. Click the "+" button to add a new action.

2. From the dropdown menu, select "Move To."

1. Enter the action name as "Move To" (the name doesn't have to match the action choice name).

2. Leave the Animation Name field empty for now.

Repeat these steps to add more actions like "Pickup" and "Drop" etc.

Adding an Object in the Scene

1. Add any object into the scene—a sphere, a cube, a rock, etc.—that can be interacted with

2. Resize and place the object in your scene.

Adding the Convai Interactables Data Script

• Create an empty GameObject and name it "Convai Interactables."

• Attach the Convai Interactables Data script to this GameObject.

• Add characters and objects to the script by clicking the "+" button and attaching the corresponding GameObjects.

• Add the "There" object in Objects list, so that we can use the Dynamic Move Target indicator.
• Add the Dynamic Move Target Indicator and setup NavMesh agent to you NPC.

Setting Up NavMesh

To ensure your NPCs can navigate the scene:

1. Bake a NavMesh for your scene if you haven't already:

   • Go to Window > AI > Navigation.

   • In the Navigation window, under the Bake tab, adjust the settings as needed.

   • Click "Bake" to generate the NavMesh.

2. Ensure that the NPC character has a NavMeshAgent component:

   • If not already attached, click "Add Component" and search for NavMeshAgent.

   • Adjust the Agent Radius, Speed, and other parameters according to your NPC's requirements.

Adding a Dynamic Move Target Indicator

To visually indicate where your NPC will move:

• Create a new empty GameObject in the scene and name it accordingly or use the pre-made prefab named Dynamic Move Target Indicator.

• Link this Move Target Indicator to your NPC's action script so it updates dynamically when you point the cursor to the ground and ask the NPC to move to "There".

Test the Setup

1. Click "Play" to start the scene.

2. Ask the NPC, "Bring me the Box."

3. If setup properly, the NPC should walk upto the box and bring it to you

Adding Custom Actions to Your Unity NPC in Convai

Introduction

Make your NPC perform custom actions like dancing.

Action that Only Requires an Animation

1. Locate the dance animation file within our plugin.
2. Incorporate this animation into your NPC's actions.

Setting Up the Animator Controller

1. Open the Animator Controller from the Inspector window.

2. Drag and drop the dance animation onto the controller, creating a new node named "Dancing."

Adding custom Animation Action

1. Go to the Action Handler Script attached to your Convai NPC.

2. Add a new action named "Dancing."

3. In the Animation Name field, enter "Dancing" (it must exactly match the Animator Controller node name).

4. Leave the enum as "None."

Testing the Custom Action

1. Click "Play" to start the scene.

2. Instruct the NPC, "Show me a dance move," and the NPC should start dancing.

Creating Complex Custom Actions in Unity with Convai: Throwing a Rock

Introduction

Adding advanced custom actions, such as a throw action, to your NPC.

Animation Requirement

1. Grab a throw animation from Mixamo or anywhere you like.

2. Import it into Unity.

Setting Up the Animator Controller

1. Drag and drop the throw animation onto the controller, creating a new node named "Throwing." (Follow steps in #action-that-only-requires-an-animation)

Action Handler Script Setup

1. Add the "Throw" enum to the script.
2. In the "Do Action" function, add a switch case for the throw action.
3. Define the "Throw()" function.

Adding the Throw Action

1. Add a new action named "Throw" and select the "Throw" enum.

2. Leave the animation name field empty.

Adding the Object (Rock) to the Convai Interactables Data script

1. Add any rock prefab into the scene.

2. Add the rock to the Convai Interactable Data script.

Adding a location to Convai Interactables Data script

1. Add a stage/new location in the ground of the scene.

2. Add that new location game object in the Convai Interactable Data.

Testing the Complex Action

1. Click "Play" to start the scene.

2. Instruct the NPC, "Pick up the rock and throw it from the stage."

3. If everything is set up properly, the NPC should pick up the rock and throw it from the stage.

To import your custom characters into your Convai-powered Unity project, you will first need to bring your model into your project. The model needs at least two animations: one for talking and one for Idle.

Prerequisites

When you want to set up your custom character with Convai, you will need your character model and two animations: Idle and Talking.

Create an animator controller with the two animations that looks like this. You should also add a 'Talk' Boolean to ensure that you can trigger the animation. Here is a YouTube tutorial on how to set up an animator controller. This is the bare minimum animator setup that you need to do.

Step 1: Add Animator to your custom character

Select your character from the Hierarchy and Add Animator Component

Convai Plugin ships with two pre-made animation controller, you can choose these controllers or can assign your custom controller, whatever fits your need. For this demo we are going with Feminine NPC Animator

Step 2: Adding a Trigger Volume

With your custom character selected, add a Collision shape of your choice, for this demo we are going with a Capsule Collider

We will make this Collider a trigger, for this we will enable the Is Trigger option in the inspector panel

We will adjust the Center, Radius and Height of the collider such that it fits our character

Step 3: Add ConvaiNPC Component

With your Custom Character Selection add ConvaiNPC component. By doing so, your Game objectgame should look like this:

Copy your character's ID and name from Convai Playground and paste them here.

Now your Custom Character is all set to work with Convai Plugin.

You can point at Interactable Objects and Characters and ask your characters about them.

To enable this, simply drag and drop the Convai Crosshair Canvas prefab into the scene.

Step 1: Setting up Convai NPC

1. Go to your Convai NPCs:

   • Select the NPCs you want to include in the conversation.

2. Enable Group NPC Controller:

   • Click on the Group NPC Controller checkbox in the inspector panel.

   • Click Apply Changes to add the group NPC controller script.
3. Create or Find the Speech Bubble Prefab:

   • Create a new speech bubble prefab or use the one provided in the Prefabs folder.
4. Attach Required Components:

   • Add the speech bubble prefab and the player transform (optional, defaults to the main camera if not provided).

   • Set the conversation distance threshold variable (set it to zero to disable this feature, meaning NPC to NPC conversations will always happen regardless of the player’s distance).
5. Add Relevant Components:

   • Add components like lip sync, eye and head tracking, character blinking, etc., to the Convai NPC.

Step 2: Setting up NPC Manager

1. Create an NPC To NPC Manager GameObject:

   • Add an empty GameObject and rename it to NPC to NPC Manager (optional).

2. Add the NPC2NPC Conversation Manager Script:

   • Attach the NPC2NPCConversationManager script to the GameObject.
3. Configure the NPC Group List:

   • In the NPC Group List, click on the + icon to add a new list element.

   • Add the NPCs you want to include in the group conversation.

   • Set the group discussion topic.
4. Post configuration of NPCs

   • Bring the NPCs close together

   • Play the to make sure everything is working as intended.

By following these steps you can set up and manage NPC to NPC conversations in your Convai-powered application. For further customization and integration, refer to the complete implementation code and adjust it as needed for your specific use case.

This is a scriptable object which is made automatically after you hit play in the editor with Convai SDK installed and in a Scene where Convai Base Scene Essentials Prefab is present

Default Player Name

You can provide a default name of your players.

Player Name

Current name of your player, out of the box if you use our settings panel, we keep it updated automatically, if you are using some custom logic, it will be your responsibility to keep it updated, as our transcript UI use this name to show it in UI

Speaker ID

Speaker ID for the player. Please note that Speaker ID is directly linked with your API key, so for each API key there should be a unique speaker ID associated with it. We handle the creation of the Speaker ID when it's not found in the Player Prefs if the Boolean is set to true.

Create Speaker ID If Not Found

This Boolean lets the SDK if it should create a unique Speaker ID for that Player Name if it is not found in the Player Prefs.

Buttons

Reset Data

It just makes the Player Name and Speaker ID fields empty.

Copy Data

Copies the data into system buffer so you can paste it anywhere for debugging purpose

Player Pref Settings Button

1. Load: Loads the Player Name and associated Speaker ID from the player Prefs

2. Save: Saves the Player Name and associated Speaker ID from the player Prefs

3. Delete: Deletes the Player Name and associated Speaker ID from the player Prefs

How to maintain the Player Data

Convai provides a pre-made component which you can add to any GameObject to make the PlayerDataContainer work out of the box.

Choose an existing GameObject or create a new GameObject in the scene and add the ConvaiPlayerDataHandler component to your chosen GameObject and it should start working

Optional Step

You can also create the required Scriptable Object by going to Assets > Convai > Resources and right clicking in the project panel and navigating to Create > Convai > Player Data and name it ConvaiPlayerDataSO

Long-Term Memory (LTM) enables the persistent storage of conversational history with NPCs, allowing players to seamlessly continue interactions from where they previously left off, even across multiple sessions. This feature significantly enhances the realism of NPCs, aligning with our goal of creating more immersive and lifelike characters within your game.

Steps to get LTM working

1. Select your Convai Character
2. Add the Long-Term Memory Component onto your character
3. Make sure that Long Term Memory is enabled for that character

Long Term Memory should now be working for your character.

Components of the LTM System

Convai Long Term Memory Component

This component will enable or disable LTM right from the unity editor

Toggling Long Term Memory

1) Click the button provided in the component

2) It will take some time to update, and after that the new status of the LTM should be visible in the inspector.

Troubleshooting

If you encounter this error, ensure that the SpeakerID was created using the same API key currently in use. If you're uncertain about the API key used, you can reset the SpeakerID and PlayerName by accessing the ConvaiPlayerDataSO file located in Assets > Convai > Resources, allowing you to start the process anew.

Management of Speaker ID(s)

It is essential for developers to efficiently manage the Speaker ID(s) generated using their API key, as the number of IDs that can be created is limited and dependent on the subscription tier. Proper management ensures optimal usage of resources and prevents potential disruptions in the application's functionality.

Speaker ID limit per API key are as follows

You can view all the Speaker ID(s) associated with a specific API key by accessing the Convai Window within your Unity project. This feature provides a comprehensive list of IDs, allowing for easier management and monitoring.

Head over to Long Term Memory Section

If the message "No Speaker ID(s) Found" appears, there is no need to proceed with this guide. However, if a Speaker ID list is displayed, it's advisable to delete any ID(s) that are no longer in use or needed to optimize your available resources.

Convai offers comprehensive transcript and voice support for a wide range of languages. To facilitate seamless integration, our Unity plugin comes with a custom TextMeshPro (TMP) package, which includes essential fonts and required settings for major languages.

Setup

To implement these language-specific features in your project:

1. Navigate to the Convai Setup Window within Unity.

2. Locate the Package Management section.

3. Click on the "Convai Custom TMP Package" button.

Once installed, just import the character for which you require the language support, talk with it and the font will automatically render in the transcript.

For now, we provide fonts for these languages:

• Arabic

• Japanese

• Korean

• Chinese

RTL Support

We also provide support for Right-to-Left languages, like Urdu, Persian and Arabic through our Chat UIs. So, for example, if you talk with an Arabic character or if the character's name is in Arabic, the text will automatically enable the RTL feature provided by unity to reflect proper transcripts.

Convai Playground

Step 1: Select your Character in which you want to enable Narrative Design

Step 2: Open Narrative Design in Convai Playground

Select the Narrative Design option from the side panel and create your narrative design

For this sample we have created the following Narrative design

You are all set to bring your character from Convai Playground to Unity, let's hope over to Unity to continue the guide

Unity Setup

Step 1: Add the Narrative Design Manager Component

Using Add Components Button in Convai NPC (Recommended Way)

1: Select your Convai Character in the scene and look for ConvaiNPC component in the inspector panel. Click on Add Components button

2: Select Narrative Design Manager checkbox and then click on Apply Changes button

Using Unity Inspector

1: Select your Convai Character and find Add Component button in the inspector panel

2: Search for Narrative Design Manager in the search box and select it

Step 2: Setup the Narrative Design Component

After adding the Narrative Design Component, you will be able to be the following component

Getting to know the Narrative Design Component

Expanding the section event, you will see two unity events you can subscribe to, one is triggered when section starts, and another one is triggered when section ends

Getting to know about Section Triggers

Section triggers are a way to directly invoke a section in narrative design and can be used to jump to a different section in your narrative design

Step 1: Select the game object you want to make a trigger, in this example we have selected a simple cube, but it's up to your imagination.

Step 2: Add Narrative design Trigger from Add Component menu by searching for it

Step 3: Make the collider a trigger.

Step 4: Assign your Convai NPC to Convai NPC field

Now you can select from the "Trigger" dropdown which trigger should be invoked when player enters this trigger box.

We have added a way for you to manually invoke this trigger also, you can use InvokeSelectedTrigger function to invoke the trigger from any where

Invoke Trigger from any script

You can use this code block as a reference to invoke the trigger from anywhere

if(convaiNPC.TryGetComponent(out NarrativeDesignTrigger narrativeDesignTrigger))
{
    //Optional message parameter if you want to send some message while invoking
    //the trigger 
    string message = "Player has collected enough resources";
    narrativeDesignTrigger.InvokeSelectedTrigger(message);
}

How to Change the Talk Button or Any Input?

1. Double click on the "Controls" asset in your project tab.
2. You can setup multiple control schemes for different devices here, currently we have it for PC (Keyboard & Mouse) and Gamepad. For mobile, we have provided joystick and buttons, which are mapped to Gamepad controls for functionality, but you can directly add touchscreen and use its different features to trigger an Input Action. You can also add your own control scheme if you want support for a different device by clicking on "Add Control Scheme".

1. Find the Input Action you want to change in the above window. If you want to add a new Input Action, refer to the other section in documentation. In this case, we selected "Talk Key Action" to change the talk button. Click on "T [Keyboard]". In the Binding Properties window, click on the " T [Keyboard] " button in the Path field.

2. Press the " Listen " button in the top left of the opened window. If you prefer, you can choose your desired input from the categories below.

1. Press the key you want to assign and it will be reflected in the control asset.

How to Add a New Input Action?

1. First, go to the controls asset mentioned above and use the add button to create an input action. For this example, we will call it interact and provide it the binding with [E] button.

2. Then, click on the <No Binding> item to setup binding for this action. As before, you can use the listen button (has a UI bug for Windows but works) or you can select the key from dropdown. After selecting the binding (we will select [E] key for this), don't forget to press on the Save Asset option on the top menu.
3. You will now get an error saying ConvaiInputManager does not implement OnInteract. We need to implement this. Open the " ConvaiInputManager.cs " script to do so. ( " Convai / Scripts / Runtime / Core / ConvaiInputManager.cs " )
4. Your IDE might suggest you to implement missing members. If it doesn't we can manually write the OnInteract function like in the last figure shown. You receive a callback context which shows which frame input started, performed or got cancelled which you can use for different purposes. And that's it the error should be gone and you are good to go!

Overview

The Dynamic UI system is a feature within the Convai Unity SDK that provides developers a robust system for in-game communication. This feature allows for displaying messages from characters and players and supports various UI components for chat, Q&A sessions, subtitles, and custom UI types. This document will guide you through the integration, usage, and creation of custom UI types of the Dynamic UI feature in your Unity project.

Usage

Accessing the Chat UI Handler

To interact with the chat system, you need to reference the ConvaiChatUIHandler in your scripts. You can find the Transcript UI prefab in the Prefabs folder.

Here's an example of how to find and assign the handler:

private ConvaiChatUIHandler _convaiChatUIHandler;

private void OnEnable()
{
    // Find and assign the ConvaiChatUIHandler component in the scene
    _convaiChatUIHandler = ConvaiChatUIHandler.Instance;
    if (_convaiChatUIHandler != null) _convaiChatUIHandler.UpdateCharacterList();
}

Sending Messages

Once you have a reference to the ConvaiChatUIHandler, you can send messages using the following methods:

Sending Player Text

To send text as the player:

_convaiChatUIHandler.SendPlayerText(input);

• input: The string containing the player's message.

Sending Character Text

To send text as a character:

_convaiChatUIHandler.SendCharacterText(characterName, currentResponseAudio.AudioTranscript.Trim());

• characterName: The name of the character sending the message.

• currentResponseAudio.AudioTranscript: The transcript of the audio response from the character, trimmed of any leading or trailing whitespace.

Adding Custom UI Types to the Dynamic Chatbox

While the Dynamic UI system within the SDK provides several pre-built UI types, you may want to create a custom UI that better fits the style and needs of your game and it designed to be extensible, allowing developers to add their custom UI types. This is achieved by inheriting from the ChatUIBase class and implementing the required methods. The ConvaiChatUIHandler manages the different UI types and provides a system to switch between them.

Creating a Custom UI Class

To create a custom UI type, follow these steps:

Step 1: Define Your Custom Class

Create a new C# script in your Unity project and define your class to inherit from ChatUIBase. For example:

using Convai.Scripts.Utils;
using UnityEngine;

public class CustomChatUI : ChatUIBase
{
    // Implement the required methods from ChatUIBase here.
}

Step 2: Implement Required Methods

Implement the abstract methods from ChatUIBase. You must provide implementations for Initialize, SendCharacterText, and SendPlayerText:

public override void Initialize(GameObject uiPrefab)
{
    // Instantiate and set up your custom UI prefab here.
}

public override void SendCharacterText(string charName, string text, Color characterTextColor)
{
    // Handle sending character text to your custom UI here.
}

public override void SendPlayerText(string playerName, string text, Color playerTextColor)
{
    Handle sending player text to your custom UI here.
}

Step 3: Add Custom Functionality

Add any additional functionality or customization options that your custom UI may require.

Step 4: Assign and Use Your Custom UI

To use your custom UI class within the dictionaryConvaiChatUIHandler, you need to add it to the GetUIAppearances dictionary. This involves creating a prefab for your custom UI and assigning it in the ConvaiChatUIHandler.

Here's an example of how to do this:

1. Create a prefab for your custom UI and add your CustomChatUI component to it.

2. Assign the prefab to a public variable in the ConvaiChatUIHandler script.

3. Modify the InitializeUIStrategies method in the ConvaiChatUIHandler script to include your custom UI type.

[Tooltip (Prefab for the customChatUI.")]
public GameObject customChatUIPrefab;

private void InitializeUIStrategies()
{
    Existing UI types
    InitializeUI(chatBoxPrefab, UIType.ChatBox);
    InitializeUI(questionAnswerPrefab, UIType.QuestionAnswer);
    InitializeUI(subtitlePrefab, UIType.Subtitle);

    // Custom UI type
    InitializeUI(customChatUIPrefab, UIType.Custom); // Make sure to define UIType.Custom in the UIType enum
}

private void InitializeUI (GameObject uiPrefab, UIType uiType)
{
    // existing code...

    Add your custom UI initialization here
    if (uiType == UIType.Custom)
    {
        CustomChatUI customUIComponent = uiPrefab.GetComponent<CustomChatUI>();
        if (customUIComponent == null)
        {
            Debug.LogError("CustomChatUI component not found on prefab.");
            return;
        }

        customUIComponent.Initialize(uiPrefab);
        GetUIAppearances[uiType] = customUIComponent;
    }
}

1. Ensure that your custom UI type is added to the UIType enum:

public enum UIType
{
    ChatBox,
    QuestionAnswer,
    Subtitle,
    CustomUI // Your custom UI type
}

1. Now you can set your custom UI type as the active UI from the Settings Panel Settings Panel.

By following these steps, you can integrate your custom UI type into the Dynamic Chatbox system and switch between different UI types at runtime.

The Convai plugin comes with default notifications, totaling four. Here they are:

Notifications

Not Close Enough to the Character

Appears when you press the talk button but there is no active NPC nearby.

Talk Button Released Early

Appears if you release the talk button in less than 0.5 seconds.

Microphone Issue Detected

Appears when the recorded audio input level is below the threshold.

Connection Problem

Appears when there is no internet connection upon launching the application.

How to Add Your Own Notification?

Adding your custom notification is straightforward.

Let's go through the steps to add a " CharacterStartedListening" notification as an example.

1. Open the script "Convai/Scripts/Notification System/Notification Type.cs." This script stores Notification Types as enums. Give a name to your desired Notification type and add it here.

1. Right-click on "Convai / Scripts / Notification System / Scriptable Objects" and select "Create > Convai > Notification System > Notification" then create a "Notification Scriptable Object".

1. Name the created Notification Scriptable Object. Click on it, and fill in the fields in the Inspector as desired.

1. Add the created Notification Scriptable Object to "Convai/Scripts/Notification System/Scriptable Objects" under "Convai Default Notification Group" (details of Notification Group here****).

1. Your Notification is now ready. The last step is to call this Notification from where you need it. For example, if you created the " CharacterStartedTalking " Notification, find the location where your character listens and write the code.

NotificationSystemHandler.Instance.NotificationRequest(NotificationType.CharacterStartedListening);

1. Replace the parameter with the NotificationType you created. (For our example, NotificationType.CharacterStartedListening)

2. Ensure that the Convai Notification System is present in your scene. (accessible from "Convai/Prefabs/ Notification System")

All steps are complete, and you're ready to test!🙂✅

Notification Scriptable Object

This Scriptable Object stores information about a Notification

• Notification Type

• Notification Icon

• Notification Title

• Notification Message

To create a new Notification Scriptable Object, right-click anywhere in the Project Window and select "Create > Convai > Notification System > Notification"

Notification Group Scriptable Object

This Scriptable Object stores Notification Scriptable Objects as groups. When a Notification is requested, it searches for the Notification using the specified Notification Group in the Convai Notification System prefab's Notification System Handler script.

You can create different Notification groups based on your needs. Note: If your referenced Notification Group does not have the Notification you want, that Notification won't be called.

The Convai Default Notification Group has four Notifications, but you can add more or create a new group with additional notifications.

Settings Panel consists of two main sections.

• Audio Settings

• Interface Settings

Audio Settings

Microphone Settings

The Microphone Settings section is primarily for troubleshooting and testing the microphone when using the Convai plugin.

• In the Input section, you can view the microphones connected to your computer and select the desired one.

• In the Test Input field, you can record your voice using the selected microphone in the Input section. After clicking Stop, you can listen to the recorded voice and observe the sound levels.

Interface Settings

Appearance

The first setting that greets us here is the Appearance setting.

In the Appearance section, you can switch between Transcript UI designs.

There are three Transcript UI options:

• ChatBox

• QuestionAnswer

• Subtitle

Display Name

The second section in Interface Settings is the Display Name section. This section allows you to change how the user's name appears in the Transcript UI.

Notifications Checkmark

The last section in Interface Settings is the Notifications Checkmark.

Convai sometimes displays notifications on the screen to inform the user. If you want to disable these notifications, you can click the checkbox here. ( If the box is green, it's active. If empty, it's inactive )

Convai Character emit character emotions when they interact with the player and these emotions help in making the character more human-like, we are starting to implement a system which you as a developer can use to make your game more interactive using the character emotions.

Whenever the character responds to the user, we send back a list of emotions to the SDK, which look something like this

For v0 of this system, we will only be sending the emotions, in future we will apply the facial expressions corresponding to each emotion which will make the character more interactive.

4. (Android/iOS)

The Dynamic Config feature enables you to pass variables to NPCs in real time, allowing them to react dynamically to changes in the game environment. This can include the player’s current health, inventory items, or contextual world information, greatly enhancing interactivity and immersion.

Step-by-Step Guide to Setting Up Dynamic Config

First, Add the Dynamic Info Controller Component to your Convai NPC.

Create a new script or use an existing script to define a variable that will store a reference to the Dynamic Info Controller Component you added to your NPC.

Example: Passing Player Health to the NPC

• Initialize the Dynamic Info: In the script’s Start method, call the SetDynamicInfo method on the Dynamic Info Controller reference. This will set the dynamic information that the NPC will use. In this example, we’ll initialize the Player’s health as a dynamic variable.

• Updating the Dynamic Info: Whenever you need to update the NPC with new information (such as a change in Player Health), call the SetDynamicInfo method on the Dynamic Info Controller.

Sample Scenario

• At the start of the game, we set the Player’s health to 100 and send this information to the NPC as the initial value.

• Then, when the player takes damage (simulated here by pressing the "K" key), we reduce the Player’s health and update the Dynamic Info in real time so that the NPC remains aware of the Player's current health status.

Example Conversation

Below, we provide a sample conversation showcasing how the NPC can react based on the dynamic health information of the Player. By dynamically updating the Player's health, NPCs can deliver responses that feel personalized and relevant to the current gameplay.

In summary

Add the Dynamic Info Controller to your NPC. Use SetDynamicInfo to initialize the dynamic variable at the start, and call SetDynamicInfo again whenever updates are needed.

Prerequisites

Before you begin, make sure you have the following:

• Unity 2022.3 or later

• Xcode (latest version recommended)

• Apple Developer account

• Project with Convai's Unity SDK integrated and running properly

• MacBook for building and deploying to iOS/iPadOS

Step 1: Prepare Your Unity Project

1. Open your Convai-powered Unity project.

2. Ensure you have the latest version of the Convai Unity SDK into your project.

Step 2: Configure Build Settings

1. In Unity, go to File → Build Settings.

2. Select iOS as the target platform.

3. Click Switch Platform if it's not already selected.

4. Check the Development Build option for testing purposes.

Step 3: Manually add Required Files

Add link.xml

1. Create a new file named link.xml in your project's Assets folder.

2. Add the following content to the file:

<linker>
  <assembly fullname="UnityEngine">
    <type fullname="UnityEngine.Application" preserve="fields">
      <property name="platform"/>
    </type>
  </assembly>
</linker>

This file prevents potential FileNotFoundException errors related to the libgrpc_csharp_ext.x64.dylib file.

Add BuildIos.cs Script

1. Create a new C# script in Assets/Convai/Scripts named iOSBuild.cs.

2. Add the following content to the script:

#if UNITY_EDITOR && UNITY_IOS
using System.IO;
using UnityEditor;
using UnityEditor.Callbacks;
using UnityEditor.iOS.Xcode;
using UnityEngine;

public class iOSBuild : MonoBehaviour
{
    [PostProcessBuild]
    public static void OnPostProcessBuild(BuildTarget target, string path)
    {
        string projectPath = PBXProject.GetPBXProjectPath(path);
        PBXProject project = new PBXProject();
        project.ReadFromString(File.ReadAllText(projectPath));
#if UNITY_2019_3_OR_NEWER
        string targetGuid = project.GetUnityFrameworkTargetGuid();
#else
        string targetGuid = project.TargetGuidByName(PBXProject.GetUnityTargetName());
#endif
        project.AddFrameworkToProject(targetGuid, "libz.tbd", false);
        project.SetBuildProperty(targetGuid, "ENABLE_BITCODE", "NO");
        File.WriteAllText(projectPath, project.WriteToString());
    }
}
#endif

Step 4: Install required gRPC dlls for iOS:

1. Go to Convai -> Custom Package Installer

2. Click on Install iOS Build Package

3. Attach the script iOSBuild.cs to any GameObject in your scene.

Step 5: Build the Xcode Project

1. In Unity, go to File → Build Settings.

2. Click Build and choose a location to save your Xcode project.

3. Wait for Unity to generate the Xcode project.

Step 6: Configure and Build in Xcode

1. Open the generated Xcode project.

2. In Xcode, select your project in the navigator.

3. Select your target under the "TARGETS" section.

4. Go to the "Signing & Capabilities" tab.

5. Ensure that "Automatically manage signing" is checked.

6. Select your Team from the dropdown (you need an Apple Developer account for this).

7. If needed, change the Bundle Identifier to a unique string.

Step 7: Build and Run

1. Connect your iOS device to your Mac.

2. In Xcode, select your connected device as the build target.

3. Click the "Play" button or press Cmd + R to build and run the app on your device.

Troubleshooting

• If you encounter any build errors, ensure all the steps above have been followed correctly.

• Check that your Apple Developer account has the necessary provisioning profiles and certificates.

• If you face any GRPC-related issues, verify that the libgrpc_csharp_ext.a and libgrpc.a files are correctly placed in the Assets/Convai/Plugins/gRPC/Grpc.Core/runtime/ios folder.

We provide several UI options to display character and user's transcript out of the box that players can use with the Convai Plugin. You can use and customize these prefabs.

The ConvaiNPC and ConvaiGRPCAPI scripts look for GameObjects with Convai Chat UI Handler as a component, and send any transcripts to the script so that it can be displayed on screen.

Types of UI

ChatBox

Prefab Name: Convai Transcript Canvas - Chat

Both the user's and the character's transcripts are displayed one after other in a scrollable chat box.

Subtitle

Prefab Name: Convai Transcript Canvas - Subtitle

The user and character transcripts are displayed in the bottom like subtitles.

Question-Answer

Prefab Name: Convai Transcript Canvas - QA

The user's transcript is displayed in the top where as the character's transcript is displayed in the bottom.

Mobile Optimised UI Styles

Prefab Name: Convai Transcript Canvas - Mobile Subtitle

Prefab Name: Convai Transcript Canvas - Mobile QA

Prefab Name: Convai Transcript Canvas - Mobile Chat

Functions to Know

Version 3.2.0 (Current)

Released: October 31, 2024

New Features

• Implemented Dynamic Config Feature:

  • This feature allows you to dynamically pass variables to NPCs. For example, you can update NPCs with the player’s current health, inventory items, or information about the world, enhancing interactivity and immersion.

• Implemented Narrative Design Keys:

  • This feature enables dynamic variable passing within the Narrative Design section and triggers. For instance, you can use placeholders like {TimeOfDay} to create personalized dialogues, such as "Welcome, player! How is your {TimeOfDay} going?"

• Added MR Demo Scene

• Added MR Automatic Installation and Manual Installation

• Added Convai XR Package (compatibility with Meta SDK and other XR SDKs provided)

Improvements

• Added Long Term Memory API(s) to View and Delete Speaker ID(s)

• Improved VR Manual Installation

• Improved Custom Package Installation

Hotfix

• Minor Bug Fixes

Version 3.1.2

Released: September 16, 2024

Hotfix

Minor Bug Fixes

Version 3.1.1

Released: September 12, 2024

Hotfix

Fixed NPC2NPC response delay

Version 3.1.0

Released: Aug 28, 2024

What's Changed

• New Convai Setup Editor Window

• Long term memory (beta) integration in Unity SDK

• ChatBox UI revamp with RTL support

• Input system revamp and Mobile UI improvements

Improvements

• UI Improvements

  • Revamped ChatBox UI

  • Improved mobile UI

  • Implemented chat disabling feature

  • Added usage limit exceeded notification

  • Dialog box added for no API Key scenario

• API and Backend

  • Refactored ProcessUserQuery for better transcript handling

  • Implemented fuzzy matching for Action System

  • Ready Player Me and CC_Tools automatic import process

• Character and Animation

  • Updated Character Importer Pipeline

  • Added OVR effectors for RPM characters

  • Fixed animators for all characters

  • Provided Weight Multiplier for LipSync user preference

  • RPM Characters will have Lipsync added when imported