Steps to add MetaHuman to your project: -

1. In Unreal Engine go to Window > Quixel Bridge.

2. In Quixel Bridge go to MetaHumans > MetaHuman Presets.

3. Choose any character and import.

4. If any popup / warning appears click ‘Enable Missing’ and restart your project.

Overview

Dynamic Environment Info is a powerful feature that allows users to pass additional environmental data to characters without direct interaction. This enables more immersive and creative gameplay scenarios by enhancing how characters perceive their surroundings.

For example:

• The character can understand the time of day (e.g., "time of day is night").

• The character can access inventory details (e.g., "You currently have a gun and a healing potion").

• The feature supports structured data formats for richer information exchange.

Follow the steps below to integrate Dynamic Environment Info into your project.

1. Open the Character Blueprint in your project.

2. In the Begin Play event, locate the ConvaiChatbot component.

1. Set the Dynamic Environment Info variable with a string value of your choice.

• Example 1 (Simple):

  Copy

  time of day is night

• Example 2 (Structured Format):

  Copy

  inventory: {weapon: gun, tools: [flashlight, rope]}

Save and Play

1. Save your blueprint changes.

2. Hit Play to test the interaction.

3. Observe how the character dynamically responds based on the information passed.

By following these simple steps, you can unlock more engaging gameplay mechanics and enrich the interactions.

1. Open your AI character blueprint.

2. Select the Floating Pawn Movement component from the components list.

3. Set the Max Speed field under the details panel to your required speed.

Steps to change parent class for MetaHuman

1. Open content browser then Content > MetaHumans > MetaHuman blueprint.

2. Go to Class Settings and then under Details panel > Class Options > Parent Class and set it to ConvaiBaseCharacter.

1. To add animation to the body go to the 'body' component and under the details > Animation > Anim Class change it to ‘Convai_MetaHuman_BodyAnim’.

1. Similarly for the face go to the face component and under the details > Animation > Anim Class change it to ‘Convai_MetaHuman_FaceAnim’.

2. Compile and you will be good to go.

Steps to add LipSync component to MetaHuman: -

1. Open your MetaHuman blueprint.

2. Navigate to the Components section and select the Add button.

1. Search for Convai Face Sync and select.

1. Finally, LipSync is added to your MetaHuman. Compile and save it and give it a try.

Steps to change parent class of a First person player (FPP): -

1. Create a new project as a first person project or import into your already existing project. Steps to import : - Content Browser > Add > Add feature or content pack to the project > First Person > Add to Project.

Steps to make your game a default First Person game: -

Steps: - Edit > Project Settings > Maps and modes > Default Mode > Default GameMode > BP_FirstPersonGameMode.

1. Then go to All > Content > FirstPerson > Blueprints > BP_FirstPersonCharacter.

2. Click Class Settings then in the Details section Under ‘Class Options’ change the parent class to ‘ConvaiBasePlayer’.

1. Hit save and compile and you will be good to go.

1. Ensure you have the latest Convai 3.1.0 plugin or later.

2. Enable Unreal Engine's Pixel Streaming and the Pixel Streaming Player Plugins from the Plugins window.
3. In the player blueprint which has the Convai Player component, add the PixelStreamingAudio component to the list of components.
4. Click on PixelStreamingAudio component, and in the details panel find Base Submix and choose AudioInput sound submix.
5. On Begin Play in the event graph, add the following blueprint function to initialize Pixel Streaming with the Player Component.

Steps to add LipSync to MetaHuman

1. Download the plugin from this link.

2. Head to your project folder and find the ‘Plugins’ folder. If not, create a new folder named ‘Plugins’.

3. Unzip the downloaded file and copy the folder named ‘ConvaiOVRLipSync’ into the ‘Plugins’ folder and restart the Unreal Engine.

4. Open the MetaHuman Blueprint and add a component named ‘ConvaiOVRLipSync’.

1. Make sure that the ‘Face’ component is using the ‘Convai_MetaHuman_FaceAnim’ animation class.

Link for Youtube Tutorial.

1. Go to your player blueprint.

2. Click on the gear icon and select Show Inherited Variables under My Blueprint tab.

3. Search for the variable MaxChatDistance.

4. Set it to a number of your choice. Note: Set it to 0 for infinite distance.

Steps to change parent class of a First person player (FPP): -

1. Create a new project as a first person project or import into your already existing project. Steps to import : - Content Browser > Add > Add feature or content pack to the project > First Person > Add to Project.

Steps to make your game a default First Person game: -

Steps: - Edit > Project Settings > Maps and modes > Default Mode > Default GameMode > BP_FirstPersonGameMode.

1. Then go to All > Content > FirstPerson > Blueprints > BP_FirstPersonCharacter.

2. Click Class Settings then in the Details section Under ‘Class Options’ change the parent class to ‘ConvaiBasePlayer’.

1. Hit save and compile and you will be good to go.

Steps to import Reallusion Character and setting up with Convai Plugin: -

1. Create / open a project in Unreal Engine and download the Reallusion auto setup for Unreal Engine from this .

2. Download > install > Open a folder based on your Unreal Engine version.

1. Copy the Content and Plugins folder and paste them in your Unreal Engine project folder.

1. Restart your project and create a new folder (say ‘Kevin’) in your content browser.

2. Copy the .Fbx file (named as the character) from the export to ‘Kevin’ in the content browser and the FBX Import Options menu will pop up.

3. Check the following options and click Import All :-

   • Use T0 As Ref Pose.

   • Import Morph Targets.

4. Create a new folder named Animations within the Kevin folder and import all the animation file (named ‘_motion’ at the end) from the exported file from Reallusion.

1. The FBX Import Options pop up again and Uncheck the Import Mesh and select your imported skeleton.

2. Under Animation > advanced. Check the Use Default Sample Rate.

3. Click Import All.

4. Now install the Convai Plugin from Epic marketplace and restart the engine.

5. Edit > Plugins.

6. Search for ‘Convai’ and enable it by clicking the checkbox and restart.

7. Create a new Blueprint Class within Kevin folder and under ALL CLASSES select ‘ConvaiBaseCharacter’.

1. Open up the blueprint. Components > add > Skeletal Mesh.

1. Under the details tab go to mesh > Skeletal Mesh and select the imported mesh named Kevin form Reallusion.

2. Finally add the Character Id by selecting the Character (from Convai website) you just added and you can enjoy talking to your AI buddy.

Steps to add a ReadyPlayerMe Character

1. Create a new blueprint and select ConvaiRPM_Character as the parent class which you can find under All Classes .
2. Drag the blueprint to the scene and the default ReadyPlayerMe character should appear.
3. Create a new character on and copy the Character ID, you can also edit the avatar by clicking on the Edit Avatar icon on the top right of the avatar preview window.
4. Back to Unreal, click the character in the scene and under the details panel find the Char ID field and paste the copied character ID into it.
5. Hit Play , and wait for a few seconds then the character should now load into the game.

Steps to add Convai ReadyPlayerMe plugin to your project: -

1. Go to this Drive Link.

2. Download the version corresponding to your Unreal Engine.

3. In your project directory create a folder named Plugins if it does not already exist.

4. Extract the contents of the downloaded zip in the Plugins folder, the final folder hierarchy should look like this:

   Copy

   ├── Your Project Name
       ├── Plugins
           ├── ConvaiReadyPlayerMe
           ├── glTFRuntime

Steps to create a Reallusion Character using Character Creator 4: -

1. Use the Character Creation tool by Reallusion.

2. Create / use default a character present there and add animations of your choice.

(Here we have used an already existing character named ‘CC4 Kevin’ and have added the idle animation and walking animation.

Link for Youtube Tutorial.)

1. Export it in FBX format.

File > Export > FBX > Clothed Character

1. Keep the following settings in Export FBX.

   • Target Tool Preset: - Unreal

   • FBX Options: - Mesh and Motion

   • Max Texture Size: - 4096 ( Choose the maximum available)

   • Frame Rate: - 30

   Check the custom section. Click the Load Perform button.

   • Uncheck First Frame in bind pose.

   • Check Export Mesh and Motion individually.

   • Check Save One Motion per File.

   Checking the Delete Hidden Faces option may avoid rendering issues.

1. Click Export.

Narrative Design can be extended using Unreal Engine Blueprints, allowing you to invoke triggers created in the Narrative Graph, as well as access dynamic variables such as the time of day or relevant information like inventory contents in the game.

Steps to bind objects to Reallusion character: -

1. Go to Window > Place Actors > Search NavMeshBoundsVolume and drag it in the scene.

1. Click the character in the scene (in this case click the Reallusion character) and head to the Details Panel.

   Details > Default > Objects > Click Add Element

1. Now we need to select a reference for the object from the scene. Click on Pick Actor from Scene.

1. Then select any object from the scene. We can also provide a name and description to the object in the scene. This will allow the player to interact with the objects.

2. Save and then hit play to test the character to perform certain actions related to the object present which you just added.

Use of Invoke Narrative Design Trigger in Unreal Engine.

1. Develop the logical flow for your specific use case. In this instance, we have created a simple museum tour guide scenario.

2. Our goal is to invoke the trigger Start Tour in the narrative Design graph, using the Invoke Narrative Design Trigger .

3. The Trigger Name in the function should be the same as the Trigger name on the graph.

1. The above example showcased only one Trigger. The use of more than one Trigger is also possible based on execution logic.

Use of Invoke Speech function

Our goal in this example is to have the character welcome the player whenever the player enters a certain area, this can be done by using the Invoke Speech node that basically invokes the AI character to talk and a simple collision box.

1. Open your AI character blueprint and select the Viewport tab. Note: the character blueprint can be a MetaHuman, ReadyPlayerMe, Reallusion or even a custom one you have created, just ensure that it has the Convai Chatbot component.

2. From the Components list add a Box Collision.

1. Switch back to the Event Graph tab.

2. Select the Box Collision you just added and scroll down in the Details panel. Under Events add the On Component Begin Overlap event to your event graph.

1. Setup the following blueprint schematic which uses the Invoke Speech node from the Convai Chatbot component.

1. Enter a Trigger Message that expresses what happened (i.e. "Player Approached") and you can add a simple instruction (i.e. Greet the player).

2. Setting the In Generate Actions and In Voice Response boolean to true will let the Convai Characters perform actions and generate audio responses respectively.

3. Hit Compile and Save then run the program.

4. On approaching a certain vicinity will trigger the event and the Convai Character will greet the character as mentioned in the Trigger Message.

• Open your Convai Character blueprint and click on Class Settings and then on ConvaiChatbot component.

• Under the Details section scroll down to the Events section and add On Transcription Received event.

• Once we have the transcription of player input, we can perform a substring search on it.

Steps to add simple actions to your Convai character:-

• Select your Convai Character and navigate to the Details panel. Within this interface, locate the Convai Info section.

• Select the Add Element icon (+) and input the desired action you wish to execute. For example, Print.

• Open the character blueprint to which you have just now added the action.

• Add a new event with the same name. Print in this case and define the logic for the function you just named and run the function Handle Actions Completion with Is Successful set to true.
• Hit compile and ask your Convai Character to perform the action.

Actions can have parameters, for example: Picks Up is an action that expects an object that will be picked up, in this guide we will see how to parse these parameters as well as other types of parameters such as text.

Referencing Objects or Characters

Referencing objects or characters in your AI character is very important for two goals: First, it will have knowledge that the object/character exists, and second, you will be able to get a reference to these objects or characters when actions that relate to them are triggered, such as Move to - Pick up - Follow will always be triggered with objects or characters

Referencing objects requires that you add the object to the Environment object that is inside the ConvaiChatbot component. There are two ways to do so:

1. The first method is to select the character in the scene and then under the details panel, add the references to the Objects array under Convai Info, this method requires that you inherit from the ConvaiBaseCharacter blueprint.
2. At the begin play event on the character blueprint, loop over all the objects that you want the character to know about then add to the environment object, in the following blueprint, we tagged the objects we want to add to the environment to make it easier to fetch those at begin play, then we used the second and third tags for names and descriptions respectively.

Parsing Object and Character Types

Once you have managed to add your objects and characters to the environment, let's go over a quick scenario to see how we handle the response:

• Create an action named Looks At that forces the AI character to look towards a certain object or character.
• Create an event with the same name in the character blueprint, and add an input parameter of type Convai Result Action, this will contain the parameters required for the action which is in our case the object/character to look at.
• Now let's finish the implementation as follows, we will break the Action Parameter structure and then set the The Related Object Or Character as the Main Character, note that this is a quick trick to get the AI character to look at the referenced object or character.

Parsing Other Parameter Types

To be continued..

1. Use follows with other Convai characters or players in the environment.

• Approach a Convai character and ask it to follow other Convai characters or a Convai player.

• Then you start moving around and you will find that the Convai Character starts following you.

2. Use follows with other characters/objects present in the Environment.

• Ask the character to follow the added character/object present in the Environment.

• Even if we change the position of the object (cube here) the Convai Character would still follow the cube.

These actions include:

• Moves To: Character can move to another character or object

• Follows: Character can follow you or follow other objects/characters

• Waits For: Character can wait for some time before doing another action, for example: Wait for 10 seconds then throw a grenade

Overview

This document details the process for adding descriptions to actions within Convai characters, improving AI understanding and action execution. Action descriptions provide contextual information, aiding in more accurate and contextually appropriate behaviors.

Why Add Action Descriptions?

Action descriptions offer contextual hints to the AI, helping determine when and how to trigger actions based on gameplay and conversation scenarios, enhancing the gaming experience through more accurate AI responses.

How to Add Descriptions to Actions

1. Identify the Action: Determine the action needing a description, such as "Sit Down" or "Crouch".

2. Format for Adding Descriptions: Use the format Action Name <Description> to add descriptions. The description should be clear and concise, providing exact indications for the AI.

3. Examples:

   • Sit Down:

     Copy

     sit down <Use this action to make the character sit, suitable for resting, healing, or engaging in conversations with NPCs. Ideal for safe zones or interaction points>

   • Crouch:

     Copy

     crouch <This action lowers the character's profile, used for stealth or navigating low spaces>

   • Attack Enemy:

     Copy

     attack enemy <Trigger this action when the player chooses to attack an enemy unit>

   • Send Email:

     Copy

     send email <Initiate this action to allow the user to compose and send an email>

Implementing Action Descriptions

1. Navigate to Convai Info Section: Select your Convai character in the editor and access the Convai Info section.
2. Adding Action with Description: Click the Add Element (+) icon to input the action along with its description as previously formatted.

Additional Tips

• Clear and Specific Descriptions: Ensure descriptions are straightforward, precisely indicating action triggering conditions.

• Use English.

1. Use Moves to with other Convai Characters present in the Environment.

Let's assume there are two Convai characters present in the virtual environment. We can ask one of the characters to move to the other character as shown here.

2. Use Moves To with other objects present in environment.

Steps: -

• Click on your Convai character in present in scene.

• In Outliner go to Convai Info section and then Click on the + icon near Objects.

• Click on Pick Actor from Scene tool

• Select any object from the scene and give it a Name and Description of choice.

• Save the changes and ask your Convai character to move to the object your named.

• The Convai character finally move to the object you added to its object list.

Steps:

1. In your player blueprint, make sure you have the Convai Player Component already included. Otherwise, you can add it manually or by following this guide.

2. Apply the following blueprint setup in the player blueprint.
3. Hit play to test, press and hold space to talk and release to print a transcription of your input voice.

Please see the instructions below or check out our latest tutorial video on YouTube.

1. Navigate to your player blueprint.

2. Under My Blueprints tab, click the gear icon and select Show Inherited Variables.
3. Find and enable the two boolean variables: - Enable Multiplayer - Enable Voice Chat

Steps to enable long term memory

1. Select your character in the playground on the Convai website.

2. Go to the Memories Tab, and then to Memory Settings. Enable Long Term Memory for the character there.

Steps to setup LTM in Unreal Engine

1. Go to, Project Settings -> Plugins -> Convai. where you have set the API key.

2. There, click on the Manage Speaker ID. button under Long Term Memory. This will spawn an editor utility to create, delete, and list speaker ids.

1. If you have already created a Speaker ID, you can click List All Speaker IDs. Then, go to Project Settings -> Plugins -> Convai, and under Long Term Memory, you will find the Speaker IDs array. Copy the desired Speaker ID.

2. Now, go to the Player Blueprint and select the ConvaiPlayer component. In the details panel, you will find a Speaker ID field under the Convai category. Past the desired Speaker ID there.

• Open your character blueprint which contains the Convai Chatbot component.

• Click Class Settings and then Convai Chatbot component.

• On the right-hand side Details panel under the Sound section modify the Pitch Multiplier.

• A value greater than 1 will ramp up the speed and vice-versa.

Steps to customize the default push to talk hotkey: -

• Navigate to ConvaiBasePlayer blueprint.

• On opening, you will find this section.

• Replace the default 'T' hotkey with your hotkey.

Steps to Set Up the 3D chat widget

1. Go to the following Google Drive link and download the Convai Convenience Pack.

2. Unzip the file and place it in ProjectDirectory/Content folder.

Adding the BP Convai 3DWidget Component

1. Navigate to your player blueprint.

2. Go to the Viewport tab.

1. Under the Components tab, click the Add button, search for BP Convai 3DWidget Component, and add it.

1. Select the newly added component and adjust the transform according to your needs.

We will create a simple scenario where the character welcomes the player and asks them about their evening or morning based on the player's time of day.

1. In the playground, enable Narrative Design on your character and change the starting section name to Welcome.

2. Add the following to the Objective field of the Welcome section: The time of day currently is {TimeOfDay}. Welcome the player and ask him how his {TimeOfDay} is going. Notice that by adding any string between curly brackets it becomes a variable, and what we did here is adding the time of day as a variable, then from Unreal we can pass either the word "Morning" or "Evening" and the character will respond accordingly.
3. Back in Unreal, open the character's blueprint.

4. Set the Narrative Template Keys variable with a map containing the same variable name TimeOfDay and for demonstration purposes we will hard code the value to "Morning".
5. Start the play mode and try it out.

6. Feel free to try other scenarios and settings to align better with your usecase.

1. Navigate to your AI character blueprint.

2. Drag the Convai Chatbot component into the event graph.

3. Get the Environment Object and make sure it is valid.

   1. Use the function Set Main Character to define the current speaker to the AI character. Additionally, if you're using Convai's animation blueprints then setting the Main Character would cause the AI character to look at the Main Character reference.

   2. Use the function Set Attention Object to set which object is currently being talked about, this function also automatically adds the input object to the list of already existing objects in the environment.