Steps to add MetaHuman to your project: -

1. In Unreal Engine go to Window > Quixel Bridge.

2. In Quixel Bridge go to MetaHumans > MetaHuman Presets.

3. Choose any character and import.

4. If any popup / warning appears click ‘Enable Missing’ and restart your project.

Prerequisites

Installing Visual Studio for Windows

• Download Visual Studio from here.

• Ensure having the required C++ toolchains mentioned here. If you already have Visual Studio installed, you may open the installer and select 'Modify' to add the above mentioned toolchains.

Installing XCode for Mac

• Download XCode from the app store

• For UE 5.3 and UE 5.0, follow this guide to enable microphone permissions.

Installing the plugin

There are two methods to install the plugin depending on your requirements

1. Directly from the Marketplace Link: Recommended for easy installation and tracking updates to the plugin. (UE 5.1 - 5.3)

2. Building from Github Source: For source UE builds and for unsupported UE versions on the marketplace, but not guranteed as the marketplace approach. (UE 4.26 - 5.0)

Set up your project

1. From the top toolbar, go to Edit > Plugins.

2. Find the Convai plugin.

3. Click the checkbox to enable the plugin.
4. Restart Unreal Engine.

Set up the API key

1. Find your API key.

2. Go to Edit > Project Settings.

3. Choose Convai under the Plugins section on the left bar.
4. Paste the API key into the API Key field.

Guides

More on the Unreal Engine SDK

Overview

Dynamic Environment Info is a powerful feature that allows users to pass additional environmental data to characters without direct interaction. This enables more immersive and creative gameplay scenarios by enhancing how characters perceive their surroundings.

For example:

• The character can understand the time of day (e.g., "time of day is night").

• The character can access inventory details (e.g., "You currently have a gun and a healing potion").

• The feature supports structured data formats for richer information exchange.

Follow the steps below to integrate Dynamic Environment Info into your project.

1. Open the Character Blueprint in your project.

2. In the Begin Play event, locate the ConvaiChatbot component.

1. Set the Dynamic Environment Info variable with a string value of your choice.

• Example 1 (Simple):

• Example 2 (Structured Format):

Save and Play

1. Save your blueprint changes.

2. Hit Play to test the interaction.

3. Observe how the character dynamically responds based on the information passed.

By following these simple steps, you can unlock more engaging gameplay mechanics and enrich the interactions.

Steps to add LipSync component to MetaHuman: -

1. Open your MetaHuman blueprint.

2. Navigate to the Components section and select the Add button.

1. Search for Convai Face Sync and select.

1. Finally, LipSync is added to your MetaHuman. Compile and save it and give it a try.

1. Open your AI character blueprint.

2. Select the Floating Pawn Movement component from the components list.

3. Set the Max Speed field under the details panel to your required speed.

Steps to change parent class of a First person player (FPP): -

1. Create a new project as a first person project or import into your already existing project. Steps to import : - Content Browser > Add > Add feature or content pack to the project > First Person > Add to Project.

Steps to make your game a default First Person game: -

Steps: - Edit > Project Settings > Maps and modes > Default Mode > Default GameMode > BP_FirstPersonGameMode.

1. Then go to All > Content > FirstPerson > Blueprints > BP_FirstPersonCharacter.

2. Click Class Settings then in the Details section Under ‘Class Options’ change the parent class to ‘ConvaiBasePlayer’.

1. Hit save and compile and you will be good to go.

Steps to change parent class for MetaHuman

1. Open content browser then Content > MetaHumans > MetaHuman blueprint.

2. Go to Class Settings and then under Details panel > Class Options > Parent Class and set it to ConvaiBaseCharacter.

1. To add animation to the body go to the 'body' component and under the details > Animation > Anim Class change it to ‘Convai_MetaHuman_BodyAnim’.

1. Similarly for the face go to the face component and under the details > Animation > Anim Class change it to ‘Convai_MetaHuman_FaceAnim’.

2. Compile and you will be good to go.

1. Ensure you have the latest Convai 3.1.0 plugin or later.

2. Enable Unreal Engine's Pixel Streaming and the Pixel Streaming Player Plugins from the Plugins window.
3. In the player blueprint which has the Convai Player component, add the PixelStreamingAudio component to the list of components.
4. Click on PixelStreamingAudio component, and in the details panel find Base Submix and choose AudioInput sound submix.
5. On Begin Play in the event graph, add the following blueprint function to initialize Pixel Streaming with the Player Component.

Steps to add LipSync to MetaHuman

1. Download the plugin from this link.

2. Head to your project folder and find the ‘Plugins’ folder. If not, create a new folder named ‘Plugins’.

3. Unzip the downloaded file and copy the folder named ‘ConvaiOVRLipSync’ into the ‘Plugins’ folder and restart the Unreal Engine.

4. Open the MetaHuman Blueprint and add a component named ‘ConvaiOVRLipSync’.

1. Make sure that the ‘Face’ component is using the ‘Convai_MetaHuman_FaceAnim’ animation class.

Link for Youtube Tutorial.

Steps to add a ReadyPlayerMe Character

1. Create a new blueprint and select ConvaiRPM_Character as the parent class which you can find under All Classes .
2. Drag the blueprint to the scene and the default ReadyPlayerMe character should appear.
3. Create a new character on Convai Playground and copy the Character ID, you can also edit the avatar by clicking on the Edit Avatar icon on the top right of the avatar preview window.
4. Back to Unreal, click the character in the scene and under the details panel find the Char ID field and paste the copied character ID into it.
5. Hit Play , and wait for a few seconds then the character should now load into the game.

Steps to change parent class of a First person player (FPP): -

1. Create a new project as a first person project or import into your already existing project. Steps to import : - Content Browser > Add > Add feature or content pack to the project > First Person > Add to Project.

Steps to make your game a default First Person game: -

Steps: - Edit > Project Settings > Maps and modes > Default Mode > Default GameMode > BP_FirstPersonGameMode.

1. Then go to All > Content > FirstPerson > Blueprints > BP_FirstPersonCharacter.

2. Click Class Settings then in the Details section Under ‘Class Options’ change the parent class to ‘ConvaiBasePlayer’.

1. Hit save and compile and you will be good to go.

1. Go to your player blueprint.

2. Click on the gear icon and select Show Inherited Variables under My Blueprint tab.

3. Search for the variable MaxChatDistance.

4. Set it to a number of your choice. Note: Set it to 0 for infinite distance.

Steps to import Reallusion Character and setting up with Convai Plugin: -

1. Create / open a project in Unreal Engine and download the Reallusion auto setup for Unreal Engine from this .

2. Download > install > Open a folder based on your Unreal Engine version.

1. Copy the Content and Plugins folder and paste them in your Unreal Engine project folder.

1. Restart your project and create a new folder (say ‘Kevin’) in your content browser.

2. Copy the .Fbx file (named as the character) from the export to ‘Kevin’ in the content browser and the FBX Import Options menu will pop up.

3. Check the following options and click Import All :-

   • Use T0 As Ref Pose.

   • Import Morph Targets.

4. Create a new folder named Animations within the Kevin folder and import all the animation file (named ‘_motion’ at the end) from the exported file from Reallusion.

1. The FBX Import Options pop up again and Uncheck the Import Mesh and select your imported skeleton.

2. Under Animation > advanced. Check the Use Default Sample Rate.

3. Click Import All.

4. Now install the Convai Plugin from Epic marketplace and restart the engine.

5. Edit > Plugins.

6. Search for ‘Convai’ and enable it by clicking the checkbox and restart.

7. Create a new Blueprint Class within Kevin folder and under ALL CLASSES select ‘ConvaiBaseCharacter’.

1. Open up the blueprint. Components > add > Skeletal Mesh.

1. Under the details tab go to mesh > Skeletal Mesh and select the imported mesh named Kevin form Reallusion.

2. Finally add the Character Id by selecting the Character (from Convai website) you just added and you can enjoy talking to your AI buddy.

Steps to add Convai ReadyPlayerMe plugin to your project: -

1. Go to this Drive Link.

2. Download the version corresponding to your Unreal Engine.

3. In your project directory create a folder named Plugins if it does not already exist.

4. Extract the contents of the downloaded zip in the Plugins folder, the final folder hierarchy should look like this:

   Copy

   ├── Your Project Name
       ├── Plugins
           ├── ConvaiReadyPlayerMe
           ├── glTFRuntime

Steps to create a Reallusion Character using Character Creator 4: -

1. Use the Character Creation tool by Reallusion.

2. Create / use default a character present there and add animations of your choice.

(Here we have used an already existing character named ‘CC4 Kevin’ and have added the idle animation and walking animation.

Link for Youtube Tutorial.)

1. Export it in FBX format.

File > Export > FBX > Clothed Character

1. Keep the following settings in Export FBX.

   • Target Tool Preset: - Unreal

   • FBX Options: - Mesh and Motion

   • Max Texture Size: - 4096 ( Choose the maximum available)

   • Frame Rate: - 30

   Check the custom section. Click the Load Perform button.

   • Uncheck First Frame in bind pose.

   • Check Export Mesh and Motion individually.

   • Check Save One Motion per File.

   Checking the Delete Hidden Faces option may avoid rendering issues.

1. Click Export.

Use of Invoke Narrative Design Trigger in Unreal Engine.

1. Develop the logical flow for your specific use case. In this instance, we have created a simple museum tour guide scenario.

1. Once the logic is decided we can move to Unreal Engine. (In this guide we will use the same setup described in this guide)

2. Our goal is to invoke the trigger Start Tour in the narrative Design graph, using the Invoke Narrative Design Trigger .

3. The Trigger Name in the function should be the same as the Trigger name on the graph.

1. The above example showcased only one Trigger. The use of more than one Trigger is also possible based on execution logic.

Narrative Design can be extended using Unreal Engine Blueprints, allowing you to invoke triggers created in the Narrative Graph, as well as access dynamic variables such as the time of day or relevant information like inventory contents in the game.

Steps to bind objects to Reallusion character: -

1. Go to Window > Place Actors > Search NavMeshBoundsVolume and drag it in the scene.

1. Click the character in the scene (in this case click the Reallusion character) and head to the Details Panel.

   Details > Default > Objects > Click Add Element

1. Now we need to select a reference for the object from the scene. Click on Pick Actor from Scene.

1. Then select any object from the scene. We can also provide a name and description to the object in the scene. This will allow the player to interact with the objects.

2. Save and then hit play to test the character to perform certain actions related to the object present which you just added.

• Open your Convai Character blueprint and click on Class Settings and then on ConvaiChatbot component.

• Under the Details section scroll down to the Events section and add On Transcription Received event.

• Once we have the transcription of player input, we can perform a substring search on it.

1. Use Moves to with other Convai Characters present in the Environment.

Let's assume there are two Convai characters present in the virtual environment. We can ask one of the characters to move to the other character as shown here.

2. Use Moves To with other objects present in environment.

Steps: -

• Click on your Convai character in present in scene.

• In Outliner go to Convai Info section and then Click on the + icon near Objects.

• Click on Pick Actor from Scene tool

• Select any object from the scene and give it a Name and Description of choice.

• Save the changes and ask your Convai character to move to the object your named.

• The Convai character finally move to the object you added to its object list.

Use of Invoke Speech function

Our goal in this example is to have the character welcome the player whenever the player enters a certain area, this can be done by using the Invoke Speech node that basically invokes the AI character to talk and a simple collision box.

1. Open your AI character blueprint and select the Viewport tab. Note: the character blueprint can be a MetaHuman, ReadyPlayerMe, Reallusion or even a custom one you have created, just ensure that it has the Convai Chatbot component.

2. From the Components list add a Box Collision.

1. Switch back to the Event Graph tab.

2. Select the Box Collision you just added and scroll down in the Details panel. Under Events add the On Component Begin Overlap event to your event graph.

1. Setup the following blueprint schematic which uses the Invoke Speech node from the Convai Chatbot component.

1. Enter a Trigger Message that expresses what happened (i.e. "Player Approached") and you can add a simple instruction (i.e. Greet the player).

2. Setting the In Generate Actions and In Voice Response boolean to true will let the Convai Characters perform actions and generate audio responses respectively.

3. Hit Compile and Save then run the program.

4. On approaching a certain vicinity will trigger the event and the Convai Character will greet the character as mentioned in the Trigger Message.

These actions include:

• Moves To: Character can move to another character or object

• Follows: Character can follow you or follow other objects/characters

• Waits For: Character can wait for some time before doing another action, for example: Wait for 10 seconds then throw a grenade

Actions can have parameters, for example: Picks Up is an action that expects an object that will be picked up, in this guide we will see how to parse these parameters as well as other types of parameters such as text.

Referencing Objects or Characters

Referencing objects or characters in your AI character is very important for two goals: First, it will have knowledge that the object/character exists, and second, you will be able to get a reference to these objects or characters when actions that relate to them are triggered, such as Move to - Pick up - Follow will always be triggered with objects or characters

Referencing objects requires that you add the object to the Environment object that is inside the ConvaiChatbot component. There are two ways to do so:

1. The first method is to select the character in the scene and then under the details panel, add the references to the Objects array under Convai Info, this method requires that you inherit from the ConvaiBaseCharacter blueprint.
2. At the begin play event on the character blueprint, loop over all the objects that you want the character to know about then add to the environment object, in the following blueprint, we tagged the objects we want to add to the environment to make it easier to fetch those at begin play, then we used the second and third tags for names and descriptions respectively.

Parsing Object and Character Types

Once you have managed to add your objects and characters to the environment, let's go over a quick scenario to see how we handle the response:

• Create an action named Looks At that forces the AI character to look towards a certain object or character.
• Create an event with the same name in the character blueprint, and add an input parameter of type Convai Result Action, this will contain the parameters required for the action which is in our case the object/character to look at.
• Now let's finish the implementation as follows, we will break the Action Parameter structure and then set the The Related Object Or Character as the Main Character, note that this is a quick trick to get the AI character to look at the referenced object or character.

Parsing Other Parameter Types

To be continued..

1. Use follows with other Convai characters or players in the environment.

• Approach a Convai character and ask it to follow other Convai characters or a Convai player.

• Then you start moving around and you will find that the Convai Character starts following you.

2. Use follows with other characters/objects present in the Environment.

• Ask the character to follow the added character/object present in the Environment.

• Even if we change the position of the object (cube here) the Convai Character would still follow the cube.

Steps to add simple actions to your Convai character:-

• Select your Convai Character and navigate to the Details panel. Within this interface, locate the Convai Info section.

• Select the Add Element icon (+) and input the desired action you wish to execute. For example, Print.

• Open the character blueprint to which you have just now added the action.

• Add a new event with the same name. Print in this case and define the logic for the function you just named and run the function Handle Actions Completion with Is Successful set to true.
• Hit compile and ask your Convai Character to perform the action.

Overview

This document details the process for adding descriptions to actions within Convai characters, improving AI understanding and action execution. Action descriptions provide contextual information, aiding in more accurate and contextually appropriate behaviors.

Why Add Action Descriptions?

Action descriptions offer contextual hints to the AI, helping determine when and how to trigger actions based on gameplay and conversation scenarios, enhancing the gaming experience through more accurate AI responses.

How to Add Descriptions to Actions

1. Identify the Action: Determine the action needing a description, such as "Sit Down" or "Crouch".

2. Format for Adding Descriptions: Use the format Action Name <Description> to add descriptions. The description should be clear and concise, providing exact indications for the AI.

3. Examples:

   • Sit Down:

     Copy

     sit down <Use this action to make the character sit, suitable for resting, healing, or engaging in conversations with NPCs. Ideal for safe zones or interaction points>

   • Crouch:

     Copy

     crouch <This action lowers the character's profile, used for stealth or navigating low spaces>

   • Attack Enemy:

     Copy

     attack enemy <Trigger this action when the player chooses to attack an enemy unit>

   • Send Email:

     Copy

     send email <Initiate this action to allow the user to compose and send an email>

Implementing Action Descriptions

1. Navigate to Convai Info Section: Select your Convai character in the editor and access the Convai Info section.
2. Adding Action with Description: Click the Add Element (+) icon to input the action along with its description as previously formatted.

Additional Tips

• Clear and Specific Descriptions: Ensure descriptions are straightforward, precisely indicating action triggering conditions.

• Use English.

Steps:

1. In your player blueprint, make sure you have the Convai Player Component already included. Otherwise, you can add it manually or by following this guide.

2. Apply the following blueprint setup in the player blueprint.
3. Hit play to test, press and hold space to talk and release to print a transcription of your input voice.

Please see the instructions below or check out our latest tutorial video on YouTube.

1. Navigate to your player blueprint.

2. Under My Blueprints tab, click the gear icon and select Show Inherited Variables.
3. Find and enable the two boolean variables: - Enable Multiplayer - Enable Voice Chat

Steps to enable long term memory

1. Select your character in the playground on the Convai website.

2. Go to the Memories Tab, and then to Memory Settings. Enable Long Term Memory for the character there.

Steps to setup LTM in Unreal Engine

1. Go to, Project Settings -> Plugins -> Convai. where you have set the API key.

2. There, click on the Manage Speaker ID. button under Long Term Memory. This will spawn an editor utility to create, delete, and list speaker ids.

1. If you have already created a Speaker ID, you can click List All Speaker IDs. Then, go to Project Settings -> Plugins -> Convai, and under Long Term Memory, you will find the Speaker IDs array. Copy the desired Speaker ID.

2. Now, go to the Player Blueprint and select the ConvaiPlayer component. In the details panel, you will find a Speaker ID field under the Convai category. Past the desired Speaker ID there.

Steps to Set Up the 3D chat widget

1. Go to the following and download the Convai Convenience Pack.

2. Unzip the file and place it in ProjectDirectory/Content folder.

Adding the BP Convai 3DWidget Component

1. Navigate to your player blueprint.

2. Go to the Viewport tab.

1. Under the Components tab, click the Add button, search for BP Convai 3DWidget Component, and add it.

1. Select the newly added component and adjust the transform according to your needs.

• Open your character blueprint which contains the Convai Chatbot component.

• Click Class Settings and then Convai Chatbot component.

• On the right-hand side Details panel under the Sound section modify the Pitch Multiplier.

• A value greater than 1 will ramp up the speed and vice-versa.

Steps to customize the default push to talk hotkey: -

• Navigate to ConvaiBasePlayer blueprint.

• On opening, you will find this section.

• Replace the default 'T' hotkey with your hotkey.

Convai Environment is used as input to the StartTalking() or SendText() functions in the Convai Player component, and it allows the character to generate actions.

Functions

We will create a simple scenario where the character welcomes the player and asks them about their evening or morning based on the player's time of day.

1. In the playground, enable Narrative Design on your character and change the starting section name to Welcome.

2. Add the following to the Objective field of the Welcome section: The time of day currently is {TimeOfDay}. Welcome the player and ask him how his {TimeOfDay} is going. Notice that by adding any string between curly brackets it becomes a variable, and what we did here is adding the time of day as a variable, then from Unreal we can pass either the word "Morning" or "Evening" and the character will respond accordingly.
3. Back in Unreal, open the character's blueprint.

4. Set the Narrative Template Keys variable with a map containing the same variable name TimeOfDay and for demonstration purposes we will hard code the value to "Morning".
5. Start the play mode and try it out.

6. Feel free to try other scenarios and settings to align better with your usecase.

Convai Player is an Actor component responsible for capturing microphone audio and streaming it to a Convai character.

Text to Speech and Speech To Speech

Microphone Settings

StartTalking() and SendText() Parameters

1. Navigate to your AI character blueprint.

2. Drag the Convai Chatbot component into the event graph.

3. Get the Environment Object and make sure it is valid.

   1. Use the function Set Main Character to define the current speaker to the AI character. Additionally, if you're using Convai's animation blueprints then setting the Main Character would cause the AI character to look at the Main Character reference.

   2. Use the function Set Attention Object to set which object is currently being talked about, this function also automatically adds the input object to the list of already existing objects in the environment.

Properties

Speech To Text From Sound Wave

• Description: Transcribes the provided audio.

Inputs:

• Sound Wave: The recorded output from the microphone, please take a look over the tutorial/sample project for an example on how to use Convai voice capture component with the API.

Outputs:

✅ [On Success]

Response: Contains the text output of the audio file if the API was successful, otherwise might contain information on why the API call failed.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Speech To Text From Sound Wave

• Description: Transcribes the provided audio file.

Inputs:

• Filename: The path to the recorded audio file on your local disk, the file should be in a .wav format.

Outputs:

✅ [On Success]

Response: Contains the text output of the audio file if the API was successful, otherwise might contain information on why the API call failed.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Create Character

• Description: Create a new character and get the character ID for it.

Inputs:

• Char Name: Name of the character.

• Voice: Voice name [MALE/FEMALE].

• Backstory: Backstory for the new character.

Outputs:

✅ [On Success]

• Char ID: character id for the new character.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Get All Character IDs

• Description: Get a list of character IDs belonging to the user.

Outputs:

✅ [On Success]

• Char IDs: list of characters.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Get Character Details

• Description: Fetch all details of a character including backstory, voice, etc.

Inputs:

• Char ID: Character ID for which to fetch all the details.

Outputs:

✅ [On Success]

• Character Name: Name of your character.

• Voice Type: Voice name.

• Backstory: Character backstory.

• Has Ready Player Me Link: True if the avatar is configured on the website.

• Ready Player Me Link: The avatar link to be used to download.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Update Character

• Description: Update a particular character.

Inputs:

• Char ID: Character ID to be updated.

• New Voice: Voice name or [MALE/FEMALE].

• New Backstory: Updated backstory.

• New Char Name: Name of the character.

Outputs:

✅ [On Success]

Nothing is returned.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

Steps to resolve this error: -

1. Disable the MetaHuman Plugin in your Unreal Engine editor.

2. Initiate the packaging process for your project.

3. Enable the MetaHuman plugin.

Properties

Refer to this guide for configuring Microsoft Visual Studio or adjusting the toolsets as per the recommended guidelines by Unreal Engine.

Convai Object Entry structure allows for parsing of information about an object or character in the scene. It takes in the name of the object or character and can also include a reference to the object or character, a description, and a position vector (optional). This information is then added to the Environment object.

Properties

Convai Chatbot is an Actor component responsible for processing and getting a response for the voice audio coming from the Convai Player component. It plays the audio response and has a variety of useful events for transcription, actions, text response, and others.

Properties and Functions

Character Variables

Conversation States

Lip Sync

Conversation Management

Events

Event Names and Description.

Event Parameters

Text To Speech

• Description: Creates audio for the corresponding text.

Inputs:

• Transcript: The transcript for the audio.

• Voice: The voice name.

Outputs:

✅ [On Success]

• Wave: Audio response.

⛔ [On Failure]

Nothing is returned, check logs for details on why it failed.

When running the Convai plugin on a machine that is missing certain popular root certificates you will receive errors when trying to fetch characters' data and you will usually see this error in the chat widget Load Failed for Character ID: <Character ID>.

To fix the issue, you need to disable SSL verification from Unreal Engine, this can be done by going to Edit menu, select Project Settings and search for Verify Peer and disable it.

Another solution is to install any browser like Google Chrome or Fire Fox which usually install the required certificates automatically.

There are two ways to resolve this issue.

1. Convert your Blueprint project to a C++ project. Steps to convert: -

   • Navigate to the Tools menu, and choose 'New C++ class...' from the provided options.

   • Select any parent class from the menu that appears. For simplicity, we opt for 'None' in this case. Proceed by clicking on 'Next'.

   • Provide any name and proceed by selecting 'Create Class'.

   • Once done, close the Unreal Engine editor and then start it again by clicking on build project in Microsoft Visual Studio code.

2. Copy the ConvaiOVRLipSync plugin to the Plugins directory within the engine.

   • Refer to this to add CovnaiOVRLipSync.

Follow the steps outlined below to conduct a microphone test: -

• Press the F10 key on your keyboard after you play your project.

• A window will open up as shown here.

• From Input select your microphone and click on Record. This will start recording your audio.

• Record your audio for a few seconds and click on Stop.

• If your microphone is working properly you must hear your audio loud and crisp.

• If the audio sounds to be low then adjust the Audio Gain accordingly.

Editor Mic Permission Setup

1. Temporarily Disable SIP - Follow the instructions provided in the guide - Note: make sure to run the command csrutil disable in the terminal.

2. Clone the tccutil Repository - Open the Terminal, and enter the following command to clone the repository: git clone https://github.com/DocSystem/tccutil

3. Navigate to the tccutil Directory by entering cd tccutil into the terminal.

4. Ensure on your system. Run the tccutil command In the Terminal to allow microphone access for Unreal Engine: sudo python3 tccutil.py -e -id com.epicgames.UnrealEditor --microphone --enable

5. Re-enable SIP Once the modifications are complete, to re-enable System Integrity Protection but this time run the command csrutil enable.

Packaging the app with Convai (Not required for MacOS 14.x)

After packaging your game, if you notice any crash or microphone not working then proceed to doing the following steps

1. locate the Info.plist file in the packaged game directory. This is typically found by right-clicking the package, selecting 'Show Package Contents', and editing /Contents/Mac/Info.plist.

2. Add the following entries to request microphone access:

1. Save the changes to the Info.plist file.

Follow the steps below to set active microphone

• Open the player blueprint or the blueprint that has the Convai Player component in it.

• Drag the ConvaiPlayer component in your Event Graph and search for Set Capture Device by Name or Set Capture Device by Index. These functions are used to set the microphone by its index or by its name.

Step to list available microphone devices: -

• Open your player blueprint or the blueprint that has the Convai Player component in it.

• Drag the Convai Player component to the scene and search for Get Available Capture Device Names function.

Steps to change Audio Gain: -

• Open the Player blueprint or the blueprint which has the ConvaiPlayer component in it.

• From the ConvaiPlayer component search for Set Microphone Volume Multiplierfunction.

• Enter any float value in the In Volume Multiplier property of the function which makes the audio loud and clear.

1. Create a new first person project.

   
2. Enable the Convai plugin and add the API key as mentioned here.

3. Create a new Actor blueprint that we will be using as the AI character.

4. Open the created blueprint then search for and add the Convai Chatbot component in the components list. Note: if you do not find the component then ensure that you have properly installed and enabled the plugin by following the Installation guide.

   
5. Select the created component and on the details panel, find the Character ID field and paste your character ID which you can get by creating a new character or using an existing one on the Convai Playground.

   
6. Add a box component so that you can see the blueprint when placed in the scene.

   
7. Place the blueprint in the scene.

   
8. Open the player blueprint which by default in First Person/Blueprints/BP_FirstPersonCharacter for the first person template.

   
9. Search and add the Convai Player component in the components list.

   
10. Add the following blueprint schematic to allow the player to talk to the AI character via the V key:

    1. Add a keyboard key event to be used as a push to talk button (i.e. the `V` key in this example).

    2. Use Convai Get Looked At Character to get the chatbot component of the character that is currently viewed by the player.

    3. Set Radius to a reasonable distance or zero if you want the player to be talk the character over an infinite distance.

    4. Set Plane View to true to only consider the plane axis (X & Y) and ignore the height axis (Z), this is made to prevent having to look directly at the pivot of the character, instead only looking in the direction of the character is sufficent.

    5. Use the Start Talking node from the Convai Player component initiates the talking session with the character, ensure you have enabled Voice Response to get the character to respond vocally.

    6. On the Released event, use the Finish Talking node on the Convai Player component to let the AI character know that we have finished talking and are now waiting for a response.

    
11. Hit play, approach the AI character and push T to talk through the microphone, the character should then respond after releasing the T key.

    

Sending Text instead of Voice

Use the following Send Text node instead if you want to text chat with the character instead of voice.

Note, over here we use a hard coded string as input to the character, you will need to create the required UI to get the text input from the user and send it to the AI character.

Prerequisites

Installing Visual Studio for Windows

• Download Visual Studio from here.

• Ensure having the required C++ toolchains mentioned here. If you already have Visual Studio installed, you may open the installer and select 'Modify' to add the above mentioned toolchains.

Installing XCode for Mac

• Download XCode from the app store

• For UE 5.3 and UE 5.0, follow this guide to enable microphone permissions.

Installing the plugin

There are two methods to install the plugin depending on your requirements

1. Directly from the Marketplace Link: Recommended for easy installation and tracking updates to the plugin. (UE 5.1 - 5.3)

2. Building from Github Source: For source UE builds and for unsupported UE versions on the marketplace, but not guranteed as the marketplace approach. (UE 4.26 - 5.0)

Set up your project

1. From the top toolbar, go to Edit > Plugins.

2. Find the Convai plugin.

3. Click the checkbox to enable the plugin.

   
4. Restart Unreal Engine.

Set up the API key

1. Find your API key.

2. Go to Edit > Project Settings.

3. Choose Convai under the Plugins section on the left bar.

   
4. Paste the API key into the API Key field.

The goal of this guide is to easily show how to add a premade chat and settings widget to the UI by re-parenting the player blueprint, we will continue on the progress from the Simple Talking Cube guide.

1. (Optional) If you do not have a player blueprint already, then you can import a first person or third person content into your already existing project.

   1. From the Content Browser, click the Add button, then Add feature or content pack to the project then choose either First Person or Third Person then click Add to Project.

   2. Click the Window menu and make sure World Settings is enabled.

   3. On the World Settings tab, Find GameMode Override and set it to BP_FirstPersonGameMode for first person or BP_thirdPersonGameMode if you imported third person content.

2. In the content browser, navigate to your player blueprint, which is by default at FirstPerson/Blueprint/BP_FirstPersonCharacter for First Person or ThirdPerson/Blueprint/BP_ThirdPersonCharacter which is the default for Third Person.

3. Open the blueprint and click Class Settings then in the Details section Under Class Options change the parent class to ConvaiBasePlayer.

1. Hit save, compile and hit play to test - use the T key to talk and Enter to text chat.

   
2. Hit F10 to open the settings menu where you have various options, like testing your microphone and changing the chat widget layout.