# Welcome Start here to create, customize, test, and share interactive AI characters with Convai, covering Playground, no-code experiences, plugins and integrations, and API reference. ## **Welcome to the Official Convai Documentation** Your platform for building, customizing, and deploying intelligent, interactive AI characters across various environments and platforms. Whether you’re a **developer**, **designer**, or **creator**, this documentation will guide you through every step from your first login to building fully immersive experiences with Convai’s powerful tools and integrations.
*** ## What You’ll Find Here Our documentation is divided into several sections so you can easily find what you need: ### 1. Convai Playground Learn to create, customize, and test your AI characters directly in Convai Playground. * [**Get Started**](/api-docs/convai-playground/get-started) – Basics of navigating the dashboard, creating your first character, and testing interactions. * [**Character Customization**](/api-docs/convai-playground/character-customization) – Deep dive into the tools for defining your character’s appearance, voice, knowledge, traits, and more. ### 2. No Code Experiences Create interactive AI experiences without writing a single line of code. * [**Avatar Studio Experiences**](/api-docs/no-code-experiences/avatar-studio-experiences) – Create and customize your character’s visual identity, including appearance, clothing, accessories, environment, animations, lighting, camera angles, and more, all within an easy-to-use no-code editor. * [**Convai Sim Experiences**](/api-docs/no-code-experiences/convai-sim-experiences) – Build interactive, large-scale simulation environments where your AI characters engage in realistic scenarios, navigate spaces, and interact with objects. * [**Convai XR Animation Capture App**](/api-docs/no-code-experiences/convai-xr-animation-capture-app) – Capture high-fidelity motion data using your XR device and apply realistic animations to your avatars for more immersive, lifelike performances. ### 3. Plugins & Integrations Extend your characters into your applications and games. * [Unity](/api-docs/plugins-and-integrations/convai-unity-sdk), [Unreal Engine](/api-docs/plugins-and-integrations/convai-unreal-engine-plugin), and [Web Plugins](/api-docs/plugins-and-integrations/web-plugins). * [Modding Frameworks](/api-docs/plugins-and-integrations/modding-framework) and [Other Integration](/api-docs/plugins-and-integrations/other-integrations) options. * [Convai Pixel Streaming Embed](/api-docs/plugins-and-integrations/convai-pixel-streaming-embed) capabilities. ### 4. API Reference In-depth [API documentation](/api-docs/api-reference/core-api-reference) for advanced customization and integration. *** ## Before You Begin To start building with Convai, you only need: * A **Convai account** – [Sign up here](https://www.convai.com/) if you don’t have one. *** ## Getting Help * If you can’t find what you’re looking for, use the search bar at the top of the documentation to quickly locate relevant topics. * For inspiration, check out the **Sample Characters** available in your [Dashboard](/api-docs/convai-playground/get-started/dashboard-overview#main-dashboard-view). * If you need further assistance, visit the [**Convai Developer Forum**](https://forum.convai.com/) to connect with the community and get support from the Convai team. # Get Started A guided overview of the first steps in Convai Playground, including navigation, character creation, testing, and essential global controls. ## Introduction ## Welcome to Convai Playground! Your workspace for creating, customizing, and testing AI-powered characters. This page gives you a high-level overview of the core tools and workflows, helping you get productive quickly. Each section below links to a dedicated page where you can dive deeper. ## Prerequisites * A Convai account. ## Core Concepts * **Character** – An AI persona you create and customize with unique personality traits, language, knowledge, and behavior settings. * **Avatar Studio** – A no-code editor where you can design your character’s visual appearance and configure its **Avatar Studio Experience**, including environment, animations, interaction settings, and more. ## Quick Start Flow * **Dashboard Overview**\ Learn the layout, view your recent characters and experiences, and see where to create a new character or Convai Simulation Experience.\ Continue to: [*Dashboard Overview*](/api-docs/convai-playground/get-started/dashboard-overview) * **Creating a New Character**\ Start building your AI character by naming it, defining its description, choosing a language and voice, and setting its personality.\ Continue to: [*Creating a New Character*](/api-docs/convai-playground/get-started/creating-a-new-character) * **Testing a Character**\ Test your character in real time using the **Chatbox** for text and voice interactions, or via **Video Call** for a more immersive experience.\ Continue to: [*Testing a Character*](/api-docs/convai-playground/get-started/testing-a-character) * **Global Character Controls**\ Learn about tools available across all character pages, such as Versioning, Update, and Character Settings.\ Continue to: [*Global Character Controls*](/api-docs/convai-playground/get-started/global-character-controls) * **Character Versioning**\ Save and switch between different versions of your character for safe experimentation and iteration.\ Continue to: [*Character Versioning*](/api-docs/convai-playground/get-started/character-versioning) # Dashboard Overview Learn how to navigate and use the Convai Playground Dashboard to manage characters, experiences, and simulations efficiently. ## Introduction The Convai Dashboard is your central hub for managing AI-powered characters and immersive experiences. From here, you can create characters, launch experiences, and access sample characters built by the Convai team.
*** ## Navigating the Dashboard ### Main Dashboard View When you log in, you land on the **Dashboard** with a welcome header: *"Create, manage, and deploy AI characters for your spatial experiences."* The page is organized into three sections: * **Recent Characters** – Your most recently updated characters, shown as cards with the character name, a short description, and a scene video indicator. A **See All** link opens the full characters list. * **Recent Experiences** – Your most recently launched experiences, each tagged with a publish status badge: **Draft**, **Public**, or **Unlisted**. A **See All** link opens the full experiences list. * **Sample Characters** – Pre-built characters from the Convai team for quick testing and inspiration, covering use cases such as retail, healthcare, hospitality, and gaming.
*** ### Creating New Content In the **top-right corner**, you’ll find: Two primary action buttons sit at the top of the Dashboard content area: * **Create Character** (primary button) – Opens the character creation flow. * **Start New Experience** (secondary button) – Opens the experience setup flow.
*** ### Sidebar Navigation The left sidebar provides access to all core sections: | Item | Description | | -------------------- | ----------------------------------------- | | **Dashboard** | Returns to the main Dashboard view | | **My Characters** | Browse and manage all your characters | | **My Experiences** | Access, edit, and manage your experiences | | **Project Settings** | Configure project-level settings | At the bottom of the sidebar: * **Legacy Playground** – Access the previous Convai Playground interface. * **Beta Feedback** – Submit feedback directly to the Convai team.
*** ### Profile and Settings Click your **profile avatar** in the top-right corner to open the account dropdown: | Option | Description | | ------------- | ------------------------------------------ | | **Account** | Manage your account details | | **Pricing** | View your current plan and billing options | | **Theme** | Toggle between light and dark mode | | **Follow Us** | Links to Convai's social channels | | **Logout** | Sign out of your account | Your current plan (e.g., **Professional**) is displayed at the top of the dropdown alongside your name and email.
*** ### API Key Access Click the **shield icon** to the left of your profile avatar to open the **API Key** modal. The modal displays: * **Your API Key** – Hidden by default. Use the eye icon to reveal it or the copy icon to copy it to your clipboard. * **API Base URL** – `https://api.convai.com` * **View Docs** – A direct link to the [API integration documentation](https://docs.convai.com/).
{% hint style="info" %} **Keep your API key secure.** Never share it publicly or commit it to version control. Use environment variables for storage. {% endhint %} *** ## Conclusion The Convai Dashboard gives you fast access to everything needed to build and manage AI characters and interactive experiences. The sidebar handles navigation, the top content area handles creation, and the profile menu handles account and API settings. # Creating a New Character Learn how to create and customize a new AI character in Convai Playground, including description, avatar, voice, and language settings. ## Introduction The Convai Playground allows you to design AI-powered characters with unique personalities, voices, and visual appearances. This guide will walk you through creating a new character, from initial setup to customization of avatar, voice, and languages. ## Step-by-Step Guide ### 1. Access the Creation Tool * From your **Dashboard**, click **Create a new character** in the top-right corner.
* A new character creation interface will open. * In the left menu, only **Character Description**, **Avatar**, and **Language and Speech** are active initially. Other sections will unlock after the character is created.
*** ### 2. Character Description * **Character’s Name** – Enter the name for your character (you can edit this later). * **Core Description** – Write a short background covering the character’s story, personality details, and distinctive features. * Alternatively, click the **Wand Icon** to **Generate Character Backstory with AI**. * **Speaking Style** *(optional)* – Define the way your character speaks, including tone and mannerisms. Also, feel free to suggest some **Sample Dialogues** here for your AI character. * **Visibility Settings** – Choose who can interact with your character: * **Public** – Available to anyone on `x.convai.com`. * **Unlisted** *(default)* – Only accessible via a direct link. * **Private** – Restricted to you and invited users.
*** ### 3. Avatar Customization * Click the **Avatar** tab in the left menu. * Select **Configure Avatar** to customize your character’s visual appearance. * Follow the steps in the [**Avatar Studio Documentation**](/api-docs/no-code-experiences/avatar-studio-experiences) to learn how to adjust facial features, clothing, and other design elements. * Click **Save Changes** when done.
*** ### 4. Voice and Language Settings * Click the **Voice And Languages** tab. * In **Language**, select one or more languages your character can speak. * In **Voice**, choose from the filtered voice options for your selected languages
*** ### 5. Finalizing Character Creation * Once all desired settings are configured, click **Create Character** at the top right. * If you skip customization, a **random avatar** and **voice** will be assigned automatically. * If you don't choose any language, **English** will be selected by default. * You can update any attribute later, so there’s no need to finalize all decisions immediately. *** ## After Creation When the character is created, all the sections in the left menu become available for deeper customization: * Character Description (with **Speaking Style** and **Personality Traits**) * Avatar * Language and Speech * Knowledge Bank * Core AI Settings * Guardrails * State of Mind * Embodied Actions (*Coming Soon to the new Playground. Available on* [*Legacy Playground*](https://playground.convai.com/pipeline/dashboard)) * Narrative Design (*Coming Soon to the new Playground. Available on* [*Legacy Playground*](https://playground.convai.com/pipeline/dashboard)) * External API (*Coming Soon to the new Playground. Available on* [*Legacy Playground*](https://playground.convai.com/pipeline/dashboard)) * Publish * Memory * Mindview Each of these features allows you to enhance and refine your character for more natural, intelligent, and engaging interactions. These are covered in separate documentation.
*** ## Conclusion The character creation process in Convai Playground is designed for flexibility — you can launch a character in minutes or spend time refining every detail. Whether you start with default settings or fully customize the avatar, voice, and description. # Testing a Character Learn the different ways to test your AI character in Convai Playground, including text chat, voice input, and video call with an avatar. ## Introduction Once you have created a character, it's important to test how it interacts. Convai provides multiple ways to test your characters — from quick text and voice interactions to fully immersive video calls with custom avatars. This ensures you can refine personality, responsiveness, and interaction style before final deployment. *** ## Testing Options ### Text and Voice Chat To test your character via text or voice, open the character's chat interface: * From your **Dashboard**, click on the character card you want to test. * The chat interface opens as a full conversation view, with your messages on the right (green) and the character's responses on the left (white). * Type a message in the **"Type a message..."** input at the bottom and press enter to send. * Or click the **microphone button** on the right side of the input bar, speak, and send your voice input. Each message shows a timestamp. A small **eye icon** appears below each message — click it to open **Mindview** for that specific exchange.
*** #### Mindview for Playground Chat **Mindview** lets you inspect how the character processed a specific message. Click the **eye icon** below any message to open it. Mindview displays: * **User Query** and **Response** at the top — the exact exchange being inspected. * **Model and Character ID** — the LLM and character version used for that response. * **Structured / Raw toggle** — switch between a readable structured view and the raw prompt output. * **Static Prompt** — the fixed parts of the character's prompt, broken into: * **Core Description** – the character's base personality and instructions. * **Languages** – the languages the character is configured to respond in. * **Output Format** – the response style instruction given to the model. * **Dynamic Prompt** — the context injected at runtime for that specific exchange, including **Context Setting** and other dynamic elements. Mindview is useful for debugging unexpected responses, verifying that your character description and knowledge bank are being applied correctly, and understanding exactly what the model receives for any given turn. Read the detailed [Mindview Docs](https://docs.convai.com/api-docs/convai-playground/character-customization/mindview) to learn more.
*** #### Chat Interface Controls **Left sidebar icons (top-left of the chat):** | Icon | Function | | ---------------- | ------------------------------------------------------ | | **Audio toggle** | Mute or unmute the character's voice output | | **Copy** | Copy the conversation text | | **Reset** | Restart the session and clear the conversation history | **Top-right icons:** | Icon | Function | | ---------- | ------------------------------------- | | **Camera** | Start a video call with the character | | **Share** | Share the character | *** #### Quick Replies After each character response, a **Quick replies** section appears below the message with contextually generated reply suggestions. Click any chip to send that reply instantly without typing. Quick replies are dynamically generated based on the ongoing conversation, so they update with each new character response. *** ### **2. Video Call** You can test your character in a **video call** for both visual and voice interaction. **From the Dashboard:** * Locate the character's card. * Click the **green camera icon** on the card to start a video call instantly.
**From the Character Page:** * Click the character’s card to open its details. * In the top-right section, under the character’s thumbnail, click the **video call button**. The video call gives you a real-time view of the character's visual appearance and voice together, offering a more immersive test environment.
This method allows you to experience both the character’s **visual appearance** and **voice** in real-time, offering a more immersive test environment.
*** ## Conclusion Convai's testing options make it easy to evaluate and refine your characters. Whether you prefer a fast text-based interaction with quick reply suggestions, a voice conversation, or a full video call with your custom avatar, you can ensure your AI character behaves as intended before deployment. Use Mindview on any message to inspect the underlying prompt and debug character behavior at any point in the conversation. # Global Character Controls A single reference for the shared toolbar controls available across all character pages in Convai Playground, including Versioning, Update, and Character Settings. ## Introduction This page explains the shared controls that appear at the top right of every character page in Convai Playground. You will see the same toolbar on Character Description, Avatar, Language and Speech, Knowledge Bank, Personality Traits, Core AI Settings, State of Mind, Embodied Actions, Narrative Design, External API, Publish, and Memory. Understanding these controls helps you work faster and avoid losing changes. *** ## Where to find these controls Look at the top right of any character screen. You will see: * **Versioning** icon * **Update** button * **Character Settings** menu (three dots) These controls behave the same way on every page.
*** ## Controls overview ### 1. Character Versioning Use Versioning to save and switch between alternative definitions of your character. * **What it does** * Saves a named snapshot of your character definition so you can test new ideas without losing a preferred setup. * Lets you switch to any saved version and continue editing from there. * **Typical uses** * Keep a stable production version while experimenting with a new Core Description or personality. * Prepare variations for different audiences or channels. * **Good practice** * Give versions clear names and short notes such as “v1.2 retail tone” or “v2.0 multi language test”. * Save a version before major edits or before handing the character to a teammate. For more information, refer to the [Character Versioning](/api-docs/convai-playground/get-started/character-versioning) documentation.
### 2. Update button Apply your unsaved changes to the character. * **States** * **Green**: there are unsaved edits. Click **Update** to save.
* **Gray**: everything is already saved.
* **Important** * If you refresh or navigate away while the button is green, your unsaved changes will be lost. * Click **Update** after edits on any tab, then proceed to testing. ### 3. Character Settings Menu Open the three dots menu to access actions that affect the entire character.
#### **Clone Character** Create a duplicate so you can branch work safely. * **What is copied**: all character configuration tabs (e.g., Description, Personality, Languages, etc.) are copied, except the Memory tab. * **What changes**: the clone receives a new Character ID. * **When to use**: large experiments, staging vs production split, A or B variants. #### **Share Character** Let others **test** your character. * **What it does:** Generates a share link so recipients can interact with the character (e.g., Chatbox or video call) without being able to modify it. * **How it works:** Open the dialog to copy a share link, respecting your current visibility setting (Public, Unlisted, or Private). #### **Delete Character** Remove the character from your characters. {% hint style="warning" %} Deletion is permanent and cannot be undone. {% endhint %} * **Checklist before deletion** * Confirm the character is not used in any live experience. * Export or copy any content you may need. * Consider cloning for archival instead of deleting. # Character Versioning Manage multiple versions of character and switch between them as required. ## Introduction In this section, we look into Character Versioning, i.e., maintaining different states of the character. This enables the user to preserve a previous stable state before trying out more changes. You can now experiment without the fear of losing an older state of the character, and in case you want discard the current changes and return to a previous version, you now have the ability to restore the version and continue working from there. We conveniently call these saved states as **Snapshots** of the character. We will go over the features and how to use them in this section. {% hint style="info" %} The idea of **Snapshot** and **Version** has been used interchangeably in the text; however, they refer to the same idea: The state / contents that define the character at a specific point in time. {% endhint %} *** ## Overview The character versioning option is available at the top right-hand side in the character editor section beside the **Update** button
Once you click on it, you get to see the list of all your previous saved revisions ordered by date.

Character Versioning section. There is no snapshot here yet.

We will go over the steps of creating and maintaining snapshots from scratch in the next section *** ## Create a Version Let us start with a character that we already have saved. The data that we see when we open the details related to a character denotes the **Current Snapshot** of the character. When you interact with the character, you are essentially referring to all the date in this **Current Snapshot** of the character. 1. To create a new version, first open the **Character Versioning** section and click on the **+ Add Snapshot** sign at the top.

Let's create our very first snapshot.

2. A pop-up appears asking you to give your snapshot a name and some description. Please note that a **Snapshot Name** is a required field to create a new version. Once you have filled the details, click on the **Submit** button.

We provide a name and a small description.

3. Now, you can see the new version in the list of snapshots. Now what does this version actually represent?\ This snapshot stores all the data related to the character at that point of time. Everything about the character ranging from character description, embodiment to knowledge bank files, narrative-design structure and other details.

The snapshot appears in the list

*** ## Restoring a Version Assuming you have gone ahead and worked on the character further, but you are unhappy with the results and want to go back and start from the previous version. This is where you have the ability to restore an old snapshot to the current state and work with them again. Here are the steps to follow: 1. To restore a version, open the **Character Versioning** section and select the snapshot you want to restore back. You will see the **Restore Version** button below come to life.

We will be restoring the data from the very first snapshot.

2. Once you click on the **Restore Version** button, a pop-up appears asking you if you want to save the current changes as a new snapshot or discard them. You have the option to store your current changes as some test version and refer back later on.

Let's directly restore the data in the snapshot to the Current Snapshot

3. For now, we are happy to discard the changes, so we will click on **Restore** button. This brings the data from the selected version to the **Current Snapshot** of the character. 4. To save the changes, you can always **Cancel** and go back to creating a new snapshot with your progress and the restoring it. *** ## Delete a Snapshot You can also go ahead and delete a snapshot that you no longer require. To that you can click on the 3-dots by the corresponding snapshot in the list of Character Version and select **Delete Version**

Click on the 3-dots beside the snapshot to see all the options.

### Some important points to remember At any given point you can interact with the **Current Snapshot** of the character. If you have any publicly available app that utilises the character, your users will only be able to interact with this current version. {% hint style="info" %} We are currently working on a feature to help developers have separate deployed version than the **Current Snapshot**. {% endhint %} # Interact in Voice Mode Enable real-time, low-latency voice conversations with your Convai character using Voice Mode for natural, hands-free interactions. ## Introduction **Voice Mode** allows you to have seamless, natural, and low-friction voice conversations with your character. This guide explains how to set up Voice Mode and maintain stable, real-time sessions for a smooth conversational experience. ## Step-by-Step Guide ### 1. Open Your Character * From your **Dashboard**, open the character you want to use with Voice Mode.
### 2. Configure Voice Settings * Navigate to **Language and Speech** and select a voice for your character. * Once configured, click **Update** to save your changes.
### 3. Using Voice Mode Click the microphone button to enter **Voice Mode** and start talking to your character hands-free in real time.
* When you exit Voice Mode, the **conversation transcript** will appear in the chat area for review.
{% hint style="warning" %} If you remain idle for more than **5 minutes**, the voice session automatically disconnects. Simply reconnect to resume your conversation. {% endhint %} # Character Customization Learn how to refine your AI character’s personality, appearance, knowledge, and behavior to create consistent and engaging interactions. ## Introduction The **Character Customization** section in Convai Playground is where you transform a basic AI character into a fully realized persona. Here, you’ll define how your character looks, speaks, thinks, remembers, and responds, ensuring a unique and immersive experience for your users. Each page in this section focuses on a specific customization area, allowing you to work step-by-step or revisit any aspect at any time. *** ## Core Customization Areas 1. **Character Description**\ Define your character’s backstory, personality, and speaking style — the foundation of its identity.\ Continue to: [*Character Description*](/api-docs/convai-playground/character-customization/character-description) 2. **Avatar Section**\ Open **Avatar Studio** to design your character’s visual appearance and configure its **Avatar Studio Experience**, including environment, animations, outfits, lighting, camera angles, and more.\ Continue to: [*Avatar Section*](/api-docs/convai-playground/character-customization/avatar-section) 3. **Language And Speech**\ Configure the language, voice, and tone your character uses for communication.\ Continue to: [*Language And Speech*](/api-docs/convai-playground/character-customization/language-and-speech) 4. **Knowledge Bank**\ Provide your character with information and reference materials to answer questions and maintain context.\ Continue to: [*Knowledge Bank*](/api-docs/convai-playground/character-customization/knowledge-bank) 5. **Personality Traits**\ Adjust behavioral sliders to shape your character’s mannerisms, confidence, empathy, and other interaction styles.\ Continue to: [*Personality Traits*](/api-docs/convai-playground/character-customization/knowledge-bank) 6. **Core AI Settings**\ Fine-tune advanced AI parameters to influence decision-making, creativity, and responsiveness.\ Continue to: [*Core AI Settings*](/api-docs/convai-playground/character-customization/core-ai-settings) 7. **Guardrails**\ Restrict and guide a character's AI behavior, define the ethical and topical boundaries of the conversation.\ Continue to: [*Guardrails*](/api-docs/convai-playground/character-customization/guardrails) 8. **State of Mind**\ Define temporary or situational mindsets that influence how your character reacts in specific contexts.\ Continue to: [*State of Mind*](/api-docs/convai-playground/character-customization/state-of-mind) 9. **Memory**\ Review your character’s past conversations or enable Long Term Memory to allow recall across sessions.\ Continue to: [*Memory*](/api-docs/convai-playground/character-customization/memory) 10. **Narrative Design**\ Create structured narratives or guided interaction flows for your character to follow.\ Continue to: [*Narrative Design*](/api-docs/convai-playground/character-customization/narrative-design) 11. **External API**\ Connect your character to external systems or APIs to retrieve live data or perform actions.\ Continue to: [*External API*](/api-docs/convai-playground/character-customization/external-api) 12. **Publish**\ Share your character with others or embed it into your website.\ Continue to: [*Publish*](/api-docs/convai-playground/character-customization/publish) *** ## Best Practices * Start with core identity settings (Character Description, Language and Speech) before moving to advanced customization. * Use **Update** frequently to save progress and avoid losing changes. * Test your character regularly to ensure each customization change has the desired effect. *** ## Next Step Begin with **Character Description** to establish the personality and tone of your AI character before moving on to appearance, speech, and advanced behaviors. # Character Description Learn how to use the Character Description page in Convai Playground to define your character’s identity, speaking style, and unique traits. ## Introduction The **Character Description** page is where you define the personality, backstory, and communication style of your AI character. Each character in Convai Playground has its own dedicated Character Description page, ensuring a unique identity that can be refined over time.
*** ## Accessing the Character Description Page You can reach the Character Description page by clicking any **Character Card** on your **Dashboard**. This opens the character’s profile, where you can edit and manage its core attributes. ## Main Features and Sections ### 1. Character Name and ID
* **Character’s Name** – Editable field for your character’s display name. * **Character’s ID** – A unique identifier for the character, essential for using it in **Convai SDKs** and API integrations. * You can copy the ID to use in your applications. {% hint style="success" %} **Support Tip**: If you need help from the support team, provide this ID when reporting character-related issues. {% endhint %} *** ### 2. Core Description / Speaking Style / Embodiment **Core Description** * Add details about your character’s story, personality traits, distinctive features, and any behavioral guidelines. * Word limit: **1000 words**.
**Speaking Style** * **Describe How the Character Speaks** – Outline the character’s tone, pace, formality, and speech patterns. Include unique expressions or phrases they commonly use. * **Sample Dialogues** – Provide example sentences showcasing the character’s typical speech style, including signature phrases that reinforce their personality.
**Embodiment** * Currently in development and will be available soon. *** ## Examples For inspiration, explore **Sample Characters** in the Dashboard. These examples show how different characters’ Core Descriptions and Speaking Styles are structured.
*** ## Conclusion The Character Description page is the foundation of your AI character’s identity. By clearly defining its personality, voice, and unique traits, you ensure consistent and engaging interactions. # Avatar Studio Learn how to access and customize your character’s avatar in Convai Playground using Avatar Studio. ## Introduction The **Avatar** section, lets you design and customize both the visual appearance of your AI character and its dedicated Avatar Studio Experience. All customization is handled through **Avatar Studio**—a powerful no-code tool where you can adjust the character’s look, clothing, and animations, as well as personalize the interactive environment it appears in.
*** ### Accessing Avatar Studio from Convai Playground * Open any character in Convai Playground. * In the left-hand menu, click the **Avatar** section. * This will launch **Avatar Studio**, where you can customize your character’s appearance and configure its **Avatar Studio Experience**, including the environment, animations, outfits, lighting, camera angles, and more. *** ## Next Step: Customize in Avatar Studio For a complete guide to using Avatar Studio and its features, see our dedicated documentation: [Read the Avatar Studio Documentation](https://docs.convai.com/api-docs/no-code-experiences/avatar-studio-experiences) *** ## Conclusion The Avatar section serves as a quick link to Avatar Studio, where you can customize both your character’s appearance and its Avatar Studio Experience. Whether you’re creating realistic personas or stylized avatars, Avatar Studio provides tools to fine-tune the environment, animations, outfits, lighting, camera angles, and more—ensuring your character’s visual presence matches its personality and role. # Language And Speech Learn how to configure languages, voices, custom pronunciations, and word recognition for your AI character in Convai Playground. ## Introduction The **Language and Speech** section allows you to define the spoken languages, select a voice, and improve pronunciation and recognition for your AI character. With support for multiple languages and voice providers, you can ensure that your character communicates naturally and effectively with your audience.
*** ## Main Features ### 1. Set Language * Choose the languages your character can speak and understand. * Supports **multilingual** select between **1 and 4 languages**. * Default language: **English**. * Over **65+ languages** are available. * Selecting a language will filter the available voices in the **Voice** section.
*** ### 2. Voice Selection * The **Voice** field provides access to over **1,200** voices in total. When you select a language, the available voices are filtered accordingly, so the number of voices varies by language. * Supported **voice providers**: * Google Cloud Platform (GCP) * Microsoft Azure * OpenAI * ElevenLabs * **Custom Voices** can be added through ElevenLabs. * See: [**ElevenLabs Voice Integration Documentation**](https://docs.convai.com/api-docs/plugins-and-integrations/other-integrations/third-party-api-integrations/elevenlabs-api-integration) for setup details.
*** ### 3. Add Custom Pronunciation Custom pronunciations help your character pronounce specific words correctly, especially unusual or brand-specific terms. * To add: * **Spelled As** – The word as it appears in text. * **Pronounced As** – How it should sound, written phonetically in plain English. * Example: * **Spelled As:** convai * **Pronounced As:** convey * **Case-sensitive:** Uppercase and lowercase entries can have different pronunciations. {% hint style="danger" %} Currently **only supports English**. {% endhint %}
*** ### 4. New Word Recognition New Word Recognition improves your character’s ability to understand unique or challenging words in speech input. * To add: * **Spelled As** – The correct spelling of the word. * **Pronounced As** – The phonetic pronunciation using simple syllables. * Example: * **Spelled As:** Ankur * **Pronounced As:** Ahnkur {% hint style="danger" %} Currently **only supports English**. {% endhint %}
*** ## Conclusion The Language and Speech settings provide complete control over how your character communicates, from language selection and voice choice to fine-tuning pronunciation and recognition. These tools help ensure your AI character delivers clear, accurate, and engaging interactions for users. # Knowledge Bank Learn how to upload, manage, and connect knowledge files to your AI character using Knowledge Bank. ## Introduction The **Knowledge Bank** is where you store and manage information that your AI character can access during conversations. By uploading documents or adding text directly, you can give your character specific domain knowledge, enabling more accurate, relevant, and context-aware responses. All files uploaded to your Knowledge Bank are linked to your Convai account and can be connected to any of your characters. This makes it an essential tool for training characters to respond with company-specific, product-specific, or topic-specific information.
*** ## Knowledge Bank Sections ### 1. My Documents * Displays all files uploaded to your account. * Information shown: * **Name** – File name. * **Size** – File size. * **Status** – Indicates if the file is available. * ```
``` * Actions available: * **Connect** – Attach the file to a character. * **Disconnect** – Remove the file from a character. * **Edit** – Modify the file content. * **Download** – Save the file locally. * **Delete** – Remove the file permanently.
*** ### 2. Upload Knowledge * Upload `.txt` files from your computer. * Currently, **only `.txt` file format** is supported. * Once uploaded, files are stored in your account’s Knowledge Bank for use with any character.
*** ### 3. Add Knowledge * Create a new file by entering **plain text** directly into the editor. * Name the file and save it in `.txt` format.
{% hint style="success" %} Frequently restart the page during the learning phase to check if the file status is “Available.” {% endhint %} *** ## Using the Knowledge Bank with Your Character ### Example We uploaded a file named **Employee Onboarding Guide.txt** with the following content: ``` This document provides step-by-step guidance for new hires. Complete HR documentation within the first 3 days of joining. Attend the mandatory orientation session. Set up company email and access credentials via IT Support. Review the Code of Conduct and Data Privacy Policy. ```
### Testing Without Connecting the File * Open the **Chatbox**. * Ask: *“I’m a new hire. What should I do during my first week here?”* * Result: The character responds using its general personality and AI model knowledge, not the uploaded file.
### Testing by Connecting the File * Go to **Knowledge Bank** → **My Documents**. * Click **Connect** on the file. * In the Chatbox, click **Reset Chat** (top left) to start a new session. * Ask the same question again. **Result:** This time, the character’s response is based on the exact steps provided in the **Employee Onboarding Guide** file.
*** {% hint style="info" %} Always **reset the chat session** after connecting a new knowledge file so the latest data is used. {% endhint %} {% hint style="info" %} The **total storage size** for uploaded files depends on your Convai subscription plan. See the [**Pricing**](https://convai.com/pricing) page for limits. {% endhint %} *** ## Conclusion The Knowledge Bank is a powerful way to give your characters precise and reliable information. By connecting domain-specific documents, you ensure that your AI not only has personality but also the expertise to answer questions with accuracy and authority. # Personality Traits Learn how to customize your AI character’s personality using presets or manual trait adjustments. ## Introduction The **Personality Traits** section defines how your AI character behaves, interacts, and responds during conversations. By adjusting personality parameters, you can align the character’s behavior with its intended role, making interactions more engaging and consistent.
*** ## Preset Personality Styles At the top of the page, you’ll find a **dropdown menu** containing predefined personality presets: * Adventurous Thinker * Friendly Optimist * Harmonious Empath * Analytical Perfectionist * Curious Mediator * Energetic Dreamer * Social Adventurer * Compassionate Idealist Selecting a preset automatically adjusts the character’s personality traits to match the chosen style.
*** ## Customizing Personality Traits If you prefer full control, you can manually adjust the **vertical sliders** for each personality dimension: 1. **Openness** * High value: Likes exploring and trying new things. * Low value: Prefers stability and routine. 2. **Meticulousness** * High value: Pays great attention to detail. * Low value: More relaxed and spontaneous. 3. **Extraversion** * High value: Outgoing and sociable. * Low value: Reserved and introverted. 4. **Agreeableness** * High value: Cooperative and empathetic. * Low value: More competitive and independent. 5. **Sensitivity** * High value: Highly emotional and expressive. * Low value: Rarely emotional or reserved. Each slider ranges from **0 to 4**, allowing precise adjustments to match your character’s personality profile.
*** ## Visual Personality Map Below the sliders, a **radar chart** displays a visual representation of the character’s personality. This helps you see how each trait contributes to the overall personality balance.
*** ## Conclusion The Personality Traits section gives you the flexibility to either choose from predefined styles or fine-tune individual traits to create a personality that matches your vision. By combining these settings with your character’s description and voice, you can create truly distinctive AI personas. # Core AI Settings Learn how to configure moderation, foundation model selection, and temperature for your AI character ## Introduction The **Core AI Settings** section defines the foundational behavior of your AI character by controlling safety filters, the underlying language model, and the creativity level of its responses. These settings have a significant impact on how your character interacts with users, balancing safety, accuracy, and creativity.
*** ## Main Features ### 1. Enable Moderation Filter * This setting allows you to filter out potentially harmful content, including hate speech, profanity, or inappropriate language. You can turn the moderation filter on or off using the toggle located at the top of the page. By default, this setting is enabled.
{% hint style="warning" %} Disabling the Moderation Filter makes some foundation models unavailable. {% endhint %} {% hint style="warning" %} Features like **Narrative Design** and **Multilingual support** will not work when moderation is disabled. {% endhint %} *** ### 2. Select Foundation Model Choose from a variety of **Large Language Models (LLMs)** from leading providers: * OpenAI * Anthropic * Google * Llama
{% hint style="info" %} Model availability depends on whether the Moderation Filter is enabled. {% endhint %} *** ### Supported LLMs Below is a list of Large Language Models (LLMs) available in the Convai Playground under **Core AI Settings**.\ Models marked as ✅ *Flagship* are the providers’ top-tier, most capable models — but usage of these is subject to the **Flagship Interaction Cap** based on your plan. > **Flagship LLMs**\ > This is the limit on the number of interactions you can perform using Flagship LLMs. > > **Example:**\ > In the *Indie Dev* plan, you have a total monthly quota of **3000 Interactions**. However, the **Flagship LLM Interaction Cap** is **1500**.\ > If you use GPT-4.1 after 1500 interactions, your Flagship LLM quota will be exhausted.\ > You will then need to switch to a non-Flagship LLM for the remaining 1500 interactions. *** ### Realtime / Live Models #### OpenAI
ModelModel CodeFlagship
GPT Realtime 1.5 (beta)gpt-realtime-1.5false
GPT Realtime Mini (beta)gpt-realtime-minifalse
#### Google
ModelModel CodeFlagship
Gemini 2.5 Flash Live (beta)gemini-2.5-flash-livefalse
Gemma 4 31B Fast (beta)realtime-gemma-4-31b-itfalse
Gemma 4 26B A4B Fast (beta)realtime-gemma-4-26b-a4b-itfalse
### Standard Models #### OpenAI
ModelModel CodeFlagship
GPT-5.4gpt-5.4false
GPT-5.x (latest)gpt-5.xtrue
GPT-OSS-120B (beta)gpt-oss-120bfalse
GPT-5.1 (beta)gpt-5.1false
GPT-4.1gpt-4.1false
GPT-5.4-nanogpt-5.4-nanofalse
GPT-5.x-nano (latest)gpt-5.x-nanotrue
GPT-5.4-minigpt-5.4-minifalse
GPT-5.x-mini (latest)gpt-5.x-minitrue
GPT-4.1-minigpt-4.1-minifalse
GPT-5.3 Instantgpt-5.3-instantfalse
GPT-4ogpt-4ofalse
GPT-4.1-nanogpt-4.1-nanofalse
GPT-4o-minigpt-4o-minifalse
#### Anthropic
ModelModel CodeFlagship
Claude 4.5 Sonnet (beta)claude-4-5-sonnetfalse
Claude 4.5 Haiku (beta)claude-4-5-haikufalse
Claude Sonnet (latest)claude-sonnettrue
Claude Haiku (latest)claude-haikutrue
#### Google
ModelModel CodeFlagship
Gemini 3.5 Flashgemini-3.5-flashfalse
Gemini Flash (latest)gemini-flashtrue
Gemini 3.1 Flash Litegemini-3.1-flash-litefalse
Gemini Flash Lite (latest)gemini-flash-litetrue
Gemini 2.5 Flashgemini-2.5-flashfalse
Gemini 2.5 Flash Litegemini-2.5-flash-litefalse
#### Qwen
ModelModel CodeFlagship
Qwen3.6 27B (beta)qwen3.6-27bfalse
Qwen3.6 35B A3B (beta)qwen3.6-35b-a3bfalse
#### Llama
ModelModel CodeFlagship
Llama 4 Maverick (beta)llama-4-maverickfalse
Llama 4 Scout (beta)llama-4-scoutfalse
Llama3 70Bllama3-70bfalse
#### xAI
ModelModel CodeFlagship
Grok 4.3grok-4.3false
*** ### 3. Temperature Control
* **Function:** Adjusts the randomness and creativity in the AI’s responses. * **Slider Range:** `0.0` (most deterministic) to `1.0` (most creative). | Temperature Range | Behavior | Use Case | | -------------------- | ------------------------------------------ | ---------------------------------------------- | | **Low (0.0–0.3)** | Deterministic, consistent | Factual Q\&A, compliance-critical interactions | | **Medium (0.4–0.7)** | Balanced accuracy and creativity | Conversational agents, customer support | | **High (0.8–1.0)** | Diverse, creative, sometimes unpredictable | Storytelling, brainstorming, roleplay | {% hint style="info" %} Lower temperature sharpens the probability distribution for more predictable word choices. Higher temperature flattens the distribution, allowing less likely words to appear more frequently. {% endhint %} *** ## Conclusion The Core AI Settings give you precise control over your character’s foundation model, safety filters, and response style. By adjusting these parameters, you can create an AI that balances safety, reliability, and creativity to suit your specific application. # Guardrails The Guardrails tab is a safety and control interface designed to restrict and guide a character's AI behavior. It allows creators to define the ethical and topical boundaries of the conversation. {% hint style="danger" %} This API is available only on the Paid Plans. {% endhint %} ## Introduction The **Guardrails tab** provides a robust control interface to define your character’s ethical and topical boundaries, preventing off-topic or inappropriate responses. By entering specific constraints in the Instructions field and toggling custom guardrails on, you ensure your AI character responds in a safe and professional manner, and is strictly aligned with your intended brand voice.
## Main Features and Sections ### 1. Custom Guardrails Toggle This toggle acts as the master switch to activate or deactivate your character's safety oversight. When enabled, it forces the AI character to prioritize your specific behavioral rules over its general conversational logic. ### 2. Guardrails Instruction This text field allows you to define explicit safety rules and response boundaries using up to 7,500 words. It is where you input constraints like "Do not talk about politics" to ensure the character remains professional and on-topic. ## Examples This example shows how you can implement guardrails for your AI character in Convai Playground {% embed url="" %} *** ## Conclusion The Guardrails tab is an essential safety tool that empowers creators to enforce strict behavioral boundaries and content restrictions on their AI characters. By combining custom instructions with real-time testing, it ensures that interactions remain secure, on-brand, and free from undesirable topics or hallucinations. # State Of Mind Learn how the State of Mind feature visualizes your AI character’s emotional state in real time during conversations. ## Introduction The **State of Mind** section provides a visual representation of your AI character’s current emotional state. This dynamic emotional map helps you understand how your character is responding internally during a conversation, allowing for fine-tuning of its personality and interaction style.
*** ## How It Works * The State of Mind interface displays a **color-coded emotion wheel**. * Each segment represents a specific emotion such as **Joy, Anger, Trust, Fear, Surprise, Sadness, Disgust,** and **Anticipation**, along with nuanced variations like **Serenity, Rage, Admiration,** and **Amazement**. * **Active emotions** — those the character is currently experiencing — are highlighted more brightly on the graph. * Emotions change dynamically based on the context, tone, and content of the conversation.
*** ## Practical Use Cases * **Character Testing:** Observe real-time emotional responses to verify that the character reacts as intended. * **Personality Tuning:** Adjust personality traits in the **Personality Traits** section and see how they influence emotional patterns. * **Storytelling & Roleplay:** Ensure emotional consistency in interactive narratives. *** ## Conclusion The State of Mind feature offers valuable insights into your AI character’s emotional behavior. By monitoring these live emotional changes, you can ensure your character responds in a way that aligns with its designed personality and intended use case. # Memory Learn how to use the Memory feature to review past sessions, manage conversation history, and enable long-term memory for your character. ## Introduction The Memory section lets you review conversation history for a character and decide whether it should remember information between sessions. Use it to audit interactions, troubleshoot issues, and enable persistent preferences. ## Recent Memory This tab lists all previous sessions with your character.\ For each session, you can view: * **Date** – The date when the session occurred. * **Time** – The session’s start time, shown in UTC time zone. * **Session ID** – A unique identifier for that session. {% hint style="info" %} If you experience any issues in a session, support team may request the Session ID so they can investigate in detail. {% endhint %} **Available Actions:** * **View conversation:** Click the downward arrow to expand and see the conversation log for that session.
* **Copy or download:** Use the three-dot menu on the right to copy or download the session data.
*** ## Memory Settings This tab allows you to enable or disable **Long Term Memory**. When Long Term Memory is **enabled**, your character can remember preferences, choices, and facts from previous sessions. For example:\ If you tell your character “My favorite color is blue” in one session, and later in a different session ask “What’s my favorite color?”, the character will respond with “blue.” When disabled, the character will not retain information between sessions.
*** ## Conclusion The Memory feature provides powerful control over how your character interacts with you over time. Use Recent Memory to inspect and share specific sessions, and adjust Memory Settings to decide whether your character should retain knowledge across conversations. # Mindview Learn how to use the Mindview feature to review the actual prompts to the LLM for your current or previous sessions and interactions. {% hint style="danger" %} This feature is available only on the Professional Plan and above. {% endhint %} ## **Introduction** The **Mindview** section provides visibility into the **prompt** that was sent to the model to generate your character’s response.\ It’s a powerful tool for: * Understanding how your character processes context. * Improving your **Character Description**, **Knowledge Bank**, and **Language Settings**. * Troubleshooting unexpected or inconsistent responses.
*** ## **Accessing Mindview** You can open the **Mindview** tab directly from the left navigation menu of the Convai Playground.
When first opening it, you’ll be asked to select a **conversation or interaction** from the **Memory** tab.\ Alternatively, you can start a new conversation — Mindview will automatically display the data for the **latest message**. To access Mindview for a **previous interaction**: 1. Navigate to the **Memory** tab. 2. Expand the desired **session**. 3. Click the **Mindview icon** next to any message to open its corresponding prompt view.
*** ## Understanding the Mindview Interface Once opened, you’ll see a structured view of how the model interpreted and responded to an input.
### **Header Information** At the top of the screen, the following details are displayed: * **Session ID** – Identifies which session the interaction belongs to. * **Model Name** – Shows the LLM used to generate the response. * **User Query** – Displays the exact message or query that initiated this prompt. ### **Main Prompt Section** This is the core of Mindview. It shows the **entire chain of messages** (System, Assistant, and User) that formed the complete prompt sent to the model. Each section provides insight into how the model understands the character’s context and instructions before producing a response. *** ## What Influences the Main Prompt The main prompt displayed in Mindview is dynamically constructed using multiple aspects of your character and session: | Source | Description | | ------------------------- | ----------------------------------------------------------------------------------------------------------- | | **Character Description** | Defines the character’s backstory and core context. Appears within `` ... `` tags. | | **Language and Speech** | Includes the allowed languages and relevant speech configuration. | | **Personality Traits** | Controls the conversational tone, emotion, and formality level of the character. | | **Narrative Design** | Incorporates objectives or context from active Narrative Design sections into the user’s input. | | **Knowledge Bank** | Adds relevant external knowledge to improve factual accuracy or domain-specific responses. | | **Long-Term Memory** | Injects persistent information learned across sessions, when applicable. | *** ## Use Cases * Debug and refine how your **character’s prompt** is constructed. * Identify missing or conflicting information within the character setup. * Validate that the right **Knowledge Bank**, **Personality Traits**, and **Narrative Design** data are being included in responses. *** ## Conclusion The Mindview tab gives creators deep transparency into the inner workings of Convai’s character response generation.\ By analyzing prompts and understanding how context is layered, you can fine-tune your characters for **more consistent**, **accurate**, and **personality-aligned** interactions. # Narrative Design | Playground Build goal‑oriented conversation flows using sections, decisions, and triggers that move the story forward without rigid dialogue trees. ## Introduction Narrative Design lets you guide a character with high‑level objectives while keeping conversations flexible. Instead of hard coding a tree of lines, you define goals and decision points, then allow the character to respond dynamically. This approach works for many domains such as games, learning and training simulations, tourism, retail assistants, and customer support kiosks. You can read more about the considerations behind Narrative Design [here.](https://convai.com/blog/convai-narrative-design)
*** ## Videos Watch this series of videos to learn how to create a Narrative Design Graph in the Convai Playground.\ The demo features a Tour Guide scenario, showing step-by-step how to design, connect, and implement your own Narrative Design flow. {% embed url="" %} *** ## Accessing Narrative Design Open your character in the Convai Playground and select '**Narrative Design**' from the left sidebar. You will see a graph editor where you can connect the flow using nodes. *** ## Narrative Graph A narrative graph is made of four building blocks: ## Sections A Section contains: * **Objectives** – The goal the character aims to achieve in this part of the narrative.\ \&#xNAN;*Example:* A virtual tour guide’s objective could be to welcome the user and ask if they want to begin the tour. * **Decisions** – Choices based on user interaction that direct the character to different sections.\ \&#xNAN;*Example:* If the user says “yes” to a tour, the next section might start the tour route; if “no,” the character might offer alternative information. {% hint style="warning" %} Ensure decisions are clear and unambiguous; otherwise, the intended section may not be triggered. {% endhint %} Each Section has a **unique ID**.
*** ## Triggers A trigger is a simple signal from your application indicating that a certain condition has been met. When fired, triggers advance the graph to the next connected section. Each Trigger has a **unique ID**. **Examples** * **Location Based (Spatial):** your app detects the user entered a zone and fires the trigger associated with that Section. * **Time Based:** a timer in your app expires and fires the trigger. * **Event Based:** an in‑app event occurs such as “safety demo completed” and you fire the trigger.
*** ## Example Scenarios To better understand how Narrative Design works in practice, here are two example characters you can explore directly in Convai Playground.\ Open each link, navigate to the **Narrative Design** tab, and review how the graph is structured with Sections and Triggers. ### Factory Tour Guide – [View Character](https://convai.com/pipeline/dashboard/character?id=8cd9fa0c-384b-11ef-a852-42010a7be00e) A training simulation scenario set in a manufacturing facility.\ This character uses location-based triggers (e.g., entering the conveyor belt area or assembly line) to guide users through the workspace, explain safety protocols, and progress the tour.\ Ideal for **industrial training** and **onboarding simulations**.
### Real Estate Home Tour Guide – [View Character](https://convai.com/pipeline/dashboard/character?id=4d31ce84-8c6a-11ef-bc7a-42010a7be011) A real estate simulation where the character guides potential buyers through different rooms in a property.\ Similar to the factory example, it uses **location-based triggers** — for instance, when the user enters a specific room (e.g., kitchen, bathroom, bedroom), the corresponding Section in the Narrative Graph is triggered.\ This allows the character to dynamically adapt its dialogue to the user’s movement through the property.\ Useful for **virtual property tours**, **sales presentations**, and **customer onboarding**.
*** ## Syntax Instructions These special characters can be added to nodes in your Narrative Design graph to control specific outcomes and behavior. | Special Characters | Example | Use | | ------------------ | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | \ | \ I'll say this exact line! \ | Forces the character to respond exactly with the phrase inside the tags, without paraphrasing or adding extra context. | | \* | | Forces an immediate transition to the next node, bypassing further decision checks. | # External API Learn how to integrate and configure the External API feature to enable your characters to access real-time information, create tasks, and interact with third-party platforms. ## Introduction The **External API** feature empowers your characters to interact intelligently with real-time data sources and third-party services. Whether it’s retrieving live weather updates, tracking sports scores, or creating tickets in platforms like Jira and Trello, this feature allows seamless API-based integration. With just a few configuration steps, your characters can fetch data, trigger workflows, and execute automated actions, making them significantly more capable. {% embed url="" %} *** ## Configuration and Usage ### 1. Accessing the External API Page Navigate to the **External API** section in your dashboard. Here you can view existing API methods, activate or deactivate them, and create new methods. To add a new API method, click **Add API Method**.
*** ### 2. Creating an API Method **Method Fields Overview** * **Method Name** – Select an existing template or enter a unique name for your method. * **Method Description** – Provide a concise explanation of the method’s functionality. * **Input Description (JSON Format)** – Define required input parameters and their descriptions. * **Implementation Code** – Write the Python implementation for your API logic. * **Inputs** – Enter test parameters for validating your method. * **Output** – Displays the result when you click **Test API**.
*** ## Example 1 – Get Weather Data **Method Name**\ `Get Weather` **Method Description**\ `Fetches current weather data for a given city` **Input Description** ```json { "parameters": { "city": { "type": "string", "description": "Name of the city to get weather information for (e.g., 'London', 'New York', 'Tokyo')" } }, "required": [ "city" ] } ``` **Implementation Code** ```python import requests API_KEY = "" def handle_event(data): city = data.get("city") url = f"https://api.openweathermap.org/data/2.5/weather?q={city}&appid={API_KEY}" response = requests.get(url) weather_data = response.json() return {"weather": weather_data["weather"][0]["description"]} ``` **Setup Notes** 1. Sign up at [OpenWeatherMap](https://openweathermap.org/) and get your API key. 2. Replace `` in the code with your key. **Test Input** ```json { "city": "New York" } ``` Click **Test API**. **A successful Output Example:** ```json { "weather": "clear sky" } ```
#### **Activate the method** If the test passes, click **Save Changes**, return to the main API list, and enable the method by toggling **Connect** to green.
#### **Test with a character** Once activated, test the method in a conversation with your character.\ As seen in the screenshot below, the character correctly returned the current weather for Roma and Wrangell.
*** ## Example 2 – Create Jira Support Ticket **Method Name**\ `Creating Support Tickets` **Method Description**\ `Creates a support ticket on Jira` **Input Description** ```json { "parameters": { "summary": { "type": "string", "description": "Short title for the Jira ticket" }, "description": { "type": "string", "description": "Detailed description of the Jira issue" } }, "required": [ "summary", "description" ] } ``` **Implementation Code** ```python import requests from requests.auth import HTTPBasicAuth import json # Jira configuration JIRA_DOMAIN = "mycompany.atlassian.net" # Replace with your Jira domain EMAIL = "user@example.com" # Replace with your Atlassian account email API_TOKEN = "abc123xyz456..." # Replace with your Jira API token JIRA_PROJECT_KEY = "EX" # Replace with your Jira project key ISSUE_TYPE = "Story" # Issue type: Story, Task, or Bug API_ENDPOINT = f"https://{JIRA_DOMAIN}/rest/api/3/issue" # Standard headers required by JIRA. headers = {"Accept": "application/json", "Content-Type": "application/json"} def create_jira_ticket(ticket_data): """ Create a JIRA ticket using provided ticket_data dictionary. Expected ticket_data keys: - summary: (str) Brief summary of the issue. - description: (str) Detailed description of the issue. Returns: - JSON response if the ticket is created successfully. - Error message if there was an error. """ # Convert plain text description to Atlassian Document Format description_adf = { "version": 1, "type": "doc", "content": [ { "type": "paragraph", "content": [ {"type": "text", "text": ticket_data.get("description", "")} ], } ], } # Construct the payload for the JIRA issue payload = { "fields": { "project": {"key": JIRA_PROJECT_KEY}, "summary": ticket_data.get("summary"), "description": description_adf, "issuetype": {"name": ISSUE_TYPE}, } } # Convert the payload to a JSON string payload_json = json.dumps(payload) # Send a POST request to the JIRA API endpoint response = requests.post( API_ENDPOINT, data=payload_json, headers=headers, auth=HTTPBasicAuth(EMAIL, API_TOKEN), ) # Check for a successful creation (HTTP 201 Created) if response.status_code == 201: return response.json() else: return {"error": f"Failed to create ticket: {response.status_code}"} def handle_event(data): return create_jira_ticket(data) ``` **Where to Find Required Values** * **JIRA\_DOMAIN** – Found in your Jira account URL. Example:\ `https://mycompany.atlassian.net` → `JIRA_DOMAIN = "mycompany.atlassian.net"` * **EMAIL** – Your Atlassian login email. * **API\_TOKEN** – Create from [Atlassian API Tokens](https://id.atlassian.com/manage-profile/security/api-tokens). * **JIRA\_PROJECT\_KEY** – Found in your project URL or next to the project name. * **ISSUE\_TYPE** – Must be valid in your Jira project (Story, Task, Bug). **Test Input** ```json { "summary": "This is to test ticket creation", "description": "Created using External API" } ``` Click **Test API**. **A successful Output Example:** ```json { "id": "10004", "key": "EX-5", "self": "https://mycompany.atlassian.net/rest/api/3/issue/10004" } ```
**Activate the method**\ If the test passes, click **Save Changes**, return to the main API list, and enable the method by toggling **Connect** to green.
**Test with a character**\ Once activated, test the method in a conversation with your character.\ As seen in the screenshot below, the character successfully created a Jira ticket and returned the ticket key.
*** ## Limitations and Supported Environment {% hint style="warning" %} **Supported LLM Models**: GPT-4o, GPT-4o-mini, Claude-3.5, Claude 4.0 {% endhint %} {% hint style="warning" %} **Max Execution Time**: 5 seconds {% endhint %} {% hint style="warning" %} **Python Version**: 3.12 {% endhint %} {% hint style="warning" %} **Libraries Available**: Standard library + requests {% endhint %} *** ## Conclusion By configuring the External API feature, you can transform your characters into powerful, data-driven assistants. From retrieving real-time weather information to creating Jira tickets directly from a conversation, the possibilities are vast. This integration capability enables highly interactive, automated, and intelligent workflows. # Publish Learn how to publish and share your Convai Experience with the public, selected users, or embed it on your own website. ## Introduction The **Publish** page allows you to share your fully created and customized Convai Experience with the world or with a selected group of people. From here, you can configure the title, description, thumbnail, and visibility settings for your experience, as well as generate links for sharing.
*** ## Publishing Options and Visibility Settings The **Details** tab contains all the essential settings for publishing your experience: * **Experience Link** – A direct link to your experience for easy sharing. * **Experience Name** – The display name for your published experience. * **Experience Description** – A short summary describing your experience. * **Thumbnail** – By default, this uses the selected environment’s image, but you can upload a custom thumbnail. * **Visibility Settings** – Controls who can see and interact with your experience: * **Public** – Your experience can be discovered and accessed by anyone on `x.convai.com`. * **Unlisted** – Only users with the direct link can access the experience. * **Private** – The experience is restricted to invited users only. * When set to **Private**, the **Share Privately** button becomes active. Here you can: * Enter the email address of the person you wish to invite. * Alternatively, share the experience link with the invited user to grant access. *** ## Embed Experience The "Embed Experience" tab allows you to embed Convai Experiences directly into your own website. This feature makes it easy to integrate interactive experiences into custom platforms or applications. {% hint style="danger" %} Convai Pixel Streaming Embed is currently accessible only with the Professional Plan and above. {% endhint %} *** ## Conclusion The Publish page provides all the tools you need to control how your experience is shared, whether you want it available to the public, only to select individuals, or embedded directly into your website. By choosing the right visibility settings, you can ensure your experience reaches the right audience in the right way. # Avatar Studio Experiences Create intelligent 3D AI avatars directly in your browser — no downloads, no code, fully customizable. ## Introduction Convai’s Avatar Studio is a user-friendly platform that allows anyone to create intelligent, high-quality 3D conversational avatars — right from your web browser. {% embed url="" %} *** ## **What You Can Do** With a simple interface, you can design and deploy fully interactive avatars that: * Speak and respond via **voice and text** * Perform **intelligent animations** * Adapt to **different virtual environments** * Are **fully customizable** * Optionally use **vision-based input** to "see" the user and react with natural, personalized responses, enhancing realism *** ## Who It’s For Avatar Studio is perfect for: * **Creators** and **developers** building digital characters * **Educators** creating engaging learning experiences * **Brands** looking to enhance digital events * **Game designers** needing lifelike NPCs * Anyone interested in AI-powered **interactive storytelling** Whether you're creating an NPC for a game or a digital host for a virtual event, Convai’s Avatar Studio helps you bring your characters to life—quickly and easily. *** ## Key Features * **Conversational AI**\ Avatars engage in natural, human-like voice or text conversations. * **High-Quality Metahuman NPCs**\ Realistic 3D Metahuman avatars with high-fidelity lip-sync, natural eye-blinking, and intelligent animations. * **Runs Entirely on the Browser**\ No downloads, installations, or GPU power needed — just open and start creating. * **Intelligent Actions and Animations**\ Avatars react with gestures such as waving, thinking, and expressing emotions based on the conversation context. * **Proactive & Agentic AI Characters**\ Characters can initiate conversations and act autonomously in response to their environment. * **Vision-Based Interaction**\ Avatars can perceive users via camera input and respond with contextually appropriate and human-like reactions. * **High-Quality Backgrounds**\ Choose from immersive environments to place and enhance your characters. * **Total Customization**\ Fully personalize the avatar’s appearance, voice, actions, environments, branding and more. # Customizing Your Avatar Learn how to visually and behaviorally personalize your Convai avatar using the Avatar Studio configurator. ## **Overview** Once your character is created, you can start customizing how they look, move, and interact using the **Avatar Studio**. {% embed url="" %} ## What You Can Customize Here’s what you can do inside the Avatar Studio: * **Choose a Sample Avatar,** Pick from a library of ready-to-use, high-fidelity avatars. * **Customize Appearance**\ Modify facial features, clothing, hairstyles, and other visual elements to reflect your character’s identity. * **Upload Your Own Avatars**\ Prefer a unique design? Upload your own 3D avatar models for full control over their look. * **Set Up Intelligent Animations**\ Configure gestures like waving, nodding, reacting, and thinking — all triggered contextually during conversations. * **Select a Virtual Environment**\ Place your avatar in immersive digital scenes that match the tone and use case of your experience. * **Adjust Interaction Behavior**\ Fine-tune how your avatar communicates — such as their speaking style, tone, and user engagement preferences. * **Device & Branding Adaptation**\ Customize how the avatar interface behaves across different devices and align it with your brand’s visual identity. After you finish customizing, simply **save and publish** your avatar to bring it into your character’s conversations. # Configure Avatar Learn how to choose, customize, or upload avatars in Avatar Studio. Once you enter the **Avatar Studio**, you can define exactly how your avatar looks and behaves. Here’s how to get started: ## **Choosing a Sample Avatar**
You can start by selecting a high-quality S**ample Avatar** from Convai’s library. These are designed to cover a wide range of character types and use cases. * Browse the available avatars by scrolling through the list. * Click on the one you want to use. * That avatar will be instantly applied to your character. ## Creating a Custom Avatar
If you want something more unique, you can create a custom avatar using the built-in editor. 1. Go to the **“Craft your own”** tab. 2. Click the **plus icon (+)** to start creating a new custom avatar. 3. Give your avatar a **name**. 4. Customize features such as: * **Facial features** * **Hairstyle & hair color** * **Skin tone & texture** * **Outfits & accessories** * **Age appearance** 5. Under the **“Brand”** section: * Use sliders to apply your logo on supported clothing items. * Make sure **“Display logo on cloth”** is enabled under Interface Settings. This allows for highly personalized avatars that align with your branding and narrative needs. ## **Uploading Your Own Avatar** If you'd like to upload a custom avatar model, please follow our detailed guide here: {% content-ref url="/pages/3gF4TWlNAR7gdcrwFIfb" %} [Uploading Avatars](/api-docs/no-code-experiences/avatar-studio-experiences/customizing-your-avatar/configure-avatar/uploading-avatars) {% endcontent-ref %} # Uploading Avatars {% hint style="warning" %} Currently, only Metahuman and Reallusion avatars are supported for upload. {% endhint %} {% content-ref url="/pages/7RiOQDqXiSshcIxvesJU" %} [Metahuman Avatars](/api-docs/no-code-experiences/avatar-studio-experiences/customizing-your-avatar/configure-avatar/uploading-avatars/metahuman-avatars) {% endcontent-ref %} {% content-ref url="/pages/KaEFOuvtwU0O5irCjn9v" %} [Reallusion Avatars](/api-docs/no-code-experiences/avatar-studio-experiences/customizing-your-avatar/configure-avatar/uploading-avatars/reallusion-avatars) {% endcontent-ref %} # Metahuman Avatars Upload custom Metahuman characters from Unreal Engine to Avatar Studio using the Convai Asset Uploader. ## Introduction This guide walks you through uploading **custom Metahuman avatars** to **Avatar Studio** using the **Convai Asset Uploader**. You'll generate a new project tailored for Metahumans, import your Metahuman asset, configure it, and finally upload it using Convai’s built-in tools. ## Prerequisites Before you begin: * Create your project using the [**Convai Asset Uploader**](/api-docs/plugins-and-integrations/asset-uploader), and answer **`Y`** when asked if you’re using a Metahuman. * Ensure you have a downloadable Metahuman available via **Quixel Bridge**. *** ## Step-by-Step Guide ### 1. Open the Project Navigate to the folder where your project was created. Double-click the `YourProjectName.uproject` file to open it in Unreal Engine. *** ### 2. Add a Metahuman via Quixel Bridge * Go to **Window > Quixel Bridge**. * In Bridge, select **Metahumans** from the left-hand menu. * Pick a Metahuman and click: * **Download** (if not already downloaded) * Then **Add** to include it in your project. *** ### 3. Locate and Open the Character Blueprint After importing: * Go to `Content/Metahumans//`. * Open the Blueprint: `BP_` > ⏳ This may take some time to load. *** ### 4. Fix Compile Errors If you see compile errors: * In the bottom-right, click **Enable Missing** under any **Missing Plugins** or **Missing Project Settings** notices.
* Click **Restart Now** when prompted. * Reopen the Blueprint and ensure it compiles successfully. *** ### 5. Prepare the Asset for Upload 1. Locate the folder:\ `Plugins/ Content/`\ (e.g., `Plugins/AHK3LNKVC7FZA3I5JG3V Content/`) 2. Move the entire `Content/Metahumans/` folder into this directory. * Use **Move Here** to complete the action. * The final structure should mirror what’s shown in the screenshot.
{% hint style="info" %} This folder determines what gets packaged and uploaded. Make sure everything is placed correctly. {% endhint %} *** ### 6. Open the Asset Uploader Tool * Navigate to `Content/Editor/AssetUploader`. * Right-click on `AssetUploader` and select **Run Editor Utility Widget**. *** ### 7. Select the Character Asset * Navigate to the `Plugins/ Content/Metahumans//` directory. * Select the `BP_` Blueprint. * Then, in the **Asset Uploader** window, click **Pick Asset**. *** ### 8. Capture a Thumbnail * In the Asset Uploader window, click **Capture Thumbnail** to generate a preview image for your avatar. *** ### 9. Verify Functionality Before Upload 1. Drag `BP_` into the Level. 2. Select the character and locate `BP_ConvaiChatbotComponent` in the **Details** panel. 3. Input a test **Character ID**. 4. Press **Play** and confirm: * Animations are working * Lip sync is functional * Character behaves as expected *** ### 10. Upload the Avatar * In the Asset Uploader, click **Create Asset**. * This will: * Package the avatar for Win64 * Upload it to Avatar Studio Monitor the **Output Log**: * Look for `Package completed` * Then wait for `Uploaded Asset` {% hint style="warning" %} If there’s an error during packaging, check the logs and share them on the [Convai Developer Forum](https://forum.convai.com/) for support. {% endhint %} {% hint style="info" %} To delete a previously uploaded asset, open AssetUploader and click **Delete**. {% endhint %} *** ## Accessing the Avatar 1. Go to [Avatar Studio](https://convai.com/) 2. Open the **Upload Your Custom Avatar** section 3. Your Metahuman will appear, ready for use. *** ## Summary Using the Convai Asset Uploader, uploading custom Metahuman avatars is quick and reliable. With proper setup and a few clicks, your characters are live in Avatar Studio and ready for real-time AI interaction. # Reallusion Avatars Upload custom Reallusion characters from Unreal Engine to Avatar Studio using the Convai Asset Uploader. ## Introduction This guide explains how to prepare and upload **Reallusion-based avatars** using the **Convai Asset Uploader**. You’ll import your Reallusion character and animations, apply Convai’s animation and lipsync systems, and then upload your avatar to Avatar Studio using the built-in AssetUploader tool. *** ## Prerequisites Make sure you have the following ready: * A project created with the [**Convai Asset Uploader**](/api-docs/plugins-and-integrations/asset-uploader), where you answered **`N`** to “Are you using a Metahuman?” * A custom Reallusion character exported and ready for import *** ## Step-by-Step Guide ### 1. Open the Project Navigate to your project directory and open the `.uproject` file to launch it in Unreal Engine. *** ### 2. Import Reallusion Character & Animations Follow this [video tutorial](https://youtu.be/UyxNliF8LKU?feature=shared) to import your Reallusion assets: * **\[00:00 – 07:25]**: Import your character and animations * **\[07:50 – 08:20]**: Create a new Blueprint Class for your character *** ### 3. Connect Convai Animations Now we’ll bind the correct animation logic to your character. We’ve already added the necessary Animation Blueprint for you: * Go to `Content/ConvaiReallusion/` * Locate and assign the **ConvaiReallusion Animation Blueprint** to your character’s Skeletal Mesh {% hint style="info" %} This blueprint ensures that your Reallusion character plays proper idle/talking animations in sync with Convai interactions. {% endhint %} * Refer to the [tutorial](https://youtu.be/UyxNliF8LKU?feature=shared) for this step: **\[10:12 – 12:48]** *** ### 4. Add FaceSync for Lipsync To enable lipsync: * Add the `FaceSync` component to your character’s Blueprint * See how in the same [video](https://youtu.be/UyxNliF8LKU?feature=shared): **\[12:48 – 12:56]** *** ### 5. Set Correct Rotation Reallusion characters typically face the wrong direction by default. Fix this by: * Opening the character Blueprint * Selecting the **SkeletalMesh** component * Set the **Z Rotation** to `-90` in the **Details** panel *** ### 6. Prepare Files for Upload 1. Go to:\ `Plugins/ Content/`\ (e.g., `Plugins/AHK3LNKVC7FZA3I5JG3V Content/`) 2. Drag and **Move** both of the following folders into this directory: * Your character’s folder (containing the Blueprint and animations) * `Content/ConvaiReallusion/` (contains the animation blueprint) * The final structure should mirror what’s shown in the screenshot.
{% hint style="info" %} This folder determines what gets packaged and uploaded. Make sure everything is placed correctly. {% endhint %} *** ### 7. Open the AssetUploader Tool * Navigate to `Content/Editor/AssetUploader` * Right-click and select **Run Editor Utility Widget** *** ### 8. Select the Character Asset * Navigate to `Plugins/ Content/YourCharacterFolder/` * Select your character’s **Blueprint Class** * Then, in the Asset Uploader window, click **Pick Asset** *** ### 9. Capture a Thumbnail Click **Capture Thumbnail** to create a preview image that will appear in Avatar Studio. *** ### 10. Verify Before Upload Before uploading, do a quick functional test: 1. Drag the character into your Level 2. Select it and locate the `BP_ConvaiChatbotComponent` in the **Details** panel 3. Paste in a test **Character ID** 4. Press **Play** and verify: * Animation is working * Lipsync is functioning * Character is correctly positioned and oriented *** ### 11. Upload the Avatar * In the Asset Uploader, click **Create Asset** * This triggers: * Packaging the asset for **Win64** * Uploading to **Avatar Studio** Monitor the **Output Log**: * Wait for `Package completed` * Then look for `Uploaded Asset` {% hint style="warning" %} If there’s an error during packaging, check the logs and share them on the [Convai Developer Forum](https://forum.convai.com/) for support. {% endhint %} {% hint style="info" %} To delete a previously uploaded asset, open AssetUploader and click **Delete**. {% endhint %} *** ## Accessing the Avatar 1. Visit [Avatar Studio](https://convai.com/) 2. Go to **Upload Your Custom Avatar** 3. Your Reallusion character will now be available for selection and use *** ## Summary Using the Convai Asset Uploader, uploading Reallusion avatars is quick and reliable. With proper setup and a few clicks, your characters are live in Avatar Studio and ready for real-time AI interaction. # Face Filter Use the Face Filter feature to make your avatar resemble a specific person based on a photo. {% hint style="danger" %} The Face Filter feature is available only with the Scale plan and above. {% endhint %} ## **What is Face Filter?** Face Filter allows you to personalize your avatar’s appearance to look like a specific person using a reference photo.
## **How to Use It?** ### **1. Enable Face Filter** Toggle on the **Face Filter** option inside the avatar customization panel. ### **2. Upload an Image** Click **“Upload your own image”** to add a photo reference. ### 3. Apply the Image Select the uploaded image by clicking on it. Your avatar’s face will automatically morph to resemble the person in the photo. ### **4. Manage Images** * To delete an image, click on it and select **“Delete image”**. * You can upload **multiple images** to try different looks. With Face Filter, you can achieve even more lifelike, personalized characters — perfect for storytelling, training simulations, or representing real individuals in virtual settings. # Environment Choose from immersive 3D and Solid environments to place your avatar in the right setting.
## **Bring Your Character to Life with the Right Setting** Selecting an environment helps anchor your avatar in a scene that matches your use case — whether it’s professional, playful, or futuristic. You can choose from a variety of **immersive 3D and Solid environments**, including: * A sleek, **modern office** * A **sci-fi room** with futuristic vibes * A warm and inviting **cozy lounge** * A minimal and practical **kiosk-style backdrop** These environments serve as the **visual context** for your avatar’s interactions, making conversations feel more realistic and engaging for your audience. {% hint style="success" %} Match the environment with the personality or purpose of your character. {% endhint %} # Uploading Scenes Upload your custom Unreal Engine Levels to Avatar Studio with Convai Modding Tool. ## Introduction This guide explains how to upload **custom scenes (Levels)** from Unreal Engine 5.3 to **Avatar Studio** using the **Convai Modding Tool**. Whether you’re building rich environments or AI-driven stages, you’ll learn how to prepare, tag, and upload them in just a few steps. *** ## Prerequisites Make sure you have: * Created your project using the [**Convai Modding Tool**](/api-docs/plugins-and-integrations/asset-uploader) * Selected **`1`** when asked whether you are uploading a Scene or Avatar * A fully prepared and tested Unreal Engine Level (with all necessary assets) *** ## Step-by-Step Guide ### 1. Open the Project Open the project created by the Modding Tool by double-clicking the `.uproject` file. *** ### 2. Import the Scene Import your custom Level and all required assets (meshes, environment packs, etc.) into your project. Make sure the Level is fully playable and doesn’t contain any broken references. *** ### 3. Editor (Player) Start Point To set the spawn location of the editor (Player) in the Level 1. Place an **Actor** in your Level 2. Move it to the desired **spawn location** 3. In the **Details** panel, set the **Tag** of the Actor to: ``` EditorSpawn ``` {% hint style="warning" %} This tag is essential. It helps the system know where to place the editor (player) once the scene is loaded in Avatar Studio. {% endhint %} *** ### 4. Prepare Files for Upload 1. Navigate to:\ `Plugins/ Content/`\ (e.g., `Plugins/AHK3LNKVC7FZA3I5JG3V Content/`) 2. Drag and **Move** your Level’s folder and its dependencies into this directory. {% hint style="info" %} This ensures the Level and all its references are included during packaging and upload. {% endhint %} *** ### 5. Open the AssetUploader Tool * Go to `Content/Editor/AssetUploader` * Right-click and choose **Run Editor Utility Widget** *** ### 6. Select the Level Asset * In the **Content Browser**, go to the directory inside\ `Plugins/ Content/` * Select your **Level asset** * Return to the **Asset Uploader** window and click **Pick Asset** *** ### 7. Capture a Thumbnail Click **Capture Thumbnail** to generate a preview image that will appear in Avatar Studio. *** ### 8. Upload the Scene Once everything is ready, click **Create Asset** in the Asset Uploader. The process includes: * **Packaging** the Level for Win64 * **Uploading** the Level to Avatar Studio Check the **Output Log**: * Look for `Package completed` * Then `Uploaded Asset` to confirm successful upload {% hint style="warning" %} If there’s an error during packaging, check the logs and share them on the [Convai Developer Forum](https://forum.convai.com/) for support. {% endhint %} {% hint style="info" %} To delete a previously uploaded asset, open AssetUploader and click **Delete**. {% endhint %} *** ## Accessing the Scene 1. Visit [Avatar Studio](https://convai.com/) 2. Open the **Upload Your Custom Scene** section 3. Your Level will appear in the list, ready for testing or integration with avatars *** ## Summary Uploading custom Levels allows you to create rich environments for your AI avatars in Convai Sim. The Modding Tool automates most of the setup — just make sure to set your spawn point and move your assets to the correct folder. # Lighting Adjustments
## Set the Right Mood with Lighting Lighting plays a key role in how your avatar looks and feels within the environment. It affects not only visibility, but also the overall **tone and atmosphere** of the scene. Here’s how you can adjust lighting for your avatar: * **Choose a Lighting Preset**\ Use the dropdown menu to select from several **preset lighting setups.** * **Adjust Lighting Power**\ Use the **power level slider** to fine-tune the light. {% hint style="success" %} Subtle lighting changes can make a big difference in realism — experiment to see what best fits your character and scene. {% endhint %} # Animation & Expression Settings Customize your avatar’s expressiveness with facial and body animations, emotions, and intelligent actions. ## Make Your Avatar Come Alive Convai’s Avatar Studio lets you fine-tune how expressive your avatar is — from subtle facial expressions to full-body gestures and smart actions. ## **Facial & Body Animation**
Use sliders to define the intensity of animations: * **Facial Animation** * Range: **-1 to 1** * Lower values result in **minimal expressiveness**, while higher values make your avatar more **emotionally responsive**. * **Body Animation** * Range: **Low to High** * A low setting keeps the avatar **more static**, while high adds **dynamic hand and body movements** for livelier interaction. ## **Initial Facial Expression**
* **Camera Focus Toggle**\ Enable or disable eye contact with the user by toggling **camera focus**. You can define how your avatar appears at the start of an interaction: * **Enable or Lock Expressions**\ Use toggles to either: * Allow expressions to change during conversation * Lock the avatar into a specific expression * **Select an expression** from the dropdown: * Joy * Trust * Fear * Surprise * Sadness * Disgust * Anger * Anticipation * And more... ## **Custom Actions**
Give your avatar intelligent behaviors during interactions — like waving hello or thinking. **How to Add a Custom Action:** 1. Click **“Add a new action”**. 2. Toggle **Eye Focus** on or off. 3. Click **“Select animation”**. 4. Choose from available animations (e.g., **Wave Animation** for greeting). 5. Name your action (e.g., **Waves Cheerfully**). 6. Click **“Preview Animation”** to test how it looks. # Interface Configuration Tailor the visual and functional interface of your avatar experience to match your device, context, and brand needs. ## Customize Your Avatar Experience Interface Convai’s Avatar Studio provides a variety of settings to adapt the interface layout, interaction mode, and branding for different platforms and use cases. ## **Screen Resolution Presets**
Choose the layout that best fits your deployment: * **Desktop** * **Tablet** * **Mobile** * **Kiosk** This ensures optimal visual presentation across different screen types. *** ## **Chatbox Settings**
Enable or customize the chat interface as needed: * **Chatbox Type**\ Select your preferred chatbox style from available templates. * **Disable Chat Interface**\ Use the toggle to hide the chatbox completely if not needed. * **Push-to-Talk Mode**\ Enable push-to-talk using the toggle for voice-activated interactions. *** ## Character Vision Through Webcam
Let your avatar “see” the user and respond accordingly using webcam input. * Enable or disable **vision-based input** with a toggle. * **Position** the webcam within your interface layout. * Adjust the **webcam display size** using a slider for optimal placement. *** ## **Camera Settings**
Control how the avatar scene is viewed by the user. * **Field of View (FOV):**\ Adjust using the FOV slider (left = narrower, right = wider view) * **Pan Camera:** * Up/Down with “Pan Up/Down” slider * Left/Right with “Pan Left/Right” slider * **Zoom and Tilt:**\ Adjust using their respective sliders for precise framing. *** ## **Branding Options**
Integrate your brand identity directly into the avatar experience: * **Display Logo**\ Toggle “Display Logo” to enable branding elements. * **Upload Your Logo**\ Click “Upload your brand logo” to add it to your scene. * **Manage Logo Display** * Click the logo to place it in the experience * Adjust its **position** and **size** using provided controls * **Logo on Clothing**\ Enable “Display logo on cloth” to embed your logo onto the avatar’s clothing. *(Only available for specific clothing items)* These configuration tools ensure that your avatar interface not only works smoothly across platforms but also aligns with your project’s style, interaction needs, and brand. # Experience Settings Control idle session handling, welcome interactions, microphone behavior, and input processing timing to fine-tune your avatar experience. ## **Final Touches Before Deployment** These settings define how your avatar behaves during live interaction and how the experience is sustained or terminated based on user activity. *** ## **AFK Timeout**
Set an **AFK (Away-From-Keyboard)** timeout to manage idle sessions and conserve your pixel-streaming minutes. * Open the **dropdown menu** under AFK Timeout. * Select a suitable timeout duration (e.g., 1 min, 5 mins, 10 mins). * The session will automatically end based on your selection if there’s no user activity. {% hint style="success" %} To optimize your account's usage, set an AFK timeout to avoid unnecessary streaming consumption. {% endhint %} *** ## **Welcome Message** Greet users as soon as they enter the experience with customizable welcome interactions. **Standard Welcome**
* Toggle **“Welcome Message”** ON. * Enable a **custom welcome prompt** (e.g., “Welcome the user and introduce yourself”). * Click **“Test Welcome Message”** to preview it. **Vision-Based Welcome**
Make the greeting more dynamic by enabling vision awareness: * Toggle **“Vision-Based Welcome Message”** ON. * Add a **custom prompt** (e.g., “A person approaches—welcome them and make a comment about their attire or an object they’re holding”). * Use **“Test Message”** to preview how the avatar responds based on webcam input. *** ## **User Microphone Settings**
Select the input style that fits your use case: * **Hands-Free Mode (WIP)**\ Avatar listens continuously. * **Push-to-Talk Mode**\ Activate microphone input only when the assigned key is pressed. * You can **assign a custom key** for push-to-talk functionality. *** ## Processing Frequency
Control how often the avatar processes and reacts to input, allowing it to act more proactively. * Open the **dropdown menu** for processing frequency. * Choose a time interval for the avatar to periodically evaluate multimodal inputs (voice, text, vision). * This enables **agentic behavior** — where the avatar can initiate interaction based on user presence or signals. {% hint style="danger" %} After making changes: * Click **“Save Changes”** * If you are creating an Avatar for this character for the first time, press the **"Create Character"** button **before proceeding to the publishing step.** {% endhint %} # Publishing an Avatar Studio Experience Learn how to publish and share your customized avatar experience for use across web, kiosks, apps, and more. ## **Ready to Share Your Experience with the World?** Once your character and avatar setup is complete, you can publish your experience directly from the Convai Character Creator dashboard. {% hint style="success" %} **Before You Begin:**\ Make sure your avatar is **saved** and the character is **created** before navigating to the **Publish** tab. {% endhint %}
*** ## **Publishing Steps** ### 1. Go to the **Publish** tab inside your character’s dashboard. ### **2. Finalizing Your Experience** Fill in the necessary details to define and present your simulation: * **Experience Name**\ e.g., *Virtual Tour of the Fire Station* * **Experience Description**\ e.g., *Get a deeper look and understanding of the inner workings of a fire station with your virtual tour guide Lina!* * **Thumbnail (Optional)**\ Upload an image to visually represent your experience. ### 3. Choose Visibility Settings Select how and with whom the experience should be shared: * **Public** * Visible to everyone * Accessible on [**x.convai.com**](https://x.convai.com) * **Private** * Only visible to you and invited users * **Unlisted** * Not listed publicly, but can be accessed via a direct link * **Embed on Your Site** *(Enterprise-only)* * Publish your experience directly to your own website {% hint style="danger" %} **Convai Pixel Streaming Embed** is currently accessible only with the **Enterprise plan**.\ To learn how to embed an avatar into your own platform, check out the [Embedding Documentation](/api-docs/plugins-and-integrations/convai-pixel-streaming-embed). {% endhint %} *** ## **After Publishing** Once published, your experience is ready to be deployed on: * Websites * Applications * Kiosk systems * Any supported digital platform *** {% content-ref url="/pages/mCC3cioDPzGqMj6BUUat" %} [Convai Pixel Streaming Embed](/api-docs/plugins-and-integrations/convai-pixel-streaming-embed) {% endcontent-ref %} # Convai Sim Experiences Create AI-powered avatars and deploy them in interactive 3D environments—directly from your browser. ## **Introduction** Convai Sim is a browser-based platform that allows you to instantly create and deploy AI-powered avatars in interactive 3D environments— no downloads, and no complex setup required. Designed for creators, educators, and developers, it enables rapid prototyping and deployment of lifelike characters inside rich, responsive scenes. *** ## **What You Can Do** With Convai Sim, you can: * Add one or more **AI-powered avatars** into a 3D scene * Set up **real-time interactions** using voice or text * Deploy avatars with **smart navigation** and context-aware behaviors * Easily position characters using **drag-and-drop scene editing** * Instantly publish your experience for testing or deployment * Run everything **directly in your browser.** *** ## **Who It’s For** Convai Sim is perfect for: * **Educators and trainers** building interactive simulations or learning environments * **Storytellers and creators** wanting to bring characters to life in immersive scenes * **Game developers** prototyping scenarios and NPC interactions * **Enterprises** creating training, onboarding, or customer-facing virtual flows * **Tourism and museum teams** looking for guided, avatar-led experiences Whether you're designing a futuristic training program or a playful game level, Convai Sim makes it easy to bring intelligence and interactivity to 3D worlds. *** ## **Key Features** * **Browser-Based Platform**\ No installations — launch and edit in-browser. * **Multi-Avatar Support**\ Add and manage multiple intelligent characters in a single scene. * **High-Quality Visuals**\ Use expressive avatars for rich storytelling and realistic simulation. * **Instant Deployment**\ Launch scenes immediately and preview interactions with one click. * **Intelligent Navigation**\ Characters move contextually, ideal for tour guide or training scenarios. * **Interactive Scene Editing**\ Easily arrange avatars and elements using drag-and-drop tools. * **Versatile Use Cases**\ Perfect for education, training, tourism, gaming, and more. # Creating Your AI Simulation with Convai Sim Bring your Convai characters to life by placing them into 3D interactive environments using Convai Sim ## Introduction Now that you’ve created a Convai character, it’s time to place them into a 3D simulation. With Convai Sim, you can bring characters to life inside immersive environments—fully interactive and embodied in high-quality avatars.
*** ## **Getting Started** Follow this step-by-step guide to launch your first AI-powered simulation using Convai Sim. ### **1. Access the Playground** Go to [**convai.com**](https://www.convai.com) and log into your account. Navigate to the **Playground** section from the dashboard. ### 2. Create a New Experience Click on **“Create a new experience”** to begin setting up your simulation. ### **3. Choose an Environment** Select an environment that fits your use case (e.g., office, museum, sci-fi room). Then click **“Start Experience”** to enter the Convai Sim. ### **4. Explore the Scene** Once the scene loads: * Use **WASD keys** to move around. * Use your **mouse** to look around the environment — just like in a first-person game. ### **5. Add an Avatar** Click the **top-left icon** to open the avatar menu. Then: * Click **“Add Avatar”**. * A hologram will appear — place it at the desired location in the scene. ### **6. Select Your Character and Avatar** You’ll be prompted to: * Choose your previously created **Convai character**. * Select a **Metahuman avatar** to visually represent that character. ### **7. Deploy the Character** Click **“Deploy Character”** to spawn the avatar into the environment. The avatar will now be active and ready to interact. ### **8. Add More Avatars (Optional)** Repeat the process to add **multiple characters** into the same scene and create more dynamic simulations. *** ## Summary You’ve now: * Created a Convai character * Selected a 3D environment * Embodied your character in a lifelike avatar * Brought them into an interactive simulation With **multi-avatar support**, you can quickly build rich, AI-driven experiences—from training simulations and virtual tours to interactive stories and games. Next, we’ll explore how to **customize your avatars and scenes** using the available tools. # Avatar Customization Fine-tune your deployed avatar’s appearance, size, and position within your 3D simulation scene ## **Refine Your Avatar for a Perfect Fit** Once you've placed your avatar into the scene, it's time to customize its model, pose, and placement to match your simulation's tone and context. *** ## **Customizing Your Avatar**
Follow these steps to adjust your avatar visually using built-in tools: ### **1. Select and Open the Character Tools** * Click directly on your avatar in the scene. * This will open the transform tools. ### **2. Customize Position, Rotation, and Scale**
You can adjust your avatar’s placement and appearance using **either visual tools or precise numeric fields**: #### **Option A – Use Transform Gizmos** * **Move Tool:** Drag the avatar along the **XYZ axes**. * **Rotate Tool:** Use the **blue ring** to turn the avatar’s facing direction. * **Scale Tool:** Resize the avatar by dragging the **top cube handle**. #### **Option B – Use the Edit & Publish Panel** * When the avatar is selected, the **Edit & Publish panel** appears. * Manually enter values for: * **Position** * **Rotation** * **Scale** This option is ideal when you need precise alignment, consistency across avatars, or exact placement within complex scenes. *** With both intuitive drag-and-drop controls and precision inputs, customizing your avatar's presence in the scene is flexible and efficient. **Next up:** Let’s bring your avatar to life with tour-guide behaviors and intelligent interactions! # Tour Guide Turn your AI avatar into an interactive tour guide using Convai Sim’s built-in tour planning tools ## **Introduction** In this guide, you'll learn how to bring your AI avatars to life by turning them into dynamic tour guides within immersive 3D environments. We’ll walk you through how to: * Set up a tour-guide simulation * Define interaction behaviors * Add tour points * Manage the tour flow * Preview and publish the experience ## Step by Step Guide ### **1. Setting Up Tour Guide Mode**
1. Select your avatar to open the **Edit & Publish** menu. 2. Locate the **Tour Planner Settings** section. 3. Set up your **Tour Prompts** – these define what the avatar says at the start and end of the tour. **Example Prompts:** * **Welcome Prompt:**\ “Hi there! Ready to explore the fire station?”\ → Greet the user, introduce yourself, and invite them to start the tour. * **End Prompt:**\ “That’s the end of the tour. Hope you had fun!”\ → Ask if the user has any questions, answer them, then say goodbye. ### **2. Defining Behavior**
Choose how your avatar initiates the tour: * **Wait for Player:**\ Avatar stays still and waits until the user approaches. * **Engage on Sight:**\ Avatar detects the user visually and initiates conversation. * **Max (Timed Engagement):**\ Avatar starts interacting after a set period of user inactivity. ### **3. Adding Tour Points**
1. Click **“Add Tour Point”** — a green gizmo will appear in the scene. 2. Use the **XYZ axes** or click the **flag icon** to position the tour marker. 3. Enter a **Tour Point Name** (e.g., “Fire Truck”). 4. Add an **Objective** describing what the avatar will explain or do at this point (e.g., “Describe the fire truck and its role in emergencies.”). 5. Repeat this process to build a full tour path. ### **4. Managing the Tour**
* To remove a tour point, click the gizmo and hit the **X icon**. * Under **User Elements**, click **“Set User Starting Point”** to define where the player begins. * Click **“Save Narrative Graph”** to save your tour configuration. * Use **Preview** to test the experience. * Click **Publish** when you're ready to share your tour. ## **Summary** Convai Sim’s **Tour Guide Mode** transforms your AI avatar into an interactive, narrative-driven host—ideal for: * Education & virtual field trips * Employee onboarding * Training simulations * Museum or product walkthroughs Once your tour is complete, you’re just one click away from publishing it across web, kiosks, and other platforms. # Publishing a Convai Sim Experience Learn how to finalize and publish your AI simulation or tour guide experience created with Convai Sim ## **Make Your Experience Live** Once you’ve finished building your AI simulation or virtual tour, Convai Sim makes it easy to publish and share your experience across platforms.
## Publishing Steps ### **1. Finalizing Your Experience** Fill in the necessary details to define and present your simulation: * **Experience Name**\ e.g., *Virtual Tour of the Fire Station* * **Experience Description**\ e.g., *Get a deeper look and understanding of the inner workings of a fire station with your virtual tour guide Lina!* * **Thumbnail (Optional)**\ Upload an image to visually represent your experience. ### 2. Choose Visibility Settings Select how and with whom the experience should be shared: * **Public** * Visible to everyone * Accessible on [**x.convai.com**](https://x.convai.com) * **Private** * Only visible to you and invited users * **Unlisted** * Not listed publicly, but can be accessed via a direct link * **Embed on Your Site** *(Enterprise-only)* * Publish your experience directly to your own website {% hint style="danger" %} **Convai Pixel Streaming Embed** is currently accessible only with the **Enterprise plan**.\ To learn how to embed an avatar into your own platform, check out the [Embedding Documentation](/api-docs/plugins-and-integrations/convai-pixel-streaming-embed). {% endhint %} *** ## **What Happens After Publishing?** Once published, your experience becomes: * **Accessible** to your intended audience * **Ready for interaction** via web, kiosk, or internal use * **Shareable** as a training tool, educational demo, or digital showcase Whether you're running a public-facing simulation or a private module for internal teams, **Convai Sim gives you complete control** over how your AI-driven experience is distributed. # Convai XR Animation Capture App Capture animations in VR using your Meta Quest and animate AI avatars—no mocap suit required. ## Introduction The **Convai XR Animation Capture** app allows you to record high-quality animations directly in virtual reality using a **Meta Quest** headset. These animations can be uploaded to your Convai account and used seamlessly across platforms like **Unity**, **Unreal Engine**, or within no-code tools like **Avatar Studio** and **Convai Sim**. {% embed url="" %} *** ## **What You Can Do** With the Convai XR Animation Capture App, you can: * Record natural **animations** in VR using your Meta Quest * Upload animations directly to your **Convai account** * Assign these animations to **AI avatars**, which perform them intelligently during conversation * Use animations in: * **Unity** * **Unreal Engine** * **Convai Sim** * **Avatar Studio** * Build **custom gesture libraries** and animation sets\ → All without the need for mocap suits or external trackers *** ## **Who It’s For** This app is ideal for: * **Developers & creators** building immersive and interactive characters * **Educators & trainers** crafting virtual learning environments * **Game designers** enhancing NPC realism in Unity or Unreal * **Brands & marketers** creating engaging virtual hosts with Avatar Studio * **Storytellers & world-builders** designing no-code simulations with Convai Sim Whether you’re building a virtual assistant, NPC, tour guide, or performer — XR Animation Capture helps you bring your AI characters to life with natural, human motion. *** ## Key Features * **VR-Based Animation** Record gestures, motions, and actions naturally with your Meta Quest headset. * **Direct Upload to Convai** Animations are automatically synced to your Convai account—no manual transfer needed. * **Cross-Platform Support** Use animations in Unity, Unreal Engine, Convai Sim, and Avatar Studio — no extra setup required. * **AI-Driven Animation Triggers** Let your avatars perform animations intelligently based on dialogue and context. * **No Mocap Suit Needed** Capture high-quality animation using just your VR headset — no external trackers or suits required. * **Works with No-Code Tools** Deploy intelligent, animated avatars directly in browser-based platforms like Avatar Studio and Convai Sim. # Convai XR Animation Capture App Setup Learn how to install and connect the Convai XR Animation Capture App on your Meta Quest headset to start recording avatar animations in VR.
## **Requirements** Before you begin, make sure you have the following: * **Meta Quest 2 / 3 / Pro** * A registered **Convai account** – [Sign up here](https://www.convai.com) * A stable **internet connection** *** ## **Installation Steps** ### Step 1: Install the App on your Quest device 1. Put on your Meta Quest headset. 2. Open the **Meta Quest Store**. 3. Search for **"Convai Animation Capture"**. 4. Select the app and click **Install**.
*** ### **Step 2: Log In to Your Convai Account** 1. Launch the **Convai Animation Capture** app on your headset. 2. When prompted, **log in** to your Convai account. *** #### You're Ready to Animate! After completing the steps above, your setup is complete. You can now begin recording animations directly in VR, which your AI avatars can intelligently perform in simulations, scenes, or guided experiences within Convai Sim.
# Creating Animations for AI Avatars Capture lifelike animations using your Meta Quest headset to bring your AI avatars to life—no mocap suit required. ## **Overview** Using the **Convai XR Animation Capture** app on your Meta Quest headset, you can create custom animations for your AI characters by simply acting them out in VR. These animations help your avatars express themselves naturally during conversations—whether in Unity, Unreal, Avatar Studio, or Convai Sim. {% hint style="warning" %} ***Haven’t set up the app yet?***\ Head over to the [Convai XR Animation Capture App Setup Guide](/api-docs/no-code-experiences/convai-xr-animation-capture-app/convai-xr-animation-capture-app-setup) before continuing. {% endhint %} *** ## **Recording Animations in VR** ### **Step 1: Review Existing Animations (Optional)** When you launch the app, you’ll see your **animation dashboard**. From here, you can: * View previously recorded animations * **Replay** them to see how they look * **Delete** any that you no longer need
*** ### **Step 2: Start Recording** 1. In the app, click **“Start Recording”**. 2. A five-second countdown will start. 3. Begin **performing your animation.** * Examples: wave, point, gesture, etc.
*** ### **Step 3: Stop & Review** 1. Once you're done, click **“Stop”**. 2. You can review the recorded animation by pressing the **"Replay Animation"** button. * This helps you decide whether to save, redo, or discard.
*** ### **Step 4: Name & Save** 1. Enter a **clear and descriptive name** (e.g., *Wave Greeting*, *Points Left*). 2. Click **“Save & Upload”**. The animation is now uploaded to your **Convai dashboard**, ready to be: * Assigned to AI avatars * Used across **Unity**, **Unreal**, **Avatar Studio**, or **Convai Sim**
*** ### **Keep Building Your Animation Library** Record and save multiple animations to populate your library. These can be reused across projects, allowing your avatars to **intelligently perform gestures** during conversations—making your virtual experiences more engaging and realistic. # Adding Your Recorded Animations to AI Avatars Inside Unity Learn how to import animations recorded in VR and apply them to your AI avatars in Unity. ## **Overview** Bring your Convai avatars to life inside Unity by integrating animations recorded via the **Convai XR Animation Capture App**. This guide walks you through importing those animations and attaching them to AI-powered characters in your Unity project. {% embed url="" %} ## **How to Add Recorded Animations to AI Avatars in Unity** ### **Step 1: Set Up Unity & Convai** Before importing animations, ensure your Unity project is correctly set up with Convai: * Install the **Convai Unity SDK**. * Retrieve your **API key** from the Convai Playground. * Add the API key to your project settings. {% hint style="success" %} **Need help setting up the SDK?**\ Check out our [Unity SDK Documentation](https://docs.convai.com/api-docs/plugins-and-integrations/unity-plugin) or follow the [video](https://youtu.be/anb9ityi0MQ) walkthrough above. {% endhint %} *** ### **Step 2: Import the Animation** 1. Go to the **Convai Dashboard** and navigate to the **Server Animations** tab. 2. Locate the animations you recorded in VR. 3. Click **Import** and select a location **within your Unity project's `Assets/` directory**. {% hint style="warning" %} The files **must be placed inside the Unity project folder** for them to be detected and used properly. {% endhint %} *** ### **Step 3: Apply the Animation to a Character** 1. In Unity, drag your AI character model into the **scene hierarchy**. 2. Adjust the character’s position if needed. 3. Open or create an **Animator Controller**. 4. Drag the imported animation clip into the **Animator Controller**. 5. If the animation should repeat, enable **Loop Time** in the Animation settings. *** ### **Step 4: Test the Scene** 1. Run your Unity scene. 2. Start a conversation with the avatar or trigger the assigned action. 3. Watch your AI avatar perform the recorded animation in real-time! *** #### **Done!** You’ve now successfully connected a custom VR-recorded animation to an AI-powered avatar in Unity.\ Repeat the process to add more animations and create rich, expressive characters in your simulations or games. # Convai Unity SDK Add real-time AI characters to Unity training simulations, interactive experiences, and games — with speech, emotion, actions, and persistent memory. The Convai Unity SDK connects Unity projects to Convai, bringing conversational AI into training simulations, interactive experiences, and games. Characters process speech in real time, respond with natural language, and synchronize lip movement, facial emotion, and body animation — all configurable per project. Each feature is a self-contained module: add only what your project needs, in the order that fits your workflow. {% hint style="info" %} **New to the SDK?** Follow [Getting Started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) to install the package, configure your API key, and run your first AI character. {% endhint %} ### Learn the SDK
Overview
What the SDK is, how it is structured, and when to use each module.		/pages/a3d7afb80b74f02afa29c790d79751f4f29227c8
Compatibility and requirements
Unity versions, render pipelines, platform support, and network requirements.		/pages/ijdGXp5AdxHr05KBoBHM
Core concepts
Session lifecycle, turn-taking modes, and the SDK event system.		/pages/e324eb9293dfc478fb2c227def03dca5d44cc512
### Features Each feature is a self-contained module you opt into.
Actions
Characters execute structured in-scene commands dispatched by Convai.		/pages/8e2d39906eb40512aa9c4c6b05ffc3352eb87e51
Emotion
Map Convai emotion signals to facial blendshapes or Animator parameters.		/pages/pzyAnwgRmeSToQBXFJYu
Long-term memory
Characters remember each player across separate sessions.		/pages/Yg6rMK5Mnh9zStBOwoca
Vision
Characters see through a Unity camera, webcam, or Meta Quest passthrough.		/pages/c776d88f01e63c32eee310e37d0ed10bc12eb355
Dynamic context
Inject runtime state and events into character knowledge at any time.		/pages/HLRHVibG6Nqvot1DGfvc
Narrative design
Trigger-based story section progression tied to conversation flow.		/pages/vN4t84KHA949ufTTNaVd
Scene metadata
Characters automatically read contextual information about scene objects.		/pages/a62866eb2b9826eb452e50eb8875192eb2464f3c
### Utilities Helper modules that run entirely within Unity — no Convai communication required.
Dialogue animation
Four-layer animator stack driving body and head movement during speech.		/pages/e7963b2cdbdecc024efd724f61855cea3fab1ea9
Gaze and attention
Eye and head gaze blended toward focus targets and conversation partners.		/pages/64f87fbb4fb6aa685a57fa969e6b5253c7ee2246
### Reference and guides
UI and presentation
Transcript UI, subtitle modes, notification system, and settings panel.		/pages/26837dafaf3d20063e1664f52a690823bbbb0eb7
Scripting reference
Full API reference for session events, character events, and the transcript system.		/pages/cd1efd3a5ca8009cafe359f0b008bc9d6038de38
Platform guides
WebGL, Android, iOS, and Meta Quest deployment guides.		/pages/d375c5c4e58f4a666f3cb8c940558eb222674e75
Advanced topics
Custom providers, performance, and SDK extension points.		/pages/50715c11049002775f940f15ab236a3bc207395c
Troubleshooting
Common failure modes, diagnostic steps, and known issues.		/pages/d5350da7bd6402bba0c66f41fe2dadd0116fb2da
### Latest release vspace.vars.unity\_sdk\_version introduces Structured Actions for precise in-scene command dispatch, Meta Quest passthrough vision for AR character awareness, runtime turn-taking mode switching, and expanded dynamic context for richer NPC knowledge. ### Next steps Install the Convai Unity SDK and connect your first AI character using the Getting Started section. If you are evaluating the SDK before installing, review [Compatibility and requirements](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements) to confirm platform and Unity version support. {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} # Convai Unity SDK overview Find explanations of what the Convai Unity SDK is, how its four-tier architecture is organized, and what feature modules ship with the package. The Convai Unity SDK layers voice, emotion, structured actions, and long-term memory onto Unity characters. The pages in this section explain the SDK fundamentals, system architecture, available features, and current release notes.
What is the Convai Unity SDK
End-to-end voice pipeline, target audience, and what the SDK ships with.		/pages/3d552fbc704f9580bbec1cc230e2eae41818f025
Convai Unity SDK architecture
Four-tier framework: Runtime, Room, Agent, and Module layers.		/pages/E0sHZRYpXvCTPJiLrAH4
Feature map
Find the right feature, tool, or guide for any development goal.		/pages/052c621eeaba9b20d93167f3c9981bf9dc20372f
Release notes
Release notes for vspace.vars.unity_sdk_version and previous releases.		/pages/ae8k7JCmNIvZyl8ctZ6K
### Next steps If you are ready to install the SDK, go directly to Getting Started. {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} # What is the Convai Unity SDK Real-time conversational AI characters for Unity — voice pipeline, opt-in feature modules, supported platforms, and minimum requirements. The Convai Unity SDK connects Unity characters to Convai so they can speak, listen, reason, and act in real time. A player speaks into a microphone; the SDK captures audio, streams it to Convai for speech recognition and language understanding, generates a response with text-to-speech, and plays it back on the character with synchronized lip sync, facial emotion, and optional in-scene actions. The SDK targets Unity developers building training simulations, interactive experiences, and games. ### What it includes The SDK ships with a complete conversation pipeline and a set of opt-in feature modules. #### Conversation pipeline Always active once connected: * **Real-time voice input** — microphone capture, streaming speech recognition * **Language understanding and generation** — Convai processes and responds in character * **Text-to-speech** — voice generated by Convai, played back through Unity audio #### Feature modules Opt-in, each added as a Unity component: * **Lip sync** — real-time blend shape mouth animation; supports ARKit, MetaHuman, and CC4 Extended maps * **Emotion** — maps Convai emotion signals to facial blend shapes or Animator parameters * **Actions** — character executes structured in-scene commands dispatched by Convai * **Long-term memory** — character remembers each player across separate sessions * **Narrative design** — trigger-based story section progression tied to conversation flow * **Vision** — character sees through a Unity camera, webcam, or Meta Quest passthrough * **Dynamic context** — inject runtime state and events into the character's knowledge at any time #### Utilities Optional helpers that run entirely in Unity without Convai communication: * **Dialogue animation** — four-layer animator stack driving body and head movement during speech * **Gaze and attention** — eye and head gaze blended toward focus targets and conversation partners #### Editor tooling Project Settings API key configuration, scene setup menu, Scene Validator, and custom Inspectors for every SDK component. ### Voice → Convai → Character flow ```mermaid graph LR MIC[Microphone] --> RM[ConvaiRoomManager] RM --> |Voice stream| CLOUD[Convai] CLOUD --> |Audio + metadata| CHAR[ConvaiCharacter] CHAR --> AUD[Audio playback] CHAR --> MOD[Feature modules] ``` `ConvaiRoomManager` handles the streaming connection to Convai. `ConvaiCharacter` receives the response — audio, transcript, emotion signals, and action commands — and routes each to the appropriate module or output. ### Requirements | Requirement | Minimum | | --------------- | -------------------------------------------------------------- | | Unity version | space.vars.unity\_min\_version | | Render pipeline | Built-in, URP, or HDRP | | Platform | Windows, macOS, Linux, Android, iOS, Meta Quest, WebGL | | Network | Internet connection to Convai | | API key | Free account at [convai.com](https://www.convai.com/) | {% hint style="info" %} The sample scenes use URP. If your project uses the Built-in render pipeline, the samples require minor material reassignment. The SDK itself works with all three pipelines. {% endhint %} The Convai Unity SDK is available on the [Unity Asset Store](https://assetstore.unity.com/packages/tools/behavior-ai/npc-ai-engine-dialog-actions-voice-and-lipsync-convai-235621). For the full platform and Unity version support matrix, see Compatibility and requirements. {% content-ref url="/pages/ijdGXp5AdxHr05KBoBHM" %} [Compatibility and requirements](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements) {% endcontent-ref %} ### Next steps Install the SDK and add your first character. {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} To understand the system architecture before setting up, see the architecture page next. {% content-ref url="/pages/E0sHZRYpXvCTPJiLrAH4" %} [Convai Unity SDK architecture](/api-docs/plugins-and-integrations/convai-unity-sdk/overview/convai-unity-sdk-architecture) {% endcontent-ref %} # Convai Unity SDK architecture Understand the Convai Unity SDK four-tier architecture — Runtime, Room, Agent, and Module — and the responsibilities of each tier. The SDK is organized into four tiers: Runtime, Room, Agent, and Module. As a developer, you interact primarily with the Agent tier (character and player components) and the Module layer (opt-in feature modules). The Runtime and Room tiers handle connection and service bootstrapping with minimal configuration required. ### System diagram ```mermaid graph TD subgraph RT["Runtime Tier"] CM["ConvaiManager\n(composition root, service hub)"] CS["ConvaiSDK\n(version metadata)"] end subgraph RM_TIER["Room Tier"] RM["ConvaiRoomManager\n(connection · audio · turn-taking)"] end subgraph AG["Agent Tier"] CC["ConvaiCharacter × N\n(session · transcripts · events)"] CP["ConvaiPlayer\n(identity · text input)"] end subgraph ML["Module Layer (opt-in per character)"] direction LR LS[LipSync] EM[Emotion] VI[Vision] NA[Narrative] DA[DialogueAnimation] FA[FacialAnimation] GA[Gaze] AT[Attention] CF[ConversationFlow] EB[Embodiment] end CM --> RM RM --> CC RM --> CP CC -.->|modules attach to| ML ``` The dashed line from `ConvaiCharacter` to the Module Layer means modules are optional components you add to the same GameObject as the character — not required for basic conversation. ### Runtime tier The Runtime tier boots when your scene loads and provides services to everything below it. `ConvaiSDK` is a static class that exposes the SDK version (`ConvaiSDK.Version`). You rarely reference it directly. `ConvaiManager` is the composition root. It is a singleton (`ConvaiManager.ActiveManager`) marked with `[DefaultExecutionOrder(-1100)]` so it initializes before other scene objects. It owns: * **Service accessors** — `TryGet` methods for every internal service (microphone, audio, agent registry, notification, settings panel, etc.) * **High-level facades** — `ConvaiManager.Audio`, `ConvaiManager.Transcripts`, `ConvaiManager.Events` for the most common scripting tasks * **Connection control** — `ConnectAsync()`, `DisconnectAsync()`, `SetConversationInputModeAsync()` * **Agent references** — `Characters`, `Player`, `ActiveConversationCharacter` {% hint style="info" %} Most integration code only needs `ConvaiManager.ActiveManager` and the character's own events. The `TryGet` service accessors are for advanced use cases where you replace or extend internal services. {% endhint %} ### Room tier `ConvaiRoomManager` owns the live connection to Convai. One `ConvaiRoomManager` per scene, managed by `ConvaiManager`. It is responsible for: * **Room connection lifecycle** — connecting, disconnecting, reconnecting * **Microphone capture** — starting and stopping audio input, mute control * **Turn-taking mode** — hands-free (`ConversationInputMode.HandsFree`) or push-to-talk (`ConversationInputMode.PushToTalk`) * **Dynamic context transport** — sending state updates and events to Convai at runtime * **Audio playback coordination** — enabling remote character audio, WebGL user-gesture handling `ConvaiRoomManager` exposes coordinators for diagnostics, audio, ownership, and connection management. These are accessible via `ConvaiManager.ActiveManager.TryGetRoomConnectionService()` for advanced scripting. ### Agent tier The Agent tier contains the components you place on scene GameObjects. #### ConvaiCharacter Add `ConvaiCharacter` to each NPC or agent GameObject. One component per character. It owns: * Character ID — the unique ID from your Convai dashboard * Session state — `Disconnected`, `Connecting`, `Connected`, `Reconnecting`, `Disconnecting`, `Error` * Conversation lifecycle — `StartConversationAsync()`, `StopConversationAsync()`, `ToggleSpeech()` * Transcript and event callbacks — `OnTranscriptReceived`, `OnEmotionChanged`, `OnActionsReceived`, `OnSpeechStarted`, `OnSpeechStopped`, `OnCharacterReady` * Action configuration — via `ConvaiActionConfigSource` component `ConvaiCharacter` can be configured inline in the Inspector or via a reusable `ConvaiCharacterProfile` ScriptableObject asset. #### ConvaiPlayer Add `ConvaiPlayer` to your player GameObject. One per scene is standard. It owns: * Player display name and name tag color for transcript attribution * Text message sending — `SendTextMessage(string message)` * Runtime identity override — `SetRuntimeDisplayName(string displayName)` {% hint style="warning" %} `ConvaiPlayer.PlayerId` is a local display identifier for transcript UI attribution. It is not the server-generated speaker ID used for Long-Term Memory tracking. {% endhint %} ### Module layer Modules are optional Unity components you add to the same GameObject as `ConvaiCharacter` (or `ConvaiRoomManager` for Vision). Each module is independent — add only what your project needs. | Module | What it does | | ----------------- | ---------------------------------------------------------------------------------------------------------------- | | LipSync | Real-time blend shape mouth animation driven by audio playback; supports ARKit, MetaHuman, and CC4 Extended maps | | Emotion | Receives Convai emotion signals, smooths them, and dispatches to blend shape or Animator parameter bindings | | Vision | Publishes camera, webcam, or Meta Quest passthrough frames to Convai for multimodal awareness | | Narrative | Manages story section progression through trigger-based events tied to conversation flow | | DialogueAnimation | Drives a four-layer animator stack (base idle, masked overlays, body talk, head talk) during dialogue | | FacialAnimation | Plays facial animation clips at runtime, composited against lip sync and emotion outputs | | Gaze | Blends eye and head actuators toward conversation partners and attention targets | | Attention | Resolves weighted focus targets, providing gaze direction to the Gaze module | | ConversationFlow | Bridges the conversation event stream to per-frame dialogue state (Idle, Speaking, Reacting, etc.) | | Embodiment | Foundational behavior profile and lifecycle management for physical presence and behavioral modules | `ConversationFlow` is provisioned automatically at runtime when a `ConvaiCharacter` initializes. All other modules — `LipSync`, `Emotion`, `Vision`, `Narrative`, `DialogueAnimation`, `FacialAnimation`, `Gaze`, `Attention`, and `Embodiment` — are opt-in. ### Configuration model Every major component supports two configuration modes, selectable in the Inspector. {% tabs %} {% tab title="Inline" %} Values are set directly on the component in the Inspector. This is the default mode and is suitable for most scenes — no additional assets required. {% endtab %} {% tab title="Asset" %} Values come from a reusable `ConvaiCharacterProfile` or `ConvaiRoomManagerProfile` ScriptableObject. Use this when you want shared defaults across multiple scenes or prefab variants, or when you need to swap character behavior without modifying individual prefabs. {% endtab %} {% endtabs %} ### Next steps {% content-ref url="/pages/052c621eeaba9b20d93167f3c9981bf9dc20372f" %} [Feature map](/api-docs/plugins-and-integrations/convai-unity-sdk/overview/feature-map) {% endcontent-ref %} {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} # Feature map Find the right Convai Unity SDK feature, module, guide, or reference page for any development goal, indexed by use case. Use this table when you know the outcome you want but are not sure which SDK feature, module, or guide covers it. ### Getting started | I want to... | Feature / Tool | Documentation | | ---------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | Install the SDK into my project | Installation | [Installation](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/installation) | | Configure my Convai API key | API key setup | [Configure API key](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-api-key) | | Add my first conversational character to a scene | Scene setup | [Build a custom scene](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/build-a-custom-scene) | | Run a working example without building from scratch | Sample scenes | [Import and run sample scenes](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/import-and-run-sample-scenes) | | Understand what each component in the scene does | Component reference | [Scene components reference](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/scene-components) | | Choose between push-to-talk and hands-free input | Input mode | [Configure conversation input mode](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-conversation-input-mode) | | Configure character audio output | Audio setup | [Configure character audio](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-character-audio) | | Configure microphone device and platform permissions | Audio setup | [Configure microphone](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-microphone) | | Add a chat or subtitle transcript display | Transcript UI | [Add chat UI](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-chat-ui) | | Add real-time lip sync to my character | Lip sync | [Add lip sync](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-lip-sync) | | Verify my scene is set up correctly before shipping | Scene Validator | [Validate your setup](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/validate-your-setup) | ### Features | I want to... | Feature | Documentation | | ----------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------- | | Let my character execute in-scene commands (trigger animations, open doors, move objects) | Actions | [Actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions) | | Inject runtime state or events into the character's knowledge | Dynamic Context | [Dynamic Context](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context) | | Let the character automatically read information about scene objects | Scene Metadata | [Scene Metadata](/api-docs/plugins-and-integrations/convai-unity-sdk/features/scene-metadata) | | Show facial emotion on my character driven by the AI response | Emotion | [Emotion](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion) | | Make the character remember players between sessions | Long-Term Memory | [Long-Term Memory](/api-docs/plugins-and-integrations/convai-unity-sdk/features/long-term-memory) | | Build branching story sections triggered by conversation | Narrative Design | [Narrative Design](/api-docs/plugins-and-integrations/convai-unity-sdk/features/narrative-design) | | Give my character vision through a camera or webcam | Vision | [Vision](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision) | ### Utilities | I want to... | Utility | Documentation | | ----------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------ | | Add body and head dialogue animations to my character | Dialogue Animation | [Dialogue Animation](/api-docs/plugins-and-integrations/convai-unity-sdk/utilities/dialogue-animation) | | Make my character look at targets, players, or points of interest | Gaze and Attention | [Gaze and Attention](/api-docs/plugins-and-integrations/convai-unity-sdk/utilities/gaze-and-attention) | ### UI and presentation | I want to... | Component | Documentation | | -------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Display a live conversation transcript in my UI | Transcript UI | [Transcript UI](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/transcript-ui) | | Switch between chat and subtitle display modes | Presentation modes | [Chat and subtitle modes](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/transcript-ui/chat-and-subtitle-modes) | | Query transcript history or react to transcript changes from code | Transcript history | [Transcript history and queries](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/transcript-ui/transcript-history-and-queries) | | Show in-world notification popups | Notification system | [Notification system](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/notification-system) | | Add a runtime settings panel (mic, transcript, notifications, input) | Settings panel | [Settings panel](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/settings-panel) | | Read or apply runtime settings from code | Runtime settings | [Runtime settings API](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/settings-panel/runtime-settings-api) | | Customize the look and layout of UI components | UI customization | [Customizing UI components](/api-docs/plugins-and-integrations/convai-unity-sdk/ui-and-presentation/customizing-ui-components) | ### Core concepts | I want to... | Concept | Documentation | | -------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------- | | Understand the system architecture end-to-end | Architecture | [Convai Unity SDK architecture](/api-docs/plugins-and-integrations/convai-unity-sdk/overview/convai-unity-sdk-architecture) | | Understand session states, reconnection, and persistence | Session lifecycle | [Session lifecycle](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/session-lifecycle) | | Compare hands-free, push-to-talk turn-taking | Turn-taking modes | [Turn-taking modes](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/turn-taking-modes) | | Subscribe to conversation events from C# or Inspector | Event system | [Event System](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/event-system) | ### Scripting reference | I want to... | API | Documentation | | ------------------------------------------------------------ | -------------------- | -------------------------------------------------------------------------------------------------------------- | | Subscribe to session connected / disconnected / error events | Session events | [Session Events](/api-docs/plugins-and-integrations/convai-unity-sdk/scripting-reference/session-events) | | Subscribe to transcript, emotion, and turn events | Character events | [Character Events](/api-docs/plugins-and-integrations/convai-unity-sdk/scripting-reference/character-events) | | Read and clear transcript history at runtime | Transcript API | [Transcript API](/api-docs/plugins-and-integrations/convai-unity-sdk/scripting-reference/transcript-api) | | Use the `ConvaiSDK` and `ConvaiAudio` static facades | Conversation facades | [ConvaiManager API](/api-docs/plugins-and-integrations/convai-unity-sdk/scripting-reference/convaimanager-api) | | Understand `IConvaiOperation` and async patterns | Async patterns | [Async Patterns](/api-docs/plugins-and-integrations/convai-unity-sdk/scripting-reference/async-patterns) | ### Platform guides | I want to... | Platform | Documentation | | ------------------------------------------ | --------------- | --------------------------------------------------------------------------------------------------------------- | | Ship to a browser with WebGL | WebGL | [WebGL](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/webgl) | | Ship to Android or iOS | Mobile | [Mobile — iOS and Android](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/ios-and-android) | | Ship to Meta Quest with passthrough vision | Meta Quest / XR | [Meta Quest and XR](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/xr-headsets) | ### Advanced topics | I want to... | Topic | Documentation | | ---------------------------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Replace the API credential provider | Custom providers | [Custom Credential Provider](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/custom-providers/custom-credential-provider) | | Replace the end-user identity provider | Custom providers | [Custom Identity Provider](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/custom-providers/custom-identity-provider) | | Replace the session persistence provider | Custom providers | [Custom Persistence Provider](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/custom-providers/custom-persistence-provider) | | Measure latency and interpret session metrics | Performance | [Performance and Optimization](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/performance-and-optimization) | | Extend the SDK with a custom module or replace internal services | Extension points | [Extending the SDK](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/extending-the-sdk) | ### Next steps Start with installation if you have not set up the SDK yet. {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} # Release notes Release notes for the Convai Unity SDK — current version highlights, previous release notes, bug fixes, and migration guidance for each release. Track changes to the Convai Unity SDK across releases, including new features, bug fixes, and configuration changes. {% updates format="full" %} {% update date="2026-05-08" tags="v4.2.0,Current" %} ## v4.2.0 **Actions System** Characters can execute in-scene commands through a structured runtime. New in this release: * `ConvaiActionDispatcher` with queued dispatch — actions execute in sequence without race conditions * `IConvaiActionExecutor` interface for custom executors * Six built-in executors: move-to (Transform and NavMesh), pick-up, look-at, Animator trigger, and UnityEvent * Inspector-driven configuration — no scripting required for standard action setups * Runtime diagnostics for monitoring action queue state **Meta Quest Passthrough Vision** `QuestVisionFrameSource` enables the Vision module on Quest 3 and Quest 3S devices without an external camera. Characters see through the device's passthrough camera during mixed reality sessions. **Runtime Turn-Taking Mode Switching** Switch between hands-free and push-to-talk modes during live sessions using `ConvaiManager.SetConversationInputModeAsync()` or the runtime Settings Panel — no scene reload required. **Dynamic Context Expansion** * Tracker APIs let you monitor current context state from scripts * Inspector tooling for authoring context commands without code * `SampleDynamicContextUI` prefab demonstrates runtime injection patterns **Scene Setup Tooling and Validation** Menu-driven component creation (**GameObject > Convai > Setup Required Components**) and scene validation (**GameObject > Convai > Validate Scene Setup**) prevent misconfiguration before entering Play mode. **Settings Panel Input Mode Control** The runtime Settings Panel now exposes input mode switching — players or developers can change turn-taking mode during a session without scripting. {% endupdate %} {% update date="2026-04-09" tags="v4.1.0" %} ## v4.1.0 * **Dynamic Context:** `ConvaiDynamicContextCommand` component enables runtime injection of state and events into character knowledge * **LipSync sample:** Removed camera dependency from the LipSync sample scene — works with any camera setup * **Vision module:** Reliability improvements to frame source lifecycle and reconnection handling * **iOS:** Fixed crash on first microphone access when `NSMicrophoneUsageDescription` was absent from build settings * **Sample scenes:** Refined setup and scene structure across Basic and LipSync samples * **Editor:** Startup time improvements for projects with large scene counts {% endupdate %} {% update date="2026-03-12" tags="v4.0.0,Initial Release" %} ## v4.0.0 Initial public release of the Convai Unity SDK. **Core components:** `ConvaiManager`, `ConvaiRoomManager`, `ConvaiCharacter`, `ConvaiPlayer` **Conversation pipeline:** Speech-to-text, language understanding and generation, text-to-speech — fully streamed in real time **Modules:** LipSync, Emotion, Vision, Narrative Design, Dynamic Context, Long-Term Memory, Scene Metadata, Dialogue Animation, Gaze and Attention **Platform support:** Windows, macOS, Linux, Android, iOS, WebGL **Editor tooling:** Scene setup, validation, Project Settings integration, and the Convai Welcome window {% endupdate %} {% endupdates %} ### Next steps To start using the SDK, follow Getting Started. {% content-ref url="/pages/1d9fa274483952ca377014dd2a4fb49c2f3e80a1" %} [Getting started](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started) {% endcontent-ref %} # Compatibility and requirements Find the Unity version, render pipeline, platform, and network requirements you need before installing the Convai Unity SDK. Confirm your environment meets these requirements before installing the Convai Unity SDK. The pages below cover Unity version support, render pipeline compatibility, platform-specific constraints, and network requirements.
Unity versions and render pipelines
Minimum Unity version, required packages, and render pipeline support.		/pages/d270f20a06743da76d0e46e64e17440ad758dad6
Platform support matrix
Feature availability across Windows, macOS, Android, iOS, Meta Quest, and WebGL.		/pages/e8f8a0a8257efa9a6ff3e80c02b6bdc1224d50fc
Network and API requirements
Domains, ports, and firewall rules needed for real-time SDK operation.		/pages/451345facd5cd16f0b40e81f6f7a06784935c82b
### Next steps Once you have confirmed compatibility, install the SDK. {% content-ref url="/pages/f1aef51364acf8a4bf5802a99774c033d9ee6e68" %} [Install the Convai Unity SDK](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/installation) {% endcontent-ref %} # Unity versions and render pipelines Reference for Convai Unity SDK environment requirements, including the minimum Unity version, required package dependencies, and render pipeline support. The Convai Unity SDK requires Unity 2023.1.1f1 or later. All three Unity render pipelines are supported with no additional configuration. Both installation methods — Package Manager and Asset Store — install required package dependencies automatically. ### Unity version requirements | Requirement | Version | | ---------------------------- | ---------------------------------------------------------------------- | | Minimum | space.vars.unity\_min\_version | | Recommended for new projects | space.vars.unity\_recommended\_version | {% hint style="warning" %} The Convai Unity SDK does not support Unity versions earlier than 2023.1.1f1. If your project is on an older LTS release, upgrade before installing. {% endhint %} ### Required package dependencies The SDK depends on three Unity packages. Both installation methods install these automatically — you do not need to add them manually unless you encounter a version conflict. | Package | Version | | --------------------------------- | ------------------------------------------------------------------------- | | `com.unity.nuget.newtonsoft-json` | space.vars.dep\_newtonsoft\_json\_version | | `com.unity.ugui` | space.vars.dep\_ugui\_version | | `com.unity.inputsystem` | space.vars.dep\_inputsystem\_version | {% hint style="warning" %} Do not downgrade these packages after installation. The SDK targets the versions listed above and behavior on lower versions is undefined. If your project already pins an older version of any of these in `Packages/manifest.json`, remove or update the pin before installing. {% endhint %} ### Render pipeline support The SDK runtime contains no pipeline-specific conditionals. All three Unity render pipelines are fully supported with no additional configuration required. | Render Pipeline | Supported | | -------------------------------------- | --------- | | Built-in Render Pipeline | ✅ Full | | Universal Render Pipeline (URP) | ✅ Full | | High Definition Render Pipeline (HDRP) | ✅ Full | {% hint style="info" %} The included sample scenes use URP materials. If your project uses the Built-in or HDRP pipeline, sample scene materials require reassignment. Optional depth-of-field camera scripts in `SamplesShared/Camera/` are also URP-specific and are not required for SDK functionality. {% endhint %} ### Next steps With your Unity version and packages confirmed, check which platforms you are targeting. {% content-ref url="/pages/e8f8a0a8257efa9a6ff3e80c02b6bdc1224d50fc" %} [Platform support matrix](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements/platform-support-matrix) {% endcontent-ref %} # Platform support matrix Reference for Convai Unity SDK platform support, including feature availability across Windows, macOS, Android, iOS, Meta Quest, and WebGL. The Convai Unity SDK runs on all major Unity deployment targets. Feature availability varies by platform — use the matrix below to confirm support before building for a specific target. ### Feature × platform matrix | Feature | Windows / macOS / Linux | Android | iOS | Meta Quest | WebGL | | -------------------------- | ----------------------- | ------------------------------ | -------------------------------------- | --------------------- | -------------------------------------- | | Voice conversation | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Microphone capture | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full — HTTPS + user gesture required | | Remote audio playback | ✅ Unity `AudioSource` | ✅ Unity `AudioSource` | ✅ Unity `AudioSource` | ✅ Unity `AudioSource` | ⚠️ Browser-routed | | Lip sync | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ⚠️ Known timing drift | | Spatial audio | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ❌ Not supported | | Actions | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Emotion | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Long-Term Memory | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Narrative Design | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Dynamic Context | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ✅ Full | | Vision — Camera | ✅ Full | ✅ Full | ✅ Full | ✅ Full | ⚠️ Canvas capture | | Vision — Webcam | ✅ Full | ⚠️ Runtime permission required | ⚠️ `NSCameraUsageDescription` required | ❌ Not applicable | ❌ Not supported | | Vision — Quest passthrough | ❌ Not supported | ❌ Not supported | ❌ Not supported | ✅ Full | ❌ Not supported | ### Platform-specific requirements {% tabs %} {% tab title="WebGL" %} WebGL is fully supported with the following constraints imposed by browser security policies: * **Microphone capture** requires HTTPS or `localhost`. HTTP deployments cannot access the microphone. Call `ConvaiManager.EnableAudioAndStartListening()` from a user gesture (button click) — do not start audio automatically on scene load. * **Remote audio playback** is routed through the browser's audio system, not Unity's `AudioSource`. Volume and spatialization controls on `AudioSource` components have no effect on WebGL. * **Vision — Camera** uses browser canvas capture (`CameraVisionFrameSource` is supported). * **Vision — Webcam** (`WebcamVisionFrameSource`) is not supported on WebGL — `AsyncGPUReadback` is unavailable in the browser runtime. Use `CameraVisionFrameSource` to stream the game canvas instead. * **Spatial audio** is not supported on WebGL. {% hint style="warning" %} WebGL has a known audio/lip-sync timing drift defect — audio and lip-sync data arrive correctly, but playback timing can drift in browser builds. This is a tracked defect, not a missing feature. Validate in your target browser before shipping. {% endhint %} {% hint style="info" %} Always validate WebGL builds in the actual hosting environment, especially if the build is embedded in an iframe. Add `allow="microphone"` to the iframe tag if you embed the build in a page you control. {% endhint %} For detailed WebGL setup, browser compatibility, and deployment steps, see the WebGL platform guide. {% content-ref url="/pages/7b2c74ff6faeac7d9557447799d0496b11e50783" %} [WebGL](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/webgl) {% endcontent-ref %} {% endtab %} {% tab title="Android" %} * **Microphone:** The SDK requests `RECORD_AUDIO` permission at runtime via `ConvaiPermissionService`. Declare the permission in your `AndroidManifest.xml` and handle both grant and denial cases in your app flow. * **Vision — Webcam:** `android.permission.CAMERA` is requested at runtime by `WebcamVisionFrameSource`. Handle permission grant and denial in your app flow. For Android build configuration, permission handling, and microphone setup, see the iOS and Android platform guide. {% content-ref url="/pages/32708cb5ab33fa2d1d0f91e747cc06da20220bc1" %} [iOS and Android](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/ios-and-android) {% endcontent-ref %} {% endtab %} {% tab title="iOS" %} * **Microphone:** `NSMicrophoneUsageDescription` must be set in **Player Settings → Other Settings → iOS → Microphone Usage Description**. Omitting this causes a crash on first microphone access. * **Vision — Webcam:** `NSCameraUsageDescription` must be set in **Player Settings → Other Settings → iOS → Camera Usage Description** if you use `WebcamVisionFrameSource`. On iOS, `WebcamVisionFrameSource` accesses the device camera via Unity's `WebCamTexture` API. * Define your app's behavior when the user denies microphone or camera permission, and when the app is interrupted or backgrounded during a conversation. For iOS build configuration, permission setup, and Info.plist requirements, see the iOS and Android platform guide. {% content-ref url="/pages/32708cb5ab33fa2d1d0f91e747cc06da20220bc1" %} [iOS and Android](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/ios-and-android) {% endcontent-ref %} {% endtab %} {% tab title="Meta Quest" %} Quest passthrough vision (`QuestVisionFrameSource`) is supported on **Quest 3 and Quest 3S only**. **Requirements:** * Meta XR SDK imported into your project * `PassthroughCameraAccess` component present in the scene The required passthrough camera permissions are declared automatically when Meta XR SDK is imported. On other Quest hardware or non-Quest platforms, `QuestVisionFrameSource` produces no frames. Use `CameraVisionFrameSource` or `WebcamVisionFrameSource` instead. `WebcamVisionFrameSource` is not applicable on Meta Quest because Quest does not expose a standard `WebCamTexture` device. For Meta Quest project setup, XR SDK configuration, and passthrough Vision integration, see the XR headsets platform guide. {% content-ref url="/pages/86442e40afdd1c7fd12eb8dc8331a0cc75827347" %} [XR headsets](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/xr-headsets) {% endcontent-ref %} {% endtab %} {% endtabs %} ### Next steps With platform constraints confirmed, review the network requirements for real-time SDK operation. {% content-ref url="/pages/451345facd5cd16f0b40e81f6f7a06784935c82b" %} [Network and API requirements](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements/network-and-api-requirements) {% endcontent-ref %} # Network and API requirements Reference for Convai Unity SDK network access, including Convai and LiveKit hosts, firewall rules, authentication, and log-based connection checks. The Convai Unity SDK requires outbound internet access during runtime. Speech, language understanding, and text-to-speech run through Convai over HTTPS and LiveKit WebRTC — there is no offline or LAN mode. Use this page when preparing a corporate network, validating a firewall allowlist, or confirming that a Play mode session reached the correct LiveKit room. ### Required outbound access Runtime sessions use two Convai endpoints plus LiveKit hosts returned in the connect response. Allow outbound traffic from the machine running Unity to the hosts below. #### Convai endpoints | Host | Port | Protocol | Purpose | | ------------------------------------------------------------ | ----- | -------- | --------------------------------------------------------- | | space.vars.live\_server\_url | `443` | HTTPS | Connect API — session setup and room credentials | | `api.convai.com` | `443` | HTTPS | Character metadata, REST API, and WebGL emotion preflight | #### LiveKit minimum requirements After Convai accepts the connect request, the SDK joins a LiveKit room using the `room_url` and `token` from the response. Minimum outbound rules for LiveKit Cloud connectivity: | Host | Port | Protocol | Purpose | | -------------------------------------------- | ----- | -------- | ------------------------------------- | | `convai-technologies-lfslae7c.livekit.cloud` | `443` | TCP | Convai LiveKit signaling endpoint | | `*.livekit.cloud` | `443` | TCP | LiveKit WebSocket signaling | | `*.turn.livekit.cloud` | `443` | TCP | TURN/TLS fallback when UDP is blocked | #### Optional direct media paths LiveKit documents these paths for direct WebRTC media and TCP fallback. Add them when your network policy permits UDP or when the LiveKit connection test shows that the minimum TCP rules are not enough. | Host | Port | Protocol | Purpose | | ---------------------- | --------------- | -------- | ---------------------- | | `*.host.livekit.cloud` | `3478` | UDP | TURN/UDP connectivity | | All LiveKit hosts | `50000`–`60000` | UDP | WebRTC media transport | | All LiveKit hosts | `7881` | TCP | WebRTC TCP fallback | No inbound ports are required on the client machine. For the full LiveKit firewall reference, see [Configuring firewalls](https://docs.livekit.io/deploy/admin/firewall/) in the LiveKit documentation. ### How realtime sessions connect Each character session follows this sequence: ```mermaid sequenceDiagram participant Unity as Unity SDK participant Convai as Convai participant LiveKit as LiveKit room Unity->>Convai: POST /connect with API key Convai-->>Unity: session_id, room_url, room_name, token Unity->>LiveKit: Join room at room_url with token LiveKit-->>Unity: Audio, video, and data streams ``` | Connect response field | Runtime use | | ---------------------- | ------------------------------------------------------------------------------------------- | | `session_id` | Convai session ID for the live session | | `character_session_id` | Conversation continuity across reconnects | | `room_url` | LiveKit WebSocket endpoint — for example `wss://convai-technologies-lfslae7c.livekit.cloud` | | `room_name` | LiveKit room name the client joins | | `token` | Temporary LiveKit room token used to join the room | The SDK reads `room_url` from the connect response and passes it to the transport layer. It does not hardcode a LiveKit hostname. The example host `convai-technologies-lfslae7c.livekit.cloud` is one Convai deployment; your logs may show a different `room_url` for the same account over time. ### Authentication Two credential types are involved in a runtime session. #### API key Your project API key authenticates requests to Convai. Store it in **Edit > Project Settings > Convai SDK**. | Request type | Header | Used for | | ----------------------- | ---------------- | ---------------------------------------------------------- | | Connect (`/connect`) | `X-API-Key` | Starting a realtime session | | REST (`api.convai.com`) | `CONVAI-API-KEY` | Character metadata, long-term memory, and other REST calls | The SDK sends the API key automatically. You do not paste the key into connect payloads manually. #### LiveKit room token Each successful connect response includes a temporary `token`. The SDK passes this token to LiveKit when joining the room. The token is short-lived and grants access only to that room. Treat it like a password — do not share it in public forums, support tickets, or version control. ### Find connection details in logs When a session starts, the SDK logs room credentials at **Debug** level under the **Transport** category. Enable verbose transport logging before searching for these lines. 1. Open **Edit > Project Settings > Convai SDK > Diagnostics**. 2. Set **Global Log Level** to `Debug`, or expand **Category Overrides** and add an override with **Category** `Transport` and **Level** `Debug`. 3. Enter Play mode and start a character session. 4. Open the Unity Console and filter by `Transport`. You should see one of these log forms: ``` [Debug][Transport]: Room details received: {"token":"...","room_name":"...","session_id":"...","room_url":"..."} ``` If the JSON line is hard to read, search for `Token:` — the SDK also prints a readable summary: ``` [Debug][Transport]: Token: ...; Room Name: ...; Room URL: ...; Session ID: ...; Character Session ID: ... ``` | Log field | Meaning | | ---------------------- | ------------------------------------------------------- | | `token` | LiveKit room token | | `room_name` | LiveKit room name / room ID | | `room_url` | LiveKit URL / WebSocket endpoint | | `session_id` | Convai session ID | | `Character Session ID` | Convai character session ID for conversation continuity | {% hint style="danger" %} Do not share the room token publicly. It is temporary and grants access to the LiveKit room. Redact `token` values before posting logs to support channels or public issue trackers. {% endhint %} Debug log calls compile out of non-development builds unless `CONVAI_DEBUG_LOGGING` is defined. Debug transport lines appear in the Unity Editor and Development Builds by default. See [Debug tools reference](/api-docs/plugins-and-integrations/convai-unity-sdk/troubleshooting/debug-tools-reference) for logging configuration. ### Test the LiveKit connection Use the [LiveKit connection test](https://livekit.com/webrtc/connection-test) when you need to confirm whether a restricted network can reach the room returned by Convai. 1. Start a Unity session and find the `Token:` log line. 2. Copy the `Room URL` value into **LiveKit URL**. 3. Copy the `Token` value into **Room Token**. 4. Select **Run test**. | Connection test field | Value from Convai logs | | --------------------- | ----------------------- | | **LiveKit URL** | `room_url` / `Room URL` | | **Room Token** | `token` / `Token` | The test validates LiveKit signaling and WebRTC connectivity for that temporary room. It does not validate your Convai API key, `api.convai.com`, or character configuration. ### Firewall validation checklist Work through this checklist with your network or IT team before deploying to a restricted environment. 1. Confirm outbound TCP `443` to space.vars.live\_server\_url and `api.convai.com`. 2. Confirm outbound TCP `443` to `convai-technologies-lfslae7c.livekit.cloud`, `*.livekit.cloud`, and `*.turn.livekit.cloud`. 3. When UDP is permitted, allow UDP `3478` to `*.host.livekit.cloud` and outbound UDP `50000`–`60000`. 4. Exclude Convai and LiveKit hostnames from TLS inspection if your proxy performs man-in-the-middle decryption on HTTPS or WSS traffic. 5. Enter Play mode, enable **Transport** logs at **Debug**, and confirm a `Room details received` or `Token:` line appears with a `wss://` `room_url`. 6. Confirm the character reaches **Connected** state and responds to voice or text input. If steps 5 or 6 fail with `transport.ice_failed` or `transport.signal_disconnected`, see [Connection and API issues](/api-docs/plugins-and-integrations/convai-unity-sdk/troubleshooting/connection-and-api-issues). ### Troubleshooting | Symptom | Likely cause | Fix | Verify | | ------------------------------------------------------------ | --------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | `connection.network_error` or `connection.timeout` | Convai endpoints blocked on TCP `443` | Allow space.vars.live\_server\_url and `api.convai.com` | Session reaches `Connected` state | | `transport.ice_failed` or `transport.peer_connection_failed` | LiveKit UDP or TURN hosts blocked | Add the LiveKit minimum rules and optional direct media paths from this page | `Room details received` log appears; character responds in Play mode | | No `Room details received` log line | Transport logging below **Debug** | Open **Diagnostics** in Project Settings; set **Transport** category override to **Debug** | Readable `Token:` line appears in the Console | | LiveKit connection test fails after connect succeeds | UDP media or TURN fallback path blocked | Allow UDP `50000`–`60000`, UDP `3478` to `*.host.livekit.cloud`, and TCP `443` to `*.turn.livekit.cloud` | LiveKit connection test succeeds with the logged `Room URL` and `Token` | | `connection.connect_invalid_api_key` | Invalid or revoked API key | Copy a fresh key from space.vars.dashboard\_url into Project Settings | Connect error no longer fires | | WebGL mic unavailable | Build served over HTTP | Serve the build over HTTPS or from `localhost` | Microphone permission prompt appears in the browser | {% hint style="warning" %} Proxy servers that perform TLS inspection on HTTPS or WSS traffic can break LiveKit signaling. Exclude Convai and LiveKit hostnames from TLS inspection when your environment uses a corporate proxy. {% endhint %} ### WebGL deployments WebGL builds must be served over HTTPS or from `localhost`. HTTP deployments block microphone access due to browser security policy. WebGL uses the same Convai and LiveKit endpoints as desktop builds. See [Platform support matrix](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements/platform-support-matrix) and the [WebGL deployment guide](/api-docs/plugins-and-integrations/convai-unity-sdk/platform-guides/webgl) for browser-specific constraints. ### Next steps With network requirements confirmed, install the SDK or review connection error codes. {% content-ref url="/pages/f1aef51364acf8a4bf5802a99774c033d9ee6e68" %} [Install the Convai Unity SDK](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/installation) {% endcontent-ref %} {% content-ref url="/pages/bf2d3c0310bc8496a310e6c0331136dd3cf2399d" %} [Connection and API issues](/api-docs/plugins-and-integrations/convai-unity-sdk/troubleshooting/connection-and-api-issues) {% endcontent-ref %} # Getting started Step-by-step path from installing the Convai Unity SDK to a validated, working conversational AI character in your scene. By the end of this section, your project contains a responsive AI character that listens, processes speech through Convai, and responds in real time. Follow the pages in order — each step builds directly on the previous one. {% hint style="info" %} **Before you begin:** Confirm your environment meets the requirements on the [Prerequisites](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/prerequisites) page — Unity version, required packages, and a Convai account with an API key. {% endhint %} ### Preparation
Prerequisites
Unity version, required packages, and account requirements before installing.		/pages/9c9a13d2a745ff9a78da0c1790759ba17484a304
Installation
Add the Convai Unity SDK to your project via the Package Manager or Asset Store.		/pages/f1aef51364acf8a4bf5802a99774c033d9ee6e68
### Scene setup
Configure API key
Connect your project to Convai with your account API key.		/pages/nLYDmLXfx19u9jY8DiYD
Import and run sample scenes
Verify the SDK works before building your own scene.		/pages/41c4fa4e32b4314bedf2eeac4548c536e758ce3f
Scene components reference
Learn what ConvaiManager, ConvaiCharacter, and ConvaiPlayer do and how they depend on each other.		/pages/0ovsiZNbY51a9fHPtneG
Build a custom scene
Build a Convai scene from scratch using the Setup Required Components command.		/pages/KM3jUMFCMu4M0nS1MSJH
Validate your setup
Run the scene validator to confirm required components are present before adding features.		/pages/35a808a76b0d30cee351c5b850b110058a2eb840
Configure conversation input mode
Choose between hands-free and push-to-talk input.		/pages/2532db82ab7e792207320a0655ef32ccfd5f99e4
Configure character audio
Tune NPC voice volume, spatial audio, and mute controls.		/pages/MGgUuRj5tRGxy8aDaXHd
Configure microphone
Select the active microphone device and set up platform permissions for Android, iOS, and WebGL.		/pages/vzAH1RAH4ciuh2ypEwaO
Add chat UI
Display conversation transcripts in your scene.		/pages/3b61dcb812f9a6f38643f1fce0883e718828e8d1
Add lip sync
Drive character blendshapes in sync with voice audio.		/pages/96d3c58b6fd5623574443837c77ebcb98c1071e6
### Next steps Once you have validated your setup, explore the Features section to add Actions, Emotion, Long-Term Memory, or Vision to your characters. {% content-ref url="/pages/d5302389270319f4ca9bf82b6b3081d22b81bd8f" %} [Features](/api-docs/plugins-and-integrations/convai-unity-sdk/features) {% endcontent-ref %} Review Core Concepts for a deeper understanding of the session lifecycle and event system before building further. {% content-ref url="/pages/e324eb9293dfc478fb2c227def03dca5d44cc512" %} [Core concepts](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts) {% endcontent-ref %} # Prerequisites System requirements, Unity version, required packages, and account prerequisites for the Convai Unity SDK. Before installing the Convai Unity SDK, confirm that your environment meets the requirements below. Missing any of these will cause installation errors or runtime failures that are harder to diagnose after the fact. ### System requirements | Requirement | Minimum | | ------------------- | -------------------------------------------------------------- | | Unity version | space.vars.unity\_min\_version | | Scripting backend | Mono or IL2CPP | | Internet connection | Required at editor time and runtime | {% hint style="warning" %} The Convai Unity SDK requires Unity space.vars.unity\_min\_version. If your project is on an older LTS release, upgrade before proceeding. {% endhint %} ### Required Unity packages The SDK depends on three Unity packages. Both installation methods (Package Manager and Asset Store) install these automatically — you do not need to add them manually unless you encounter a version conflict. | Package | Minimum Version | | --------------------------------- | --------------- | | `com.unity.nuget.newtonsoft-json` | 3.2.2 | | `com.unity.ugui` | 2.0.0 | | `com.unity.inputsystem` | 1.18.0 | If your project already pins `com.unity.inputsystem` or `com.unity.ugui` to an older version in `Packages/manifest.json`, the automatic install will fail silently or produce a version conflict. Remove or update the existing version pins before installing the SDK. ### Supported render pipelines | Render Pipeline | Supported | | -------------------------------------- | --------- | | Built-in Render Pipeline | ✓ | | Universal Render Pipeline (URP) | ✓ | | High Definition Render Pipeline (HDRP) | ✓ | For detailed platform and render pipeline compatibility, see [Compatibility & Requirements](/api-docs/plugins-and-integrations/convai-unity-sdk/compatibility-and-requirements). ### Account requirements You need an active Convai account and an API key to connect your project to Convai. 1. Create an account at [convai.com](https://convai.com) if you do not have one. 2. Retrieve your API key from the **API Keys** section of the Convai dashboard. 3. Create at least one character in the Convai dashboard and note its **Character ID** — you will need it during scene setup. Your API key is stored in `Assets/Resources/ConvaiSettings.asset`. If your project uses source control, decide whether to commit this file based on your team's security policy. See [Configure the API key](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-api-key) for full setup steps and secure-deployment options. ### Next steps Once your environment meets all requirements above, install the SDK. {% content-ref url="/pages/f1aef51364acf8a4bf5802a99774c033d9ee6e68" %} [Install the Convai Unity SDK](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/installation) {% endcontent-ref %} # Install the Convai Unity SDK Add the Convai Unity SDK to your Unity 2023.1+ project via the Package Manager or Asset Store. Both methods install the identical package. The Convai Unity SDK is available through two channels. Use **Package Manager** for new projects or when you prefer not to manage Asset Store downloads — the package resolves directly from the Convai registry with no manual download. Use **Asset Store** if your project already sources packages from your Asset Store library or if your studio manages package versions through My Assets. Both methods install SDK version space.vars.unity\_sdk\_version and the same three required dependencies. {% tabs %} {% tab title="Package Manager" %} {% stepper %} {% step %} #### Open Package Manager In the Unity Editor menu bar, select **Window > Package Manager**. The Package Manager window opens. Confirm you are connected to the internet before proceeding. {% endstep %} {% step %} #### Add package by name Click the **+** button in the top-left corner of the Package Manager window. Select **Add package by name** from the dropdown. A text field appears prompting for the package name. {% endstep %} {% step %} #### Enter the package name Type or paste the following identifier into the Name field, then click **Add**: ``` com.convai.convai-sdk-for-unity ``` Unity contacts the registry, resolves the package, and begins downloading. Three dependencies install automatically: | Package | Version | | --------------------------------- | ------- | | `com.unity.nuget.newtonsoft-json` | 3.2.2 | | `com.unity.ugui` | 2.0.0 | | `com.unity.inputsystem` | 1.18.0 | Wait for the progress bar in the bottom-right of the Unity Editor to complete before continuing. {% endstep %} {% endstepper %} {% hint style="warning" %} **Dependency conflict:** If your project already pins `com.unity.inputsystem` or `com.unity.ugui` to an older version in `Packages/manifest.json`, the install will fail or produce a version mismatch. Open `Packages/manifest.json`, remove or update the conflicting version entries, then retry. {% endhint %} {% hint style="success" %} **Installation complete** when the Convai SDK for Unity entry appears in the Package Manager list with a green checkmark and version space.vars.unity\_sdk\_version. You will also see a new **Convai** menu item in the Unity menu bar. {% endhint %} {% endtab %} {% tab title="Asset Store" %} {% stepper %} {% step %} #### Add the SDK to your Asset Store account Open the [Unity Asset Store](https://assetstore.unity.com/) in your browser. Search for **Convai SDK for Unity** and open the listing. Click **Add to My Assets**, signing in with your Unity ID if prompted. The button changes to **Open in Unity** when the asset has been added to your account. {% endstep %} {% step %} #### Open Package Manager In the Unity Editor menu bar, select **Window > Package Manager**. {% endstep %} {% step %} #### Switch to My Assets In the Package Manager window, click the packages source dropdown in the top-left (it shows **Packages: In Project** or similar) and select **My Assets**. Your Asset Store library loads. Locate **Convai SDK for Unity** in the list. {% endstep %} {% step %} #### Download and import Select **Convai SDK for Unity** in the list. Click **Download**, then click **Import** once the download completes. Unity imports the package and installs three dependencies automatically: | Package | Version | | --------------------------------- | ------- | | `com.unity.nuget.newtonsoft-json` | 3.2.2 | | `com.unity.ugui` | 2.0.0 | | `com.unity.inputsystem` | 1.18.0 | Wait for the progress bar in the bottom-right of the Unity Editor to complete before continuing. {% endstep %} {% endstepper %} {% hint style="info" %} To update the SDK to a newer version, return to **My Assets** in the Package Manager, select the SDK, and click **Update**. {% endhint %} {% hint style="success" %} **Installation complete** when the Convai SDK for Unity entry appears in the Package Manager list with version space.vars.unity\_sdk\_version. You will also see a new **Convai** menu item in the Unity menu bar. {% endhint %} {% endtab %} {% endtabs %} ### Next steps With the SDK installed, connect your project to Convai by entering your API key. {% content-ref url="/pages/nLYDmLXfx19u9jY8DiYD" %} [Configure the API key](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-api-key) {% endcontent-ref %} # Configure the API key Enter your Convai API key in the SDK settings to authenticate your project and enable character communication. The Convai SDK for Unity authenticates every session using an API key tied to your Convai account. You enter this key once in the Unity Editor — it is stored in your project and used automatically at runtime. {% stepper %} {% step %} #### Copy your API key Log in to your Convai dashboard at [convai.com](https://convai.com), navigate to **Account Settings**, and copy your API key. {% endstep %} {% step %} #### Open the Convai configuration window In the Unity Editor menu bar, open **Convai > Account**. The Convai Configuration window opens to the Account section. {% endstep %} {% step %} #### Paste the API key Paste your API key into the **API Key** field. The key is saved immediately to `Assets/Resources/ConvaiSettings.asset`. No additional save step is required. {% endstep %} {% step %} #### Verify the key is accepted Run **GameObject > Convai > Validate Scene Setup**. A missing API key appears as a warning in the validator dialog. If no API key warning appears, your key is configured correctly.
{% endstep %} {% endstepper %} {% hint style="warning" %} `Assets/Resources/ConvaiSettings.asset` stores the API key in plain text. If your project uses source control, decide whether to commit this file based on your team's security policy. {% endhint %} ### How the key is used at runtime The SDK reads the key from `ConvaiSettings` via the `ICredentialProvider` interface before initiating any connection to Convai. You do not need to pass the key manually in code — the SDK resolves it automatically on startup. For production deployments where storing the key in source control is not acceptable, implement a custom `ICredentialProvider` that reads the key from a secure source. {% content-ref url="/pages/31f40f584b5fb6b136f239a3d28d2ec74eb938e7" %} [Custom credential provider](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/custom-providers/custom-credential-provider) {% endcontent-ref %} ### Next steps With your API key in place, import a sample scene to verify the SDK is working before you build your own scene. {% content-ref url="/pages/41c4fa4e32b4314bedf2eeac4548c536e758ce3f" %} [Import and run sample scenes](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/import-and-run-sample-scenes) {% endcontent-ref %} # Import and run sample scenes Import the bundled sample scenes and verify the SDK is installed and connected correctly before building your own scene. The Convai SDK for Unity ships with two sample scenes. Running one is the fastest way to confirm your installation, API key, and audio setup are working before you build your own scene. | Sample | Description | | ------------------ | ---------------------------------------------- | | **Basic Sample** | Basic conversation with a robot character | | **LipSync Sample** | High-quality character with real-time lip sync | The steps for locating the samples differ depending on how you installed the SDK. {% tabs %} {% tab title="Package Manager" %} {% stepper %} {% step %} #### Open Package Manager In the Unity Editor, open **Window > Package Manager**. In the top-left dropdown, select **In Project**, then select **Convai SDK for Unity** from the list. {% endstep %} {% step %} #### Import a sample In the detail panel on the right, click the **Samples** tab. Click **Import** next to the sample you want to run. Unity copies the sample assets into `Assets/Samples/Convai SDK for Unity//`. A new folder appears under `Assets/Samples/` in the Project window. {% endstep %} {% step %} #### Open the scene In the Project window, navigate to the imported sample folder and open its `.unity` scene file. {% endstep %} {% endstepper %} {% endtab %} {% tab title="Asset Store" %} {% stepper %} {% step %} #### Locate the samples When installed via the Asset Store, all sample scenes are imported into your project automatically. In the Project window, navigate to: ``` Assets/Convai SDK For Unity/Samples/ ``` Two folders are present: `BasicSample` and `LipSyncSample`. {% endstep %} {% step %} #### Open the scene Open the `.unity` scene file inside the sample folder you want to run. {% endstep %} {% endstepper %} {% endtab %} {% endtabs %} Once the scene is open: {% stepper %} {% step %} #### Enter Play Mode Press **Play**. The Unity Console logs the following lines as the SDK initializes: * `[ConvaiRuntime] Started successfully` — SDK initialized all internal services * `[RoomConnectionRuntimeAdapter] Character connected successfully (mode=create).` — character connected to Convai Speak into your microphone. The character responds with voice and text output. If no response appears and the Console shows warnings, check [Validate your setup](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/validate-your-setup) for a diagnostic checklist. {% endstep %} {% endstepper %} ### Sample render pipeline notes Sample scenes are built with the Universal Render Pipeline (URP). If your project uses a different render pipeline, materials may appear pink after import. Convert the materials to match your active render pipeline using **Edit > Rendering > Materials**, then select the appropriate conversion option for your pipeline. ### Next steps Now that you have confirmed the SDK is working, learn what each scene component does before building your own setup. {% content-ref url="/pages/0ovsiZNbY51a9fHPtneG" %} [Scene components reference](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/scene-components) {% endcontent-ref %} # Scene components reference Learn the role of ConvaiManager, ConvaiRoomManager, ConvaiCharacter, and ConvaiPlayer, and how they depend on each other. Every Convai-powered scene is built from four core components. Understanding what each one does and how they relate to each other makes building and debugging your setup straightforward. ### Component overview The diagram below shows how the components depend on each other at runtime. ```mermaid graph TD A[ConvaiManager] --> B[ConvaiRoomManager] A --> C[ConvaiPlayer] A --> D[ConvaiCharacter] D --> E[ConvaiAudioOutput] E --> F[AudioSource] ``` `ConvaiManager` is the root. It bootstraps the SDK, manages the room connection through `ConvaiRoomManager`, and owns references to all `ConvaiCharacter` and `ConvaiPlayer` instances in the scene. ### ConvaiManager `ConvaiManager` is the SDK's entry point. It must be present in every scene that uses Convai. It initializes all internal services and injects dependencies into the other components automatically. **Add it:** Use **GameObject > Convai > Setup Required Components** to add `ConvaiManager` and its companion components in one step. Do not add it manually via Add Component — the wizard ensures the correct setup. **Key behavior:** * Singleton. Only one `ConvaiManager` may exist per scene. * Runs at execution order `-1100` — it initializes before all other components. * Auto-discovers `ConvaiCharacter` and `ConvaiPlayer` instances in the scene on startup. * Injects dependencies into discovered components automatically when `_autoInject` is enabled (default: on). **Useful properties at runtime:** | Property | Type | Description | | ----------------------------- | -------------------------------- | -------------------------------------- | | `IsBootstrapped` | `bool` | SDK internal services are initialized | | `IsInitialized` | `bool` | Bootstrap complete and event hub ready | | `IsConnected` | `bool` | Room connection is active | | `Characters` | `IReadOnlyList` | All characters owned by this manager | | `Player` | `ConvaiPlayer` | The player component in this scene | | `ActiveConversationCharacter` | `ConvaiCharacter` | Currently active conversation target | ### ConvaiRoomManager `ConvaiRoomManager` manages the connection lifecycle between your scene and Convai. It handles connecting, disconnecting, and reconnecting the audio session. It lives on the same GameObject as `ConvaiManager`. **Key behavior:** * Auto-connects on `Start()` when `ConnectOnStart` is `true` (default: `true`). * Reconnects automatically on transient failures, up to `_maxReconnectAttempts` (default: `3`). * Manages the microphone — starts capturing audio `_autoMicStartDelaySeconds` (default: `0.5s`) after the connection is established. **Inspector fields:** | Field | Default | Description | | --------------------------- | ----------- | ---------------------------------------------------------------- | | `ConnectOnStart` | `true` | Connect to Convai automatically when the scene starts | | `_connectionType` | `Audio` | `Audio` for voice-only; `AudioVideo` to also send camera frames | | `_pushToTalkKey` | `KeyCode.T` | Keyboard key used for push-to-talk input mode | | `_maxReconnectAttempts` | `3` | Attempts before giving up on reconnection | | `_autoMicStartDelaySeconds` | `0.5` | Seconds to wait after connection before opening the microphone | | `_roomRejoinTtlSeconds` | `60` | Seconds after disconnect during which the session can be resumed | Turn-taking settings are also configured here. See [Configure conversation input mode](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-conversation-input-mode). ### ConvaiCharacter `ConvaiCharacter` represents one AI character in your scene. Each NPC or virtual instructor that talks to players needs its own `ConvaiCharacter` component. Multiple characters are fully supported. The SDK connects to one character at a time — when the player addresses a different character, the session switches to that character automatically. **The Character ID field is required.** Get this value from your character's profile on the [Convai dashboard](https://convai.com). **Inspector fields:** | Field | Default | Description | | ------------------------------- | --------- | --------------------------------------------------------------- | | `_characterId` | *(empty)* | **Required.** Unique ID from your Convai dashboard | | `_characterName` | *(empty)* | Display name shown in transcripts and logs | | `_nameTagColor` | White | Color used to identify this character in the transcript UI | | `_autoConnect` | `false` | Start a conversation immediately after the scene loads | | `_enableRemoteAudio` | `true` | Play back the character's voice audio | | `_enableSessionResume` | `false` | Resume the previous session on reconnect | | `_characterReadyTimeoutSeconds` | `30` | Seconds to wait for the character-ready signal (0 = no timeout) | **Useful properties at runtime:** | Property | Type | Description | | -------------------- | -------------- | ------------------------------------------------------------- | | `IsCharacterReady` | `bool` | Character has received the ready signal from Convai | | `IsSessionConnected` | `bool` | Connected to the room (ready signal may not have arrived yet) | | `IsInConversation` | `bool` | Connected and ready — true conversation state | | `IsSpeaking` | `bool` | Character is currently outputting audio | | `SessionState` | `SessionState` | Full connection state enum | **Component dependencies:** `ConvaiAudioOutput` (handles audio playback for this character) must be on the same GameObject. `ConvaiAudioOutput` requires an `AudioSource` on the same GameObject. ### ConvaiPlayer `ConvaiPlayer` identifies the user in the conversation. It provides the player's display name and color to the transcript UI and lets Convai attribute player speech to the correct participant. **One `ConvaiPlayer` per scene.** Multiple player components in the same scene are not supported. **Inspector fields:** | Field | Default | Description | | --------------- | ---------- | --------------------------------------------------------------- | | `_playerName` | `"Player"` | Display name shown in the transcript UI | | `_playerId` | *(empty)* | Local ID for transcript attribution (empty = use `_playerName`) | | `_nameTagColor` | Green | Color used to identify the player in the transcript UI | {% hint style="info" %} `PlayerId` is a local display identifier for the transcript UI only. It is not the server-generated speaker ID used for Long-Term Memory. The server-assigned speaker ID is resolved after connection and is not set manually. {% endhint %} **Useful methods:** ```csharp // Override display name at runtime (for example, after a player logs in) GetComponent().SetRuntimeDisplayName("Dr. Reyes"); // Set both name and ID together GetComponent().Configure("Dr. Reyes", "user-123"); ``` ### Optional components #### ConvaiAudioOutput Handles audio playback for a single character. Add it to the same GameObject as `ConvaiCharacter`. Requires an `AudioSource` on the same GameObject. | Field | Default | Description | | -------------- | ------- | ---------------------------------- | | `Volume` | `1.0` | Playback volume (0–1) | | `IsMuted` | `false` | Mute this character's audio output | | `_use3DAudio` | `true` | Enable spatial (3D) audio | | `_minDistance` | `1` | Spatial audio minimum distance | | `_maxDistance` | `50` | Spatial audio maximum distance | #### ConvaiSceneConfig An optional ScriptableObject (**Assets > Create > Convai > Scene Config**) that lets you define character IDs, prefabs, and auto-connect behavior in a reusable asset rather than inline in the Inspector. Useful for managing multiple characters across scenes. See Advanced Topics for full details. ### Next steps Now that you understand the components, build your own scene from scratch. {% content-ref url="/pages/KM3jUMFCMu4M0nS1MSJH" %} [Build a custom scene](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/build-a-custom-scene) {% endcontent-ref %} # Build a custom scene Add required Convai components to a new Unity scene using the Setup Required Components command and configure your first character. This page walks you through setting up a new scene with a Convai AI character from scratch. By the end, your scene will have the minimum required components for a character to receive voice input and respond. ### Minimum required hierarchy Every working Convai scene needs these three things: ``` [Manager GameObject] → ConvaiManager + ConvaiRoomManager [NPC GameObject] → ConvaiCharacter + ConvaiAudioOutput + AudioSource [Player GameObject] → ConvaiPlayer ``` The setup wizard creates the first and third automatically. You add the NPC components yourself. {% stepper %} {% step %} #### Add the required manager components In the Unity Editor menu bar, select **GameObject > Convai > Setup Required Components**. Unity creates a **ConvaiManager** GameObject with `ConvaiManager` and `ConvaiRoomManager` attached, and a **ConvaiPlayer** GameObject with `ConvaiPlayer` attached. Both appear in the Hierarchy. `ConvaiRoomManager` always lives on the same GameObject as `ConvaiManager`. Do not move it to a separate GameObject. {% endstep %} {% step %} #### Add ConvaiCharacter to your NPC In the Hierarchy, select the NPC GameObject you want to make conversational. In the Inspector, click **Add Component** and add `ConvaiCharacter`. {% endstep %} {% step %} #### Add AudioSource and ConvaiAudioOutput On the same NPC GameObject, add `AudioSource`, then add `ConvaiAudioOutput`. All three components — `ConvaiCharacter`, `ConvaiAudioOutput`, and `AudioSource` — should now appear on the same GameObject in the Inspector. {% endstep %} {% step %} #### Set the Character ID In the `ConvaiCharacter` component, set the **Character ID** field to the ID of your character from the [Convai dashboard](https://convai.com). {% hint style="warning" %} The Character ID field is required. If it is empty, the character cannot connect to Convai and the Scene Validator will report an error. {% endhint %} {% endstep %} {% step %} #### Validate the scene In the menu bar, select **GameObject > Convai > Validate Scene Setup**. A dialog appears listing errors, warnings, and recommended next steps. **Errors (must fix):** | Error | Fix | | ------------------------------------- | -------------------------------------------- | | No `ConvaiManager` found | Run **Setup Required Components** | | No `ConvaiCharacter` found | Add `ConvaiCharacter` to your NPC GameObject | | `ConvaiCharacter` has no Character ID | Set the Character ID from your dashboard | | No `ConvaiPlayer` found | Run **Setup Required Components** | **Warnings:** | Warning | Fix | | ---------------------- | -------------------------------------------- | | API key not configured | Open **Convai > Account** and enter your key | When the validator reports no errors, the scene is ready for Play Mode. {% endstep %} {% step %} #### Enter Play Mode Press **Play**. The Unity Console logs: * `[ConvaiRuntime] Started successfully` — SDK initialized * `[RoomConnectionRuntimeAdapter] Character connected successfully (mode=create).` — character connected to Convai Speak into your microphone. The character responds within a few seconds. {% endstep %} {% endstepper %} ### Usage examples #### Example 1: Safety training simulation **Scenario:** An industrial safety trainer NPC responds to trainee questions about equipment procedures. **Setup:** * NPC GameObject: `SafetyTrainer` with `ConvaiCharacter`, `ConvaiAudioOutput`, `AudioSource` * Character ID: ID of your safety trainer character from the Convai dashboard * `ConvaiCharacter._characterName`: `"Safety Trainer"` * `ConvaiCharacter._enableRemoteAudio`: `true` **Expected outcome:** Trainees speak to the NPC and receive voice responses about safety procedures. The character name appears in the transcript UI. #### Example 2: Multiple characters in one scene **Scenario:** A medical training simulation with two characters — a supervising doctor and a nurse. **Setup:** * Two separate NPC GameObjects, each with `ConvaiCharacter`, `ConvaiAudioOutput`, `AudioSource` * Each `ConvaiCharacter` has its own unique Character ID * Only one `ConvaiManager` and one `ConvaiPlayer` in the scene **Expected outcome:** Both characters are discovered and registered automatically. Conversation switches between them based on which character the player addresses. Character A and Character B do not share conversation context unless your Convai character configuration explicitly links them. ### Next steps With the scene built, run the validator to confirm everything is wired correctly before adding features. {% content-ref url="/pages/35a808a76b0d30cee351c5b850b110058a2eb840" %} [Validate your setup](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/validate-your-setup) {% endcontent-ref %} # Validate your setup Run the scene validation tool to confirm all required components are present, connected, and configured correctly. Before deploying or sharing your scene, run the SDK's built-in validator and verify the Play Mode startup sequence. This page covers every check the validator performs, the expected console output on success, and a troubleshooting table for common failures. ### Run the scene validator The Scene Validator inspects your scene for missing components, empty required fields, and common misconfigurations. Run it at any point during development — not just at the end. In the Unity Editor menu bar, select **GameObject > Convai > Validate Scene Setup**. A dialog appears with a list of **Errors** (must fix), **Warnings** (recommended), and **Next Steps** (suggested actions). ### Validator checks #### Errors — must fix These prevent the scene from connecting to Convai. | Error | Cause | Fix | | ------------------------------------- | ------------------------ | ------------------------------------------------------- | | No `ConvaiManager` found in scene | SDK is not initialized | Run **GameObject > Convai > Setup Required Components** | | No `ConvaiCharacter` found in scene | No characters registered | Add `ConvaiCharacter` to your NPC GameObject | | `ConvaiCharacter` has no Character ID | Required field is empty | Enter the Character ID from your Convai dashboard | | No `ConvaiPlayer` found in scene | Player component missing | Run **GameObject > Convai > Setup Required Components** | #### Warnings — recommended These do not block connection but may affect functionality. | Warning | Cause | Fix | | -------------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------ | | API key not configured | `ConvaiSettings.HasApiKey` returns false | Open **Convai > Account** and enter your API key | | Video mode active but no vision source found | `_connectionType` is `AudioVideo` but no `IVisionFrameSource` component exists | Add a frame source component or switch to `Audio` mode | {% hint style="success" %} When the validator shows zero errors and zero warnings, your scene is ready for Play Mode. {% endhint %} ### Play mode startup checklist After the validator passes, enter Play Mode and watch the Console for these log lines in order. * [ ] `[ConvaiRuntime] Started successfully` — SDK initialized all internal services * [ ] `[RoomConnectionRuntimeAdapter] Character connected successfully (mode=create).` — character connected to Convai * [ ] `[ChatTranscriptUI] Dependencies injected via explicit initialization` — transcript UI connected (if present) * [ ] Character `IsCharacterReady` becomes `true` within 30 seconds — Convai has acknowledged the character {% hint style="info" %} The character-ready signal may arrive 2–10 seconds after the room connects, depending on server load. If it does not arrive within `_characterReadyTimeoutSeconds` (default: 30s), the SDK logs a timeout warning. {% endhint %} To check `IsCharacterReady` at runtime: ```csharp void Start() { var character = FindFirstObjectByType(); character.OnCharacterReady += () => Debug.Log("Character is ready to converse."); } ``` ### Troubleshooting | Symptom | Likely cause | Fix | | ----------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[ConvaiRuntime] Started successfully` not in Console | `ConvaiManager` missing or failed to bootstrap | Check that `ConvaiManager` is in the scene. Look for earlier errors in the Console. | | Room never connects — no character-connected log | API key invalid or missing; network issue | Verify your API key in **Convai > Account**. Check firewall rules allow WebSocket/HTTPS to `live.convai.com`. | | `[ChatTranscriptUI] Dependencies not injected...` | `ConvaiManager` not found at UI startup | Ensure `ConvaiManager` is in the scene. Its execution order (-1100) guarantees it runs first. | | Character `IsCharacterReady` stays `false` | Character ID is wrong or character does not exist on your account | Verify the Character ID matches exactly what is shown on your Convai dashboard. | | Mic never opens — character hears nothing | Push-to-talk mode is on and mic starts muted | In `ConvaiRoomManager`, confirm **Mode** is `HandsFree`, or press **T** if using push-to-talk. | | Character voice plays but blendshapes do not animate | `ConvaiLipSyncComponent` not configured or profile ID mismatch | Add `ConvaiLipSyncComponent` to the character. Verify `_lockedProfileId` matches your character's transport format. Assign target `SkinnedMeshRenderer`(s). | | Materials appear pink in sample scenes | Render pipeline mismatch (Built-in vs URP) | Convert materials via **Edit > Rendering > Materials > Convert All Built-in Materials to URP**, or reassign URP shaders manually. | ### Setup complete Your scene now has: * The SDK installed and connected to Convai with a valid API key * A scene with `ConvaiManager`, `ConvaiRoomManager`, `ConvaiCharacter`, and `ConvaiPlayer` * Validator passing with zero errors * A character that connects, becomes ready, and responds to voice input ### Next steps Continue the getting started path to configure input mode, audio, and UI. {% content-ref url="/pages/2532db82ab7e792207320a0655ef32ccfd5f99e4" %} [Configure conversation input mode](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-conversation-input-mode) {% endcontent-ref %} Or explore the Features section to add Actions, Emotion, Long-Term Memory, or Vision to your characters. {% content-ref url="/pages/d5302389270319f4ca9bf82b6b3081d22b81bd8f" %} [Features](/api-docs/plugins-and-integrations/convai-unity-sdk/features) {% endcontent-ref %} Review Core Concepts for a deeper understanding of the session lifecycle and event system. {% content-ref url="/pages/e324eb9293dfc478fb2c227def03dca5d44cc512" %} [Core concepts](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts) {% endcontent-ref %} # Configure conversation input mode Choose between hands-free voice activation and push-to-talk, and configure the trigger key or button for your project. The Convai SDK for Unity supports two conversation input modes: **Hands Free** (the player speaks naturally, the SDK detects when they stop) and **Push to Talk** (the player holds a key to speak). Both modes are configured on `ConvaiRoomManager` in the Inspector. ### Where to find the settings Select the `ConvaiManager` GameObject in the Hierarchy. In the Inspector, find `ConvaiRoomManager`. The **Turn-Taking Options** section contains all input mode settings.
### Input mode comparison | | Hands Free | Push to Talk | | ----------------- | ------------------------------------------- | ------------------------------------------------ | | **How it works** | SDK detects end-of-speech automatically | Player holds a key to speak, releases to send | | **Best for** | Natural conversation, kiosk experiences, VR | Noisy environments, multiplayer, precise control | | **Latency** | Slightly higher (silence detection delay) | Lower (send on key release) | | **Player effort** | None | Must hold a key | ### Hands free mode Hands Free is the default. Set **Mode** to `HandsFree`. #### Turn detection Control how the SDK decides the player has finished speaking. | Setting | Default | Description | | --------------- | ------------ | ---------------------------------------------------------------------------------------- | | `TurnDetection` | `UseDefault` | `UseDefault` = server default, `Disabled` = always-on stream, `Custom` = configure below | When `TurnDetection` is set to `Custom`, the **Smart Turn Settings** appear: | Setting | Default | Description | | ----------------- | ------- | ---------------------------------------------------- | | `StopSecs` | `3.0` | Seconds of silence before the turn ends | | `MaxDurationSecs` | `8.0` | Maximum turn length before forced end | | `PreSpeechMs` | `0` | Milliseconds of audio before speech onset to include | {% hint style="info" %} Increasing `StopSecs` gives players more time to pause mid-sentence without triggering a turn end. Useful for training simulations where learners think before answering. {% endhint %} ### Push to talk mode Set **Mode** to `PushToTalk`. The default key is **T** — change it via `_pushToTalkKey` on `ConvaiRoomManager`. #### Local audio policy Controls microphone behavior on the player's device. | Setting | Default | Description | | -------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------- | | `StartMutedInPushToTalk` | `true` | Microphone starts muted; activates on key press | | `EnableAcousticEchoCancellation` | `false` | Enable AEC for speakerphone use (Android/iOS) | | `PushToTalkStartupMode` | `PrewarmMuted` | `PrewarmMuted` = mic open but muted from start; `OpenOnFirstPress` = mic opens only when key is first pressed | #### Push to talk policy Controls what happens when the player presses and releases the push-to-talk key. | Setting | Default | Description | | -------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `InterruptBotOnPress` | `true` | Pressing the key while the character is speaking interrupts it immediately | | `EnableServerSttToggle` | `true` | Pauses Convai's speech-to-text on the server while the player is not holding the key. Reduces server processing cost; disable if you observe recognition delays on key press. | | `RequireTurnCompletionBeforeNextPress` | `true` | Player must wait for the character to finish before speaking again | | `TurnCompletionTimeoutMs` | `5000` | Fallback timeout (ms) to unlock push-to-talk if the completion event never arrives | | `AllowSpeechStoppedFallbackAfterSpeechStart` | `false` | Allow a speech-stopped event to clear the waiting state after speech has started | ### Runtime mode switching `SetConversationInputModeAsync()` switches the active input mode for the current connected session — **no reconnection required**. The switch takes effect immediately on the live session and does not mutate configured defaults or room profile assets. ```csharp using System.Threading; using Convai.Runtime.Components; using Convai.Runtime.Room; using UnityEngine; public sealed class InputModeSwitcher : MonoBehaviour { // Switch to Hands Free — call from a UI button or game event public async void SwitchToHandsFree() { await ConvaiManager.ActiveManager .SetConversationInputModeAsync(ConversationInputMode.HandsFree, CancellationToken.None) .AsTask(); } // Switch to Push to Talk — call from a UI button or game event public async void SwitchToPushToTalk() { await ConvaiManager.ActiveManager .SetConversationInputModeAsync(ConversationInputMode.PushToTalk, CancellationToken.None) .AsTask(); } } ``` To read the current active mode or react to changes: ```csharp using Convai.Runtime.Room; // Read current mode ConversationInputMode current = ConvaiManager.ActiveManager.ActiveConversationInputMode; // Subscribe to changes via the room connection service if (ConvaiManager.ActiveManager.TryGetRoomConnectionService(out IConvaiRoomConnectionService roomService)) roomService.ConversationInputModeChanged += OnModeChanged; void OnModeChanged(ConversationInputMode newMode) { // Update UI, analytics, tutorial prompts, etc. } ``` {% hint style="warning" %} `SetConversationInputModeAsync()` is valid only while the room is actively **Connected**. Calls made while the room is `Disconnected`, `Connecting`, `Reconnecting`, or `Disconnecting` fail with `SessionErrorCodes.SessionInvalidState`. Check `ConvaiManager.IsConnected` before calling. {% endhint %} Connect-time `TurnTakingOptions` define the session's baseline policy (custom turn detection thresholds, push-to-talk startup behavior, AEC preference). Runtime switching changes only the active mode — all other options carry over from the connected session's configuration. ### Usage examples #### Example 1: Medical training — hands free with extended silence **Scenario:** Nursing students answer scenario questions. They often pause while thinking, so the default 3-second silence threshold causes premature turn ends. **Setup in Inspector:** * Mode: `HandsFree` * TurnDetection: `Custom` * StopSecs: `5.0` * MaxDurationSecs: `30.0` **Expected outcome:** Students can pause for up to 5 seconds mid-answer without the turn ending. The character waits until the student finishes. #### Example 2: Industrial site inspection — push to talk **Scenario:** Workers in a noisy manufacturing environment use push-to-talk to avoid accidental voice activation. They press **T** to ask questions about equipment status. **Setup in Inspector:** * Mode: `PushToTalk` * `_pushToTalkKey` on ConvaiRoomManager: `KeyCode.T` * `InterruptBotOnPress`: `true` (workers can cut off a long response to ask a follow-up) * `EnableAcousticEchoCancellation`: `true` (machine noise present) **Expected outcome:** Only intentional key presses send audio to Convai. Background noise does not trigger responses. Workers can interrupt long answers with a new press. #### Example 3: Cinematic to gameplay mode switch **Scenario:** An onboarding cinematic uses Hands Free. When gameplay starts, the game switches to Push to Talk without reloading the scene. ```csharp public async void OnCinematicEnd() { if (ConvaiManager.ActiveManager.IsConnected) { await ConvaiManager.ActiveManager .SetConversationInputModeAsync(ConversationInputMode.PushToTalk, CancellationToken.None) .AsTask(); } } ``` **Expected outcome:** Mode switches seamlessly mid-session. The character continues without interruption. Push-to-talk controls become active immediately. ### Next steps With input mode configured, tune character voice volume and audio playback settings. {% content-ref url="/pages/MGgUuRj5tRGxy8aDaXHd" %} [Configure character audio](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-character-audio) {% endcontent-ref %} # Configure character audio Set character voice volume, enable spatial audio, and control NPC playback through the ConvaiAudio facade for scripted mute and unmute. The `ConvaiAudioOutput` component controls how a character's voice plays back in the scene. Pair it with the `ConvaiAudio` facade on `ConvaiManager` for scripted runtime control of mute state, per-character volume, and audio events. ### Character audio output Add `ConvaiAudioOutput` to the same GameObject as `ConvaiCharacter`. An `AudioSource` on the same GameObject is required. **Inspector fields:** | Field | Default | Description | | -------------- | ------- | ----------------------------------------- | | `Volume` | `1.0` | Playback volume (0–1) | | `IsMuted` | `false` | Mute this character's audio output | | `_use3DAudio` | `true` | Enable Unity spatial audio | | `_minDistance` | `1` | Distance at which audio is at full volume | | `_maxDistance` | `50` | Distance at which audio falls to zero | Disable `_use3DAudio` for non-positional scenarios — for example, a kiosk interface where the character always sounds "present" regardless of where the player stands. ### Audio facade For scripted audio control, use the `ConvaiAudio` facade accessed through `ConvaiManager.Audio`. This is the recommended API for runtime audio management. #### Microphone control ```csharp // Mute/unmute the local microphone ConvaiManager.ActiveManager.Audio.SetMicMuted(true); // Toggle and get the new state bool isMuted = ConvaiManager.ActiveManager.Audio.ToggleMicMuted(); // Start microphone capture manually (if ConnectOnStart is false) await ConvaiManager.ActiveManager.Audio.StartListeningAsync(); ``` #### Per-character playback control ```csharp string characterId = character.CharacterId; // Mute a specific character ConvaiManager.ActiveManager.Audio.MuteCharacter(characterId); // Unmute ConvaiManager.ActiveManager.Audio.UnmuteCharacter(characterId); // Check mute state bool muted = ConvaiManager.ActiveManager.Audio.IsCharacterMuted(characterId); // Disable remote audio entirely for a character ConvaiManager.ActiveManager.Audio.SetRemoteAudioEnabled(characterId, false); ``` #### Audio events ```csharp void OnEnable() { ConvaiManager.ActiveManager.Audio.OnMicMuteChanged += HandleMicMuteChanged; } void OnDisable() { ConvaiManager.ActiveManager.Audio.OnMicMuteChanged -= HandleMicMuteChanged; } void HandleMicMuteChanged(bool isMuted) { muteButton.SetIsOnWithoutNotify(isMuted); } ``` ### Usage examples #### Example 1: Mute toggle UI button **Scenario:** A corporate onboarding simulation includes a mic mute button in the corner of the screen. ```csharp public class MuteButtonController : MonoBehaviour { [SerializeField] private Toggle _muteToggle; void OnEnable() { _muteToggle.onValueChanged.AddListener(OnMuteToggled); ConvaiManager.ActiveManager.Audio.OnMicMuteChanged += OnMicMuteChanged; } void OnDisable() { _muteToggle.onValueChanged.RemoveListener(OnMuteToggled); ConvaiManager.ActiveManager.Audio.OnMicMuteChanged -= OnMicMuteChanged; } void OnMuteToggled(bool muted) => ConvaiManager.ActiveManager.Audio.SetMicMuted(muted); void OnMicMuteChanged(bool muted) => _muteToggle.SetIsOnWithoutNotify(muted); } ``` **Expected outcome:** The toggle stays in sync with the actual microphone state. Pressing it mutes or unmutes the mic. External changes (for example, from push-to-talk logic) also update the toggle automatically. #### Example 2: Per-character volume in a multi-instructor scene **Scenario:** A language learning simulation has two AI instructors — a main teacher and a conversation partner. Players can independently adjust their volumes. ```csharp public class CharacterVolumeController : MonoBehaviour { [SerializeField] private ConvaiCharacter _character; [SerializeField] private Slider _volumeSlider; void Start() { var audioOutput = _character.GetComponent(); _volumeSlider.value = audioOutput.Volume; _volumeSlider.onValueChanged.AddListener(v => audioOutput.Volume = v); } } ``` **Expected outcome:** Each character's volume slider controls only that character's `AudioSource` volume. The two characters can be heard at different levels independently. ### Next steps Configure the microphone device and platform-specific audio permissions. {% content-ref url="/pages/vzAH1RAH4ciuh2ypEwaO" %} [Configure microphone](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-microphone) {% endcontent-ref %} # Configure microphone Select the active microphone device and configure platform-specific permissions for Android, iOS, and WebGL builds. The Convai SDK for Unity opens the system microphone automatically when a session starts. This page covers how to enumerate and select a specific device at runtime, and how to satisfy the platform-specific requirements for Android, iOS, and WebGL. ### Microphone device selection To list available microphone devices and let the player choose one: ```csharp using Convai.Runtime.Settings; // Get the microphone device service from the SDK if (ConvaiManager.ActiveManager.TryGetMicrophoneDeviceService(out IMicrophoneDeviceService micService)) { // List all available devices IReadOnlyList devices = micService.GetAvailableDevices(); foreach (var device in devices) { Debug.Log($"{device.Name} (ID: {device.Id}, Index: {device.Index})"); } // Start listening with a specific device index await ConvaiManager.ActiveManager.Audio.StartListeningAsync(microphoneIndex: device.Index); } ``` {% hint style="warning" %} On WebGL, `GetAvailableDevices()` returns an empty list outside the Editor. Microphone access on WebGL goes through the browser's Web Audio API and does not support Unity's native device enumeration. {% endhint %} ### Platform-specific setup #### Android The SDK requests microphone permission at runtime automatically when a recording starts. You must declare the permission in your `AndroidManifest.xml`: ```xml ``` If your project does not have a custom manifest, create one or enable **Override Default Manifest** in **Player Settings > Publishing Settings**. When the SDK requests the permission, Android shows its standard permission dialog. If the player grants it, recording starts automatically. If denied, the SDK logs a warning and the microphone remains inactive. #### iOS Add a microphone usage description to your `Info.plist`. In Unity, set this via **Player Settings > iOS > Other Settings > Microphone Usage Description**: ``` "This app uses the microphone to support voice conversations with AI characters." ``` The SDK requests authorization automatically using Unity's `Application.RequestUserAuthorization`. The app does not need to call any permission API directly. {% hint style="danger" %} Submitting to the App Store without a microphone usage description will cause App Store review rejection. Set this value before building for iOS distribution. {% endhint %} #### WebGL Browsers block audio playback and microphone access until the user has interacted with the page. The SDK provides two methods depending on your use case: * **`Audio.EnableAudioPlayback()`** — unlocks browser audio output only. Use this when you want character voice to play but are not yet starting the microphone (for example, during a tutorial before the player speaks). * **`ConvaiManager.EnableAudioAndStartListening()`** — unlocks browser audio output **and** opens the microphone. Use this when the player is ready to begin a full conversation. ```csharp // Call from a UI button's onClick event public void OnStartButtonClicked() { // Option A: unlock audio only (no microphone yet) if (ConvaiManager.ActiveManager.Audio.RequiresUserGesture) { ConvaiManager.ActiveManager.Audio.EnableAudioPlayback(); } // Option B: unlock audio and open microphone in one step // ConvaiManager.ActiveManager.EnableAudioAndStartListening(); } ``` If you skip this step on WebGL, the character's voice will not play even though Convai sends audio data. `RequiresUserGesture` returns `true` only on WebGL. ### Next steps With audio configured, add a transcript UI to display conversation text. {% content-ref url="/pages/3b61dcb812f9a6f38643f1fce0883e718828e8d1" %} [Add chat UI](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-chat-ui) {% endcontent-ref %} # Add chat UI Add a transcript UI component to display conversation text on screen during character interactions. The Convai SDK for Unity includes a ready-made transcript UI prefab that displays conversation text in real time. The prefab includes its own Canvas — drag it into the scene, and the UI connects to the SDK automatically. ### Transcript display modes The SDK supports three presentation modes for conversation text. | Mode | Identifier | Description | | ------------------ | ------------------ | --------------------------------------------------------------------------- | | **Chat** | `"Chat"` | Scrolling message bubbles — separate entries for player and character turns | | **Subtitle** | `"Subtitle"` | Single text line at the bottom of the screen, replaced each turn | | **QuestionAnswer** | `"QuestionAnswer"` | Split display — question above, answer below | ### Add the chat UI prefab {% stepper %} {% step %} #### Locate the prefab In the Project window, navigate to: ``` Packages/Convai SDK for Unity/Prefabs/TranscriptUI/TranscriptUI_Chat.prefab ``` {% endstep %} {% step %} #### Drag the prefab into the scene Drag `TranscriptUI_Chat.prefab` into the Hierarchy. The prefab includes its own Canvas — no separate Canvas setup is required. The chat UI overlay appears in the Game view. The component finds `ConvaiManager` automatically when the scene starts — no manual wiring is needed. {% hint style="warning" %} The chat input field requires an **EventSystem** in the scene. If your scene does not already have one, add it via **GameObject > UI > Event System**. {% endhint %} {% hint style="warning" %} If no `ConvaiManager` is found at startup, the Console logs: `[ChatTranscriptUI] Dependencies not injected - ensure ConvaiManager is present in scene`. Check that `ConvaiManager` is in the scene. {% endhint %} {% endstep %} {% endstepper %} ### ChatTranscriptUI Inspector fields If you need to customize the layout, select the prefab instance and inspect the `ChatTranscriptUI` component. **UI References:** | Field | Type | Description | | ------------------------ | ---------------- | ------------------------------------------------------- | | `scrollRect` | `ScrollRect` | The scroll container for the message list | | `chatContainer` | `RectTransform` | Parent transform where message bubbles are instantiated | | `characterMessagePrefab` | `GameObject` | Template for character speech bubbles | | `playerMessagePrefab` | `GameObject` | Template for player speech bubbles | | `chatInputField` | `TMP_InputField` | Text field for typed input (optional) | **Fade Settings:** | Field | Default | Description | | -------------- | ------- | ----------------------------------- | | `fadeDuration` | `0.5` | Seconds for fade-in/out transitions | {% hint style="warning" %} If `chatContainer` is not assigned, messages will not appear and the Console logs: `[ChatTranscriptUI] chatContainer is not assigned - messages will not display`. The bundled prefab has all references pre-wired. {% endhint %} ### Usage examples #### Example 1: Full-screen chat overlay in a corporate training simulation **Scenario:** A corporate onboarding experience displays a full-screen chat history so trainees can review everything the AI mentor said. **Setup:** * Drag `TranscriptUI_Chat.prefab` into the Hierarchy * Leave all references at prefab defaults **Expected outcome:** Each turn appears as a new bubble — player text on the right, character text on the left. The list scrolls automatically as the conversation grows. ### Next steps With the transcript UI in place, add lip sync to drive character blendshapes from audio. {% content-ref url="/pages/96d3c58b6fd5623574443837c77ebcb98c1071e6" %} [Add lip sync](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-lip-sync) {% endcontent-ref %} # Add lip sync Connect Convai audio output to your character's facial blendshapes to synchronize mouth movement with speech. The Convai SDK for Unity includes a real-time lip sync system that drives `SkinnedMeshRenderer` blendshapes in sync with the character's voice audio. It supports three industry-standard blendshape formats and handles playback buffering, smoothing, and fade-out automatically. ### How it works When Convai sends voice audio, it also streams a sequence of blendshape frames in the character's transport format (ARKit, MetaHuman, or CC4 Extended). The SDK buffers and interpolates these frames, applies optional smoothing, and writes the result to your character's `SkinnedMeshRenderer` every frame. ```mermaid graph LR A[Convai: voice + blendshape frames] --> B[ConvaiLipSyncComponent] B --> C[Frame buffer + interpolation] C --> D[Smoothing] D --> E[SkinnedMeshRenderer blendshapes] ``` ### Quick setup {% stepper %} {% step %} #### Add the component Add `ConvaiLipSyncComponent` to the same GameObject as your `ConvaiCharacter` (or to any child GameObject). {% endstep %} {% step %} #### Set the profile ID In the Inspector, set **Locked Profile ID** to the transport format your character uses: * `arkit` — Apple ARKit (61 blendshapes) * `metahuman` — Unreal MetaHuman (275+ blendshapes) * `cc4extended` — Character Creator 4 Extended (240+ blendshapes) {% endstep %} {% step %} #### Assign target meshes In the **Target Meshes** list, add all `SkinnedMeshRenderer` components that have facial blendshapes. {% endstep %} {% step %} #### Enter Play Mode Leave **Mapping** empty — the SDK auto-selects a matching bundled map for the chosen profile. Enter Play Mode and speak to the character. The character's mouth moves in sync with its voice output. {% hint style="warning" %} If the mouth does not move, confirm that your `SkinnedMeshRenderer` blendshape names match the expected naming convention for the chosen profile. ARKit uses camelCase names (e.g., `jawOpen`, `mouthSmileLeft`). MetaHuman uses the `CTRL_expressions_` prefix. Use a custom map if your rig uses different names — see [Profiles and mappings](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-lip-sync/profiles-and-mappings). {% endhint %} {% endstep %} {% endstepper %} ### Bundled profiles Choose the profile that matches the blendshape format your character was rigged with. | Profile | Locked Profile ID | Blendshapes | Typical character source | | ------------ | ----------------- | ----------- | ----------------------------------------- | | ARKit | `arkit` | 61 | Apple-rigged characters, some custom rigs | | MetaHuman | `metahuman` | 275+ | Unreal MetaHuman exported to Unity | | CC4 Extended | `cc4extended` | 240+ | Reallusion Character Creator 4 characters | If your character was rigged with non-standard blendshape names, create a custom map to route the SDK's output channels to your rig's actual names. {% content-ref url="/pages/bWXMsjnl9w0v1XGbrvox" %} [Profiles and mappings](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-lip-sync/profiles-and-mappings) {% endcontent-ref %} ### Playback settings **Core setup:** | Field | Default | Description | | ------------------ | -------------- | ---------------------------------------------------------------------- | | `_lockedProfileId` | `arkit` | Transport format the SDK streams (`arkit`, `metahuman`, `cc4extended`) | | `_mapping` | *(none)* | Optional custom mapping asset (leave empty to use bundled auto-map) | | `_targetMeshes` | *(empty list)* | `SkinnedMeshRenderer` components to write blendshapes to | **Playback & behavior:** | Field | Default | Range | Description | | ------------------ | ------- | -------- | -------------------------------------------------------------- | | `_smoothingFactor` | `0.5` | 0–0.9 | Exponential smoothing per frame (higher = smoother but slower) | | `_fadeOutDuration` | `0.2` | 0.05–2.0 | Seconds to fade all blendshapes to 0 after audio ends | | `_timeOffset` | `0.0` | -0.5–0.5 | Shift playback timing relative to audio (negative = earlier) | **Streaming & latency:** | Field | Default | Range | Description | | --------------------------- | ---------- | -------- | ---------------------------------------------------- | | `_latencyMode` | `Balanced` | — | Preset that controls buffer depth vs. responsiveness | | `_maxBufferedSeconds` | `3.0` | 1–10 | Ring buffer capacity in seconds | | `_minResumeHeadroomSeconds` | `0.12` | 0.05–0.3 | Buffer refill threshold after starvation | **Latency mode options:** | Mode | Use case | | ----------------- | ---------------------------------------------------------------- | | `Balanced` | Default. Recommended for most deployments | | `UltraLowLatency` | Minimal delay; susceptible to starvation on unstable connections | | `NetworkSafe` | High buffering; best for unreliable or high-latency networks | | `Custom` | Unlocks manual control over buffer fields above | ### Usage examples #### Example 1: ARKit character **Scenario:** A corporate training simulation uses a character rigged with Apple ARKit blendshapes. **Setup:** 1. Add `ConvaiLipSyncComponent` to the NPC GameObject (same as `ConvaiCharacter`). 2. Set `_lockedProfileId` to `arkit`. 3. In the **Target Meshes** list, add the `SkinnedMeshRenderer` from the avatar's head mesh. 4. Leave `_mapping` empty — the bundled ARKit auto-map covers standard camelCase ARKit blendshape names (`jawOpen`, `mouthSmileLeft`, etc.). **Expected outcome:** The avatar's mouth, lips, and jaw animate in sync with the character's voice during conversation. Blendshapes return to neutral smoothly after each response ends (`_fadeOutDuration` = 0.2s default). #### Example 2: MetaHuman character **Scenario:** A high-fidelity medical simulation uses an Unreal MetaHuman character exported to Unity. **Setup:** 1. Add `ConvaiLipSyncComponent` to the NPC GameObject. 2. Set `_lockedProfileId` to `metahuman`. 3. In the **Target Meshes** list, add all `SkinnedMeshRenderer` components on the MetaHuman head and teeth meshes — MetaHuman separates these into multiple renderers. 4. Leave `_mapping` empty — the bundled MetaHuman map targets `CTRL_expressions_` prefixed blendshapes. 5. Increase `_smoothingFactor` to `0.7` for more fluid animation on high-poly rigs. **Expected outcome:** All facial regions animate together — lips, jaw, cheeks, and tongue shapes — producing highly realistic mouth movement. Smoothing reduces per-frame jitter visible on high-resolution meshes. ### Next steps After lip sync is configured, validate your complete setup. {% content-ref url="/pages/35a808a76b0d30cee351c5b850b110058a2eb840" %} [Validate your setup](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/validate-your-setup) {% endcontent-ref %} # Profiles and mappings Reference for the two ScriptableObject types — profiles and maps — that control how Convai blendshape data reaches your character's SkinnedMeshRenderer. The lip sync system uses two ScriptableObject types to describe how blendshape data flows from Convai to your character's mesh: **profiles** and **maps**. Most setups work with the bundled assets — create custom ones only when your rig uses non-standard blendshape names or a transport format not covered by the built-in profiles. ### What is a profile? A **profile** defines the transport format — it tells the SDK which blendshape channel names to expect in the data stream from Convai. The profile's ID (e.g., `arkit`, `metahuman`, `cc4extended`) is what you enter in `ConvaiLipSyncComponent._lockedProfileId`. **Three profiles are bundled:** | Profile asset | ID | Format | | ---------------------------------- | ------------- | --------------------------------- | | `ConvaiLipSyncProfile_ARKit` | `arkit` | 61 standard ARKit channels | | `ConvaiLipSyncProfile_MetaHuman` | `metahuman` | 275+ MetaHuman CTRL expressions | | `ConvaiLipSyncProfile_CC4Extended` | `cc4extended` | 240+ Character Creator 4 channels | Create a custom profile only if your character uses a proprietary blendshape format that Convai streams under a custom ID. In practice, this is rare — most pipelines use one of the three bundled formats. ### What is a map? A **map** routes the source blendshape channels (from the profile) to the actual blendshape names on your character's `SkinnedMeshRenderer`. It also lets you apply per-channel multipliers, offsets, and clamps. **Four maps are bundled:** | Map asset | Routes | | -------------------------------------------- | --------------------------------------- | | `ConvaiLipSyncDefaultMap_ARKit` | ARKit → ARKit (passthrough) | | `ConvaiLipSyncDefaultMap_MetaHuman` | MetaHuman → MetaHuman (passthrough) | | `ConvaiLipSyncDefaultMap_CC4Extended` | CC4Extended → CC4Extended (passthrough) | | `ConvaiLipSyncDefaultMap_ARKitToCC4Extended` | ARKit → CC4Extended (conversion) | When `ConvaiLipSyncComponent._mapping` is left empty, the SDK selects the matching passthrough map automatically based on the locked profile ID. Create a custom map when your character's blendshape names differ from the expected names, or when you need to adjust weight multipliers to match your rig's calibration. ### When to create custom assets | Situation | Create | | ------------------------------------------------------- | ------------------------------------------------------ | | Your rig uses ARKit/MetaHuman/CC4 names exactly | Nothing — use the bundled map (leave `_mapping` empty) | | Your rig uses different names for standard blendshapes | Custom **map** only | | You receive an ARKit stream but your rig uses CC4 names | Use the bundled `ARKitToCC4Extended` map | | Your character uses a completely custom blendshape set | Custom **profile** + custom **map** |
Create a lip sync profile
Define a new transport format identifier for proprietary blendshape sets.		/pages/W7KKkYg5RjVc2F3b63Vz
Create a custom blendshape map
Route stream channels to your rig's actual blendshape names with per-channel tuning.		/pages/dZnu5jciNDbTRscNenck
# Create a lip sync profile Create a ConvaiLipSyncProfile asset to define a custom transport format identifier for blendshape formats beyond the three bundled profiles. A lip sync profile defines the transport format identifier — it tells the SDK which blendshape channel names to expect from Convai. {% hint style="warning" %} Convai currently streams blendshape data in three formats only: `arkit`, `metahuman`, and `cc4extended`. A custom profile is only useful if Convai explicitly supports a custom format for your deployment. Creating a profile with an unsupported format ID will result in no blendshape data being received. {% endhint %} For all standard setups, use one of the three bundled profiles and leave this page for advanced or future use. {% stepper %} {% step %} #### Create the asset In the Project window, navigate to the folder where you want to store the profile. Right-click and select **Create > Convai > Lip Sync Profile**. Name the asset descriptively — for example, `ConvaiLipSyncProfile_MyRig`. {% endstep %} {% step %} #### Configure the fields Select the new asset to open it in the Inspector. | Field | Description | | -------------------- | ------------------------------------------------------------------------------------------------ | | **Profile ID** | Transport format identifier. Must exactly match the format ID Convai streams for this character. | | **Display Name** | Human-readable name shown in the Inspector. Does not affect runtime behavior. | | **Transport Format** | Network transport format string — typically the same as Profile ID. | {% hint style="danger" %} **Profile ID** must exactly match the identifier Convai uses when streaming blendshape data. A mismatch causes the SDK to fall back to a no-op map — no blendshapes animate. {% endhint %} {% endstep %} {% step %} #### Assign the profile ID to the component Set `ConvaiLipSyncComponent._lockedProfileId` to the **Profile ID** string you defined. The SDK looks up this profile from the registry at runtime. {% endstep %} {% endstepper %} You will also need a matching map asset that routes the custom channels to your rig's blendshape names. ### Next steps After creating the profile, create a map to route its channels to your rig's blendshape names. {% content-ref url="/pages/dZnu5jciNDbTRscNenck" %} [Create a custom lip sync map](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/add-lip-sync/create-a-custom-blendshape-map) {% endcontent-ref %} # Create a custom lip sync map Create a ConvaiLipSyncMapAsset to route stream blendshape channels to your character's actual SkinnedMeshRenderer blendshape names, with per-channel multiplier and clamp tuning. A lip sync map routes source blendshape channels (from the transport stream) to the actual blendshape names on your character's `SkinnedMeshRenderer`. Create a custom map when your rig uses different blendshape names than the bundled passthrough maps expect, or when you need to tune weights for your specific character. {% stepper %} {% step %} #### Create the asset In the Project window, navigate to the folder where you want to store the map. Right-click and select **Create > Convai > Lip Sync Map**. Name the asset descriptively — for example, `ConvaiLipSyncMap_MyRig_FromARKit`. {% endstep %} {% step %} #### Configure top-level fields Select the new asset to open it in the Inspector. **Top-level fields:** | Field | Default | Description | | ------------------------------ | --------- | --------------------------------------------------------------------------------------- | | **Target Profile ID** | *(empty)* | The profile ID this map targets (e.g., `arkit`, `metahuman`, or your custom profile ID) | | **Description** | *(empty)* | Optional designer notes — not used at runtime | | **Global Multiplier** | `1.0` | Scale applied to all output weights before writing (0–3) | | **Global Offset** | `0.0` | Offset added to all output weights (-1–1) | | **Allow Unmapped Passthrough** | `true` | Channels with no explicit mapping entry are written using their source name directly | **Mapping entries:** Click **+** in the **Mappings** list to add a new entry. Each entry maps one source channel to one or more target blendshape names. | Entry field | Default | Description | | --------------------------- | -------------- | ------------------------------------------------------------------------- | | **Source Blendshape** | *(empty)* | Name of the channel as it arrives from Convai (e.g., `jawOpen`) | | **Target Names** | *(empty list)* | Names of the blendshapes on your `SkinnedMeshRenderer` to drive | | **Multiplier** | `1.0` | Per-entry scale applied before the global multiplier (0–5) | | **Offset** | `0.0` | Per-entry offset (-1–1) | | **Enabled** | `true` | Toggle this entry on or off without deleting it | | **Clamp Min Value** | `0.0` | Minimum output value (0–1) | | **Clamp Max Value** | `1.0` | Maximum output value (0–1) | | **Use Override Value** | `false` | When enabled, always write **Override Value** instead of the stream value | | **Override Value** | `0.0` | Constant value written when **Use Override Value** is on | | **Ignore Global Modifiers** | `false` | Skip **Global Multiplier** and **Global Offset** for this entry | {% hint style="info" %} **Allow Unmapped Passthrough** is useful when most of your blendshape names match the source channels — you only need to add entries for the ones that differ. Unmapped channels write directly using their source name. {% endhint %} {% endstep %} {% step %} #### Assign the map to the component In the `ConvaiLipSyncComponent` Inspector, drag your new map asset into the **Mapping** field. Enter Play Mode and speak to the character. All mapped blendshapes animate. Check the Unity Console for any warnings about unresolved blendshape names. {% endstep %} {% endstepper %} ### Usage examples #### Example 1: ARKit stream to a custom rig **Scenario:** A simulation character was rigged by an artist who used snake\_case names (`jaw_open`, `mouth_smile_left`) instead of the ARKit standard camelCase names (`jawOpen`, `mouthSmileLeft`). **Setup:** * Create `ConvaiLipSyncMap_CustomRig_FromARKit.asset` * Target Profile ID: `arkit` * Allow Unmapped Passthrough: `false` (all names differ) * Add entries for each blendshape: | Source blendshape | Target names | | ------------------------------------ | ------------------- | | `jawOpen` | `jaw_open` | | `mouthSmileLeft` | `mouth_smile_left` | | `mouthSmileRight` | `mouth_smile_right` | | *(continue for all needed channels)* | | **Expected outcome:** The character's mouth animates correctly despite using different naming conventions than the ARKit standard. #### Example 2: Reducing jaw movement intensity **Scenario:** An AI receptionist character's jaw opens too wide during speech, which looks unnatural. **Setup:** * Duplicate the bundled `ConvaiLipSyncDefaultMap_ARKit.asset` and rename it * Find the `jawOpen` entry * Set **Multiplier** to `0.6` and **Clamp Max Value** to `0.7` **Expected outcome:** The character's jaw opens only 60–70% as much as the raw stream value, resulting in more restrained, naturalistic mouth movement. ### Next steps With lip sync configured, validate your complete scene setup. {% content-ref url="/pages/35a808a76b0d30cee351c5b850b110058a2eb840" %} [Validate your setup](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/validate-your-setup) {% endcontent-ref %} # Core concepts Find explanations for how the Convai Unity SDK works at runtime — connection management, session state, turn-taking, and event-driven scene responses. The Core Concepts section explains the systems that run beneath every Convai character in your scene — the runtime layers that manage connections, the session state machine that governs each character's lifecycle, the turn-taking system that controls who speaks and when, and the event components that let your scene react to what happens at runtime. Start with **Runtime architecture** if you are new to the SDK internals. Read the other pages when you need to configure a specific system or understand why the SDK behaves a certain way.
Runtime architecture
Runtime layers, interface inventory, and what you can replace.		/pages/LV9IsLoeHhBA2EovBKod
Session lifecycle
State machine, per-character sessions, persistence, and reconnection.		/pages/TvL9WjMsqfa9XQdTFSUo
Turn-taking modes
Hands-free vs. push-to-talk — every field and policy explained.		/pages/b9e67d1d27aaccf9e9bc9b260045411047aa37c3
Event system
Relay components, payload types, and subscription patterns.		/pages/bcd284a409f00820e1693ab998dc7a99a4fbd780
# Runtime architecture Understand the four-layer Convai Unity SDK runtime — what each layer owns, which components are replaceable, and how RuntimeState transitions are managed. The Convai Unity SDK is built in layers. Each layer has a defined responsibility and communicates inward — outer layers depend on inner ones, never the reverse. Understanding this structure tells you which parts of the SDK are developer-facing, which are replaceable, and which are internal implementation details you do not need to touch. *** ### System layers The diagram below shows the four main layers and how they relate. `ConvaiRuntime` holds four direct sub-systems at the second tier — `IRoomRuntime`, `IEventHub`, `IAgentRegistry`, and the module list. Characters and players surface beneath `IRoomRuntime` and `IAgentRegistry`, and all modules share a common context layer. ```mermaid graph TD A[ConvaiRuntime
IConvaiRuntime] --> B[IRoomRuntime
Connection · Audio · Ownership · Diagnostics] A --> C[IEventHub
Decoupled pub/sub] A --> D[IAgentRegistry
Characters · Players] A --> E[IReadOnlyList<IConvaiModule>
Feature modules] B --> F[ConvaiCharacter
per-character session + state] B --> G[ConvaiPlayer
local participant identity] F --> H[Module Context
LipSync · Emotion · Actions · …] G --> H ``` **`ConvaiRuntime` (top layer)** — owns every sub-system and coordinates the full lifecycle. Created once per application lifetime via `ConvaiRuntimeBuilder`. Exposes start, pause, resume, and stop operations that propagate to all registered modules. It holds four direct sub-systems: * **`IRoomRuntime`** — manages the real-time session: connecting, disconnecting, audio routing, character ownership, and diagnostics. Characters and the local player surface beneath this layer. * **`IEventHub`** — decoupled publish/subscribe bus used throughout the SDK for cross-system communication. * **`IAgentRegistry`** — registry of all active `ConvaiCharacter` and `ConvaiPlayer` instances. * **`IReadOnlyList`** — the set of registered feature modules. **Character / Player layer** — `ConvaiCharacter` and `ConvaiPlayer` register with `IAgentRegistry` and receive their per-session context through `IRoomRuntime`. Each character maintains its own session state. **Module context layer** — feature modules (`IConvaiModule`) share an `IModuleContext` that provides access to runtime services. Modules are isolated: they do not call each other directly. *** ### Runtime interface inventory `IConvaiRuntime` exposes the following sub-systems as properties. Each property is the entry point for a specific domain of functionality. | Property | Type | What It Owns | | -------------------- | ------------------------------- | --------------------------------------------- | | `State` | `RuntimeState` | Current lifecycle state of the runtime | | `Room` | `IRoomRuntime` | Connection, audio, ownership, and diagnostics | | `Events` | `IEventHub` | Decoupled publish/subscribe communication | | `Agents` | `IAgentRegistry` | Registry of all active characters and players | | `Modules` | `IReadOnlyList` | All registered feature modules | | `Transport` | `ITransportProvider` | Platform-specific real-time transport | | `Conversation` | `IConversationProvider` | AI backend communication | | `Config` | `ConvaiBootstrapConfigSnapshot` | Immutable bootstrap configuration | | `RuntimePreferences` | `IRuntimePreferences` | Mutable runtime preferences | | `FeatureVariants` | `IFeatureVariantProvider` | Feature variant / A-B selection | | `Persistence` | `IPersistenceProvider` | Runtime-owned data storage | | `Telemetry` | `ITelemetryProvider` | Observability and analytics | Developers interact with `Room`, `Agents`, and `Events` most frequently. `Transport`, `Conversation`, `Persistence`, `Telemetry`, and `FeatureVariants` are replaceable via `ConvaiRuntimeBuilder`. *** ### What you can replace `ConvaiRuntimeBuilder` is the fluent API for composing the runtime before it starts. Every method returns `this`, so calls chain. ```csharp var runtime = new ConvaiRuntimeBuilder() .UsePersistence(myPersistenceProvider) .UseTelemetry(myTelemetryProvider) .WithEndUserIdentityProvider(myIdentityProvider) .AddModule() .Build(); ``` The table below lists what is replaceable vs. internal-only. | Component | Replaceable via Builder | Default | | ------------------------ | -------------------------------- | -------------------------------------- | | Transport provider | `UseTransport()` | Platform-default (WebSocket / LiveKit) | | Conversation provider | `UseConversation()` | Convai RTVI conversation backend | | Persistence provider | `UsePersistence()` | `PlayerPrefs`-backed key-value store | | Telemetry provider | `UseTelemetry()` | No-op telemetry | | Feature variant provider | `WithFeatureVariants()` | Static feature flags | | Runtime preferences | `WithRuntimePreferences()` | Defaults from `ConvaiSettings` | | Event hub | `UseEventHub()` | Default in-memory event hub | | Agent registry | `UseAgentRegistry()` | Default registry | | End-user identity | `WithEndUserIdentityProvider()` | Device ID provider | | End-user metadata | `WithEndUserMetadataProvider()` | None | | Modules | `AddModule()` / `AddModule()` | SDK feature modules only | | Room runtime | `UseRoomRuntime()` | Internal LiveKit-backed room | {% hint style="info" %} `ConvaiRuntime` is created by the `ConvaiManager` MonoBehaviour automatically. Most projects never call `ConvaiRuntimeBuilder` directly. Use it only when you need to replace a default provider or add a custom module. {% endhint %} *** ### `IRoomRuntime` sub-structure The room layer is itself composed of four coordinators, all accessible via `IConvaiRuntime.Room`. | Property | Type | Responsibility | | ------------- | ---------------------------- | --------------------------------------------- | | `Connection` | `IRoomConnectionCoordinator` | Connect, disconnect, session state | | `Audio` | `IRoomAudioCoordinator` | Microphone capture, remote audio playback | | `Ownership` | `IRoomOwnershipCoordinator` | Which characters this client owns and focuses | | `Diagnostics` | `IRoomDiagnostics` | Session metrics, health monitoring | Connection and audio are the coordinators you are most likely to call from scripting. Ownership is managed automatically when you have multiple characters in the scene. Diagnostics are used for performance monitoring and debugging. *** ### Module layer Modules are feature extensions that run inside the runtime lifecycle. They receive a shared `IModuleContext` and can register services that other modules or the presentation layer consume. Add a module via the builder before `Build()` is called: ```csharp new ConvaiRuntimeBuilder() .AddModule() .AddModule(new MyCustomModule(someConfig)) .Build(); ``` Modules are started, paused, resumed, and stopped alongside the runtime. The `IConvaiModule` interface defines these lifecycle hooks. See [Extending the SDK](/api-docs/plugins-and-integrations/convai-unity-sdk/advanced-topics/extending-the-sdk) for the full module authoring reference. {% hint style="warning" %} Modules cannot depend on each other directly. If two modules need to share data, use the `IEventHub` or a shared service registered via `IModuleContext`. {% endhint %} *** ### `RuntimeState` lifecycle The runtime moves through the following states from creation to disposal. | State | Meaning | | ---------- | ----------------------------------------------- | | `Created` | Runtime built but not yet started | | `Starting` | `StartAsync()` in progress | | `Running` | Fully operational | | `Pausing` | `PauseAsync()` in progress | | `Paused` | Paused; can be resumed | | `Resuming` | `ResumeAsync()` in progress | | `Stopping` | `StopAsync()` in progress | | `Stopped` | Shut down; cannot restart | | `Disposed` | `DisposeAsync()` called; all resources released | ```mermaid stateDiagram-v2 [*] --> Created Created --> Starting : StartAsync() Starting --> Running Running --> Pausing : PauseAsync() Pausing --> Paused Paused --> Resuming : ResumeAsync() Resuming --> Running Running --> Stopping : StopAsync() Stopping --> Stopped Stopped --> [*] : DisposeAsync() ``` All state transitions are async operations wrapped in `IConvaiOperation`. Check `Status` on the returned operation to confirm the transition completed before proceeding. {% hint style="info" %} `ConvaiManager` handles `DisposeAsync()` automatically on destroy. If you build a custom host using `ConvaiRuntimeBuilder` directly, call `DisposeAsync()` after `StopAsync()` to release all resources. {% endhint %} *** ### Next steps You now understand how the Convai runtime is composed and which parts are replaceable at each layer. Read Session lifecycle next to understand how each character's session is created, persisted, and recovered, then continue through Turn-taking modes and Event system. {% content-ref url="/pages/TvL9WjMsqfa9XQdTFSUo" %} [Session lifecycle](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/session-lifecycle) {% endcontent-ref %} {% content-ref url="/pages/b9e67d1d27aaccf9e9bc9b260045411047aa37c3" %} [Turn-taking modes](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/turn-taking-modes) {% endcontent-ref %} # Session lifecycle Understand how ConvaiCharacter sessions transition through states, where session IDs are stored, and how to configure reconnection and conversation-resume behavior. Every `ConvaiCharacter` in your scene maintains an independent session with Convai. That session tracks whether the character is connected, what its current state is, and — when persistence is enabled — what conversation it was in the last time you connected. Understanding how sessions are created, persisted, and recovered lets you build reliable, resumable character interactions across training simulations, interactive experiences, and games. *** ### Session state machine Each character session moves through the following states. ```mermaid stateDiagram-v2 [*] --> Disconnected Disconnected --> Connecting : connect initiated Connecting --> Connected : connection established Connecting --> Error : configuration or auth failure Connected --> Disconnecting : disconnect initiated Disconnecting --> Disconnected : clean shutdown Connected --> Reconnecting : connection lost Reconnecting --> Connected : reconnect succeeded Reconnecting --> Error : max attempts exceeded ``` | State | Value | Meaning | | --------------- | ----- | ----------------------------------------------------------------------------- | | `Disconnected` | 0 | No active session. Initial state and final state after a clean disconnect. | | `Connecting` | 1 | Connection attempt in progress. Transitioning from Disconnected to Connected. | | `Connected` | 2 | Session is active. Audio streams and conversations are live. | | `Reconnecting` | 3 | Connection was lost. SDK is attempting to re-establish it automatically. | | `Disconnecting` | 4 | Graceful shutdown in progress. Transitioning from Connected to Disconnected. | | `Error` | 5 | Unrecoverable error. Manual intervention required to reconnect. | You receive state transitions as `SessionStateChangedRelayData` events via `ConvaiSessionEventRelay`. See [Event system](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/event-system) for how to subscribe. *** ### Per-character sessions Each `ConvaiCharacter` has its own independent session. Sessions are not shared between characters. In multi-character scenes, each character connects and disconnects independently — session IDs are keyed to the character's ID string set in the Inspector (not the scene or object name), reconnect policy applies per character, and a session error on one character does not affect others. `ConvaiSessionData` is the persistent session store that maps each character to its current session identifier. It loads from disk automatically at startup and writes to `{Application.persistentDataPath}/Convai/sessions.json` on every change — session IDs survive application restarts without any additional setup. | Method | Description | | ---------------------------------------- | --------------------------------------------------------------------------- | | `GetSessionId(characterId)` | Returns the current session ID for the character, or `null` if none exists. | | `StoreSessionId(characterId, sessionId)` | Stores a session ID for the character and saves it to disk immediately. | | `ClearSessionId(characterId)` | Removes the session ID for one character and saves. | | `ClearAllSessionIds()` | Removes all stored session IDs and saves. | | `GetAllSessionIds()` | Returns a read-only snapshot of all current character→sessionId mappings. | {% hint style="info" %} `ConvaiSessionData` is a singleton. Data is stored at `{Application.persistentDataPath}/Convai/sessions.json` and persists across application restarts. Call `ClearAllSessionIds()` explicitly if you need a clean slate. {% endhint %} *** ### Session persistence When a session ID is persisted, the SDK can resume a previous conversation on the next connect — the character remembers context from prior interactions. #### What persists vs. what resets | On Reconnect | Behavior | | ---------------------------- | --------------------------------------------------- | | Session ID | Persisted via `ConvaiSessionData` — enables resume | | Conversation history | Managed by Convai; resumed when session ID is valid | | In-flight audio | Reset — any audio mid-stream is discarded | | Active turn state | Reset — the turn restarts clean | | Module state (e.g., emotion) | Reset — modules reinitialize on reconnect | #### Default persistence stack The SDK exposes a pluggable persistence layer via `ISessionPersistence` for projects that need a custom backing store (encrypted storage, cloud saves, a database). The default stack is: ``` ISessionPersistence └─ KeyValueStoreSessionPersistence ← maps characterId → sessionId with prefix "convai.session." └─ PlayerPrefsKeyValueStore ← default IKeyValueStore implementation; wraps Unity PlayerPrefs └─ UnityEngine.PlayerPrefs ← persisted to disk ``` Session IDs are stored under keys formatted as `convai.session.`. #### Replacing the persistence store Implement `IKeyValueStore` to use any backing store — a database, encrypted storage, a cloud save system. `PlayerPrefsKeyValueStore` marshals all reads and writes to the Unity main thread internally; apply the same thread-safety pattern if your backing store has thread restrictions. ```csharp public sealed class SecureKeyValueStore : IKeyValueStore { public string GetString(string key, string defaultValue = null) { return SecureStorage.GetValue(key) ?? defaultValue; } public void SetString(string key, string value) { SecureStorage.SetValue(key, value); } public bool HasKey(string key) => SecureStorage.HasKey(key); public void DeleteKey(string key) => SecureStorage.DeleteKey(key); public void Save() => SecureStorage.Flush(); } ``` Register it via `ConvaiRuntimeBuilder`: ```csharp var runtime = new ConvaiRuntimeBuilder() .UsePersistence(new MyPersistenceProvider(new SecureKeyValueStore())) .Build(); ``` *** ### Reconnect policy `ReconnectPolicy` controls what the SDK does when a connection drops unexpectedly. | Field | Type | Default | Description | | -------------------------- | -------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `RoomRejoinTtlSeconds` | `double` | `60` | Window in seconds during which the SDK can rejoin an existing room after a drop. After this window, a new room is created instead. | | `ResumePolicy` | `ResumePolicy` | `ResumeIfPossible` | Controls whether the SDK attempts to resume the previous conversation via `character_session_id`. | | `MaxReconnectAttempts` | `int` | `3` | Maximum number of automatic reconnect attempts before the session moves to `Error` state. | | `SpawnAgentOnRejoin` | `bool` | `true` | Whether to re-spawn the AI agent when rejoining an existing room. | | `StartWaitTimeoutMs` | `int` | `5000` | Timeout in milliseconds for the connection `Start()` phase before the attempt is considered failed. | | `AutoMicStartDelaySeconds` | `float` | `0.5` | Seconds to wait after connection before starting the microphone. Prevents audio capture before the session is fully ready. | #### `ResumePolicy` options | Value | Behavior | | ------------------ | ---------------------------------------------------------------------------------------------------------------------- | | `AlwaysFresh` | Always start a new conversation. The character has no memory of the previous session. | | `ResumeIfPossible` | Attempt to resume the previous conversation. If the session has expired or resume fails, fall back to a fresh session. | | `AlwaysResume` | Always resume. If resume fails, the connection fails — no fallback to a fresh session. | #### Preset policies | Preset | Description | | --------------------------------- | --------------------------------------------------------------- | | `ReconnectPolicy.Default` | 60 s TTL, `ResumeIfPossible`, 3 attempts, mic delay 0.5 s | | `ReconnectPolicy.AlwaysCreateNew` | No rejoin attempt. Always creates a new room and fresh session. | ```csharp var policy = new ReconnectPolicy( roomRejoinTtlSeconds: 120, resumePolicy: ResumePolicy.AlwaysFresh, maxReconnectAttempts: 5, autoMicStartDelaySeconds: 1.0f ); ``` {% hint style="warning" %} `AlwaysResume` will put the session into `Error` state if Convai cannot resume the session (e.g., if the session expired on the backend). Use `ResumeIfPossible` unless your training simulation requires strict continuity and you have handled the error state explicitly. {% endhint %} *** ### Usage examples #### Example 1: Medical training simulation — resume after network drop A learner is mid-assessment when the network drops. When connection is restored, the patient character resumes the same conversation — no context is lost. ```csharp var policy = new ReconnectPolicy( roomRejoinTtlSeconds: 120, // 2-minute window to rejoin the existing room resumePolicy: ResumePolicy.ResumeIfPossible, maxReconnectAttempts: 5, autoMicStartDelaySeconds: 1.0f // extra delay for slow mobile networks ); ``` **Expected outcome:** The SDK automatically retries up to 5 times within the 2-minute window. If the session is still valid on Convai's side, the conversation resumes from where it left off. If the session expired, the character starts a fresh conversation rather than blocking. *** #### Example 2: Corporate onboarding kiosk — always-fresh conversations Each new employee who approaches the kiosk should start from the beginning with no memory of previous users. `AlwaysFresh` and `AlwaysCreateNew` ensure a clean slate every time. ```csharp // Apply policy in the ConvaiRoomManager's reconnect settings var policy = ReconnectPolicy.AlwaysCreateNew; // ResumePolicy defaults to AlwaysFresh in AlwaysCreateNew — no prior session carried over ``` To guarantee previous user data is removed before the next session starts: ```csharp public class KioskSessionReset : MonoBehaviour { [SerializeField] private string _characterId; public void OnUserLogOut() { ConvaiSessionData.Instance.ClearSessionId(_characterId); } } ``` **Expected outcome:** Every new user starts a completely fresh conversation. The character has no memory of previous interactions, which is correct for a shared kiosk deployment. *** #### Example 3: Handling the error state in a training simulation When all reconnect attempts are exhausted, the session enters `Error` state. Surface this to the facilitator and allow manual retry rather than silently hanging. ```csharp public class SessionErrorHandler : MonoBehaviour { [SerializeField] private ConvaiSessionEventRelay _relay; [SerializeField] private GameObject _errorPanel; [SerializeField] private ConvaiManager _manager; private void OnEnable() => _relay.OnSessionStateChanged.AddListener(HandleStateChange); private void OnDisable() => _relay.OnSessionStateChanged.RemoveListener(HandleStateChange); private void HandleStateChange(SessionStateChangedRelayData data) { _errorPanel.SetActive(data.IsError); } // Called by the facilitator's "Retry" button public async void RetryConnection() { _errorPanel.SetActive(false); await _manager.ConnectAsync(); } } ``` **Expected outcome:** The error panel appears when the session enters `Error` state. The facilitator clicks "Retry" to attempt a fresh connection without restarting the simulation. *** ### Troubleshooting | Symptom | Likely Cause | Fix | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Session stays in `Error` state after a drop | `AlwaysResume` could not resume the expired session on the backend | Switch to `ResumeIfPossible`; call `ClearSessionId(characterId)` to remove the stale session ID, then reconnect | | Character starts a fresh conversation on every launch despite `ResumeIfPossible` | A previous `ClearAllSessionIds()` call wiped the session file, or the character ID changed between runs | Verify the `characterId` string is stable across runs; check `{persistentDataPath}/Convai/sessions.json` | | Session stuck in `Connecting` forever | `StartWaitTimeoutMs` not configured for slow network; or firewall blocking the transport | Increase `StartWaitTimeoutMs` in `ReconnectPolicy`; verify network access to Convai endpoints | | Reconnect loop never succeeds; session eventually reaches `Error` | `MaxReconnectAttempts` exhausted | Subscribe to `ConvaiSessionEventRelay.OnSessionStateChanged` and surface the error to the user; call reconnect manually after the user acknowledges | | Two characters share a session ID | Character ID strings are identical in the Inspector | Assign unique character IDs to each `ConvaiCharacter` in the scene | *** ### Next steps You now know how character sessions are created, how state transitions work, how session IDs are persisted across restarts, and how to configure reconnection behavior. Read Turn-taking modes next to configure how the SDK detects speech input, then Event system to learn how to subscribe to session and character events from your scene scripts. {% content-ref url="/pages/b9e67d1d27aaccf9e9bc9b260045411047aa37c3" %} [Turn-taking modes](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/turn-taking-modes) {% endcontent-ref %} {% content-ref url="/pages/bcd284a409f00820e1693ab998dc7a99a4fbd780" %} [Event system](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/event-system) {% endcontent-ref %} # Turn-taking modes Reference for TurnTakingOptions, SmartTurnSettings, and PushToTalkPolicy — field reference for hands-free voice detection and push-to-talk modes. Turn-taking determines who speaks, when a turn ends, and how the SDK handles the transition between the user speaking and the character responding. The SDK supports two modes: hands-free automatic detection and explicit push-to-talk. Choosing the right mode — and tuning it correctly — directly affects how natural and reliable the conversation feels in your training simulation, interactive experience, or game. For the Inspector-based setup steps, see [Configure conversation input mode](/api-docs/plugins-and-integrations/convai-unity-sdk/getting-started/configure-conversation-input-mode). This page is the full field reference. *** ### Mode comparison | Mode | `ConversationInputMode` Value | Best For | | ---------------- | ----------------------------- | -------------------------------------------------------------------------------------------------- | | **Hands-Free** | `HandsFree` (0) | Training simulations with natural dialogue, ambient interaction, accessibility-first experiences | | **Push-to-Talk** | `PushToTalk` (1) | Noisy environments, factory safety drills, scenarios where accidental triggering must be prevented | *** ### `TurnTakingOptions` — root fields `TurnTakingOptions` is the top-level configuration object. Set it on `ConvaiRoomManager` in the Inspector (inline) or via a `ConvaiRoomManagerProfile` asset, or pass it to `RoomSessionConnectOptions` for per-connection overrides. | Field | Type | Default | Description | | --------------------- | ----------------------- | ------------ | ------------------------------------------------------------------------------- | | `Mode` | `ConversationInputMode` | `HandsFree` | Sets the active conversation mode for this session. | | `TurnDetection` | `TurnDetectionMode` | `UseDefault` | Controls automatic end-of-turn detection. Only applies in Hands-Free mode. | | `CustomTurnDetection` | `SmartTurnSettings` | See below | Fine-tuned smart-turn parameters. Only active when `TurnDetection` is `Custom`. | | `InitialServerStt` | `ServerSttInitialState` | `UseDefault` | Controls whether backend speech-to-text is enabled at session start. | | `LocalAudioPolicy` | `LocalAudioPolicy` | See below | Microphone behavior on this device. Applies to both modes. | | `PushToTalkPolicy` | `PushToTalkPolicy` | See below | Push-to-talk interaction rules. Only applies in PushToTalk mode. | *** ### Hands-free mode In hands-free mode, the SDK continuously captures microphone audio and detects when the user has finished speaking. No button press is required. #### `TurnDetectionMode` Controls how the end-of-turn is detected. | Value | Behavior | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `UseDefault` | Convai's default server-side voice activity detection. Suitable for most cases. | | `Disabled` | No automatic turn detection. The SDK will not end the user's turn automatically. Use only if you manage turn transitions entirely in code. | | `Custom` | Use `SmartTurnSettings` to configure the detection parameters yourself. | #### `SmartTurnSettings` Active when `TurnDetection` is set to `Custom`. | Field | Type | Default | Description | | ----------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `StopSecs` | `float` | `3.0` | Seconds of silence required before the SDK ends the user's turn. Reduce for faster response; increase in noisy environments. | | `PreSpeechMs` | `int` | `0` | Milliseconds of audio before detected speech onset to include in the captured turn. Increase if the first word of a turn is frequently clipped. | | `MaxDurationSecs` | `float` | `8.0` | Hard cap on a single user turn in seconds. The turn ends regardless of whether the user stopped speaking. | ```csharp var options = new TurnTakingOptions { Mode = ConversationInputMode.HandsFree, TurnDetection = TurnDetectionMode.Custom, CustomTurnDetection = new SmartTurnSettings { StopSecs = 2.0f, // faster response for medical assessment flow PreSpeechMs = 100, MaxDurationSecs = 10.0f } }; ``` {% hint style="warning" %} Setting `StopSecs` too low causes premature turn endings when the user pauses mid-sentence. In training simulations where learners think before they respond, keep `StopSecs` at 2.5 or higher. {% endhint %} *** ### Push-to-talk mode In push-to-talk mode, the user explicitly starts and ends their turn by pressing and releasing a control (button, key, or UI element). The SDK does not use voice activity detection to end turns. #### `PushToTalkPolicy` Controls all push-to-talk interaction rules. Set `RequireTurnCompletionBeforeNextPress = true` for most training simulations — it enforces a natural dialogue rhythm where the character finishes before the learner responds. | Field | Type | Default | Description | | -------------------------------------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EnableServerSttToggle` | `bool` | `true` | Mutes and unmutes backend speech-to-text when the push-to-talk control is pressed and released. Reduces server cost and prevents accidental processing of background audio. | | `InterruptBotOnPress` | `bool` | `true` | If the character is speaking when the user presses push-to-talk, the character is interrupted immediately so the user can start talking. | | `RequireTurnCompletionBeforeNextPress` | `bool` | `true` | The user must wait for the character to finish its full response before pressing push-to-talk again. Prevents overlapping turns. | | `TurnCompletionTimeoutMs` | `int` | `5000` | Fallback timeout in milliseconds. If the character's turn-complete event never arrives (e.g., a network hiccup), this releases the push-to-talk lock after the timeout. | | `AllowSpeechStoppedFallbackAfterSpeechStart` | `bool` | `false` | If enabled, a speech-stopped event from the character can also release the push-to-talk waiting state after speech has actually started. Useful for recovering from edge cases where the turn-complete event is missed. | ```csharp var options = new TurnTakingOptions { Mode = ConversationInputMode.PushToTalk, PushToTalkPolicy = new PushToTalkPolicy { InterruptBotOnPress = false, // let the character finish before the user can speak RequireTurnCompletionBeforeNextPress = true, TurnCompletionTimeoutMs = 8000 } }; ``` *** ### Local audio policy `LocalAudioPolicy` controls microphone behavior on the local device. It applies to both Hands-Free and Push-to-Talk modes. | Field | Type | Default | Description | | -------------------------------- | -------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `StartMutedInPushToTalk` | `bool` | `true` | The microphone starts muted when push-to-talk mode is active. Audio is only captured while the push-to-talk control is held. | | `EnableAcousticEchoCancellation` | `bool` | `false` | Opt in to acoustic echo cancellation. Intended for Android and iOS when using device speakers (speakerphone) instead of headphones. | | `PushToTalkStartupMode` | `PushToTalkMicStartupMode` | `PrewarmMuted` | Controls how the microphone is initialized when push-to-talk mode starts. | #### `PushToTalkMicStartupMode` | Value | Behavior | Trade-Off | | ------------------ | ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `PrewarmMuted` | The microphone is opened and warmed up at session start, but kept muted until the user presses the control. | Eliminates the delay on the first press; uses a small amount of background resources. | | `OpenOnFirstPress` | The microphone is not opened until the user presses push-to-talk for the first time. | Saves resources; introduces a brief delay (\~100–300 ms) on the first press as the mic initializes. | ```csharp var options = new TurnTakingOptions { Mode = ConversationInputMode.PushToTalk, LocalAudioPolicy = new LocalAudioPolicy { EnableAcousticEchoCancellation = true, // factory floor scenario, device speakers PushToTalkStartupMode = PushToTalkMicStartupMode.PrewarmMuted } }; ``` *** ### `ServerSttInitialState` Controls whether Convai's speech-to-text is enabled at the moment the session starts. | Value | Behavior | | ------------ | --------------------------------------------------------------------------------------------------------------------------------- | | `UseDefault` | Server-side default: STT enabled for Hands-Free, disabled for Push-to-Talk. | | `Enabled` | STT starts enabled regardless of mode. | | `Disabled` | STT starts disabled regardless of mode. This is an advanced option — manual STT control is not exposed in the current public API. | *** ### Runtime mode switching Switch between Hands-Free and Push-to-Talk without disconnecting the session: ```csharp // Switch to push-to-talk mid-session await _manager.SetConversationInputModeAsync(ConversationInputMode.PushToTalk); ``` `SetConversationInputModeAsync` returns an `IConvaiOperation`. The active mode after the switch is available via `_manager.ActiveConversationInputMode`. {% hint style="warning" %} Runtime mode switching applies the new `LocalAudioPolicy` defaults for the new mode. If you switch to Push-to-Talk, the microphone will be muted according to `StartMutedInPushToTalk`. The session does not reconnect. {% endhint %} *** ### Usage examples #### Example 1: Medical training simulator — hands-free with tight turn detection A learner performs a patient assessment. The AI character responds as the patient. Shorter silence threshold keeps the conversation moving. ```csharp var options = new TurnTakingOptions { Mode = ConversationInputMode.HandsFree, TurnDetection = TurnDetectionMode.Custom, CustomTurnDetection = new SmartTurnSettings { StopSecs = 2.0f, PreSpeechMs = 80, MaxDurationSecs = 12.0f // learners can give longer answers } }; ``` **Expected outcome:** The character responds about 2 seconds after the learner stops speaking. Longer responses from the learner (describing symptoms, asking questions) are captured up to 12 seconds. *** #### Example 2: Factory safety drill — push-to-talk with echo cancellation A safety trainer interacts with an AI safety officer in a noisy plant simulation. Push-to-talk prevents ambient noise from triggering unintended turns. Speakers are used, so AEC is enabled. ```csharp var options = new TurnTakingOptions { Mode = ConversationInputMode.PushToTalk, PushToTalkPolicy = new PushToTalkPolicy { InterruptBotOnPress = true, RequireTurnCompletionBeforeNextPress = false, // urgency override for safety scenarios TurnCompletionTimeoutMs = 6000 }, LocalAudioPolicy = new LocalAudioPolicy { EnableAcousticEchoCancellation = true, PushToTalkStartupMode = PushToTalkMicStartupMode.PrewarmMuted } }; ``` **Expected outcome:** The trainer can interrupt the AI at any time by pressing the button. No echo feedback from device speakers. *** #### Example 3: Runtime toggle between modes via UI button A scenario that starts hands-free but lets facilitators switch to push-to-talk during a live session. ```csharp public class InputModeToggle : MonoBehaviour { [SerializeField] private ConvaiManager _manager; private bool _isPushToTalk; public async void ToggleMode() { _isPushToTalk = !_isPushToTalk; var mode = _isPushToTalk ? ConversationInputMode.PushToTalk : ConversationInputMode.HandsFree; await _manager.SetConversationInputModeAsync(mode); } } ``` **Expected outcome:** Mode switches mid-session without interrupting the connection. Check `_manager.ActiveConversationInputMode` to confirm the active mode after the switch. *** ### Troubleshooting | Symptom | Likely Cause | Fix | | ----------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Character responds mid-sentence before the user finishes speaking | `StopSecs` too low for the pacing of this scenario | Increase `StopSecs` to 2.5–3.0 in `SmartTurnSettings` | | First word of the user's turn is clipped | Speech onset is captured too late | Increase `PreSpeechMs` to 80–150 ms in `SmartTurnSettings` | | Push-to-talk button stays locked after the character finishes | Turn-complete event was not received (network hiccup) | `TurnCompletionTimeoutMs` releases the lock after timeout; lower the value, or set `AllowSpeechStoppedFallbackAfterSpeechStart = true` | | Background noise triggers responses in Hands-Free mode | Environment too noisy for automatic voice detection | Switch to Push-to-Talk mode, or increase `StopSecs` | | Brief delay on first push-to-talk press | `PushToTalkMicStartupMode` is `OpenOnFirstPress` — mic initializes on first press | Switch to `PrewarmMuted` | *** ### Next steps You now have the full field reference for turn-taking configuration. Read Event System next to learn how to react to speech, transcript, and emotion events at runtime. {% content-ref url="/pages/bcd284a409f00820e1693ab998dc7a99a4fbd780" %} [Event system](/api-docs/plugins-and-integrations/convai-unity-sdk/core-concepts/event-system) {% endcontent-ref %} {% content-ref url="/pages/d5302389270319f4ca9bf82b6b3081d22b81bd8f" %} [Features](/api-docs/plugins-and-integrations/convai-unity-sdk/features) {% endcontent-ref %} # Event system Reference for Convai event relay components — available events, payload fields, subscription patterns, and the ConvaiNotificationEventBridge service. The Convai SDK communicates what happens during a session — connections, character speech, transcripts, emotions — through a set of relay components. Add one of these MonoBehaviours to a GameObject in your scene, wire up UnityEvents in the Inspector or subscribe in code, and your scene logic responds to whatever the SDK broadcasts. *** ### Two wiring approaches {% tabs %} {% tab title="Inspector (UnityEvents)" %} 1. Add the relay component to any GameObject in your scene via **Add Component → Convai → Events**. 2. Assign the required reference (`ConvaiManager` or `ConvaiCharacter`) in the Inspector, or enable **Auto Resolve** to let the component find it automatically. 3. Wire handlers to the UnityEvent fields in the Inspector — no code required. Best for: connection indicators, animation triggers, UI visibility toggles — anything driven by a single event without conditional logic. {% endtab %} {% tab title="C# Scripting" %} Subscribe to relay component events from code: ```csharp public class MyHandler : MonoBehaviour { [SerializeField] private ConvaiCharacterEventRelay _relay; private void OnEnable() { _relay.OnEmotionChanged.AddListener(HandleEmotion); _relay.OnSpeechStarted.AddListener(HandleSpeechStarted); } private void OnDisable() { _relay.OnEmotionChanged.RemoveListener(HandleEmotion); _relay.OnSpeechStarted.RemoveListener(HandleSpeechStarted); } private void HandleEmotion(CharacterEmotionRelayData data) { /* … */ } private void HandleSpeechStarted() { /* … */ } } ``` Best for: conditional logic, multi-event coordination, data routing across multiple systems. {% endtab %} {% endtabs %} *** ### Relay component quick-reference | Component | Inspector Menu Path | Use When | | ---------------------------- | ------------------------------------------- | --------------------------------------------------------------------------- | | `ConvaiSessionEventRelay` | Convai/Events/Convai Session Event Relay | Tracking session connection state, handling errors, driving connection UI | | `ConvaiCharacterEventRelay` | Convai/Events/Convai Character Event Relay | Reacting to a specific character's speech, transcript, turn, and emotion | | `ConvaiTranscriptEventRelay` | Convai/Events/Convai Transcript Event Relay | Scene-wide transcript feed with optional filtering by character or finality | *** ### `ConvaiSessionEventRelay` Tracks the session lifecycle for the entire scene. Add one per scene — it monitors the session managed by `ConvaiManager`. {% hint style="info" %} If `ConvaiManager` initializes after the relay's `OnEnable` (for example, due to script execution order), the relay retries its subscription automatically in `LateUpdate()` while enabled. No manual retry logic is needed. {% endhint %} **Inspector fields:** | Field | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `Manager` | Reference to the `ConvaiManager` in the scene. | | `AutoResolveManager` | When enabled, the component finds `ConvaiManager.ActiveManager` at runtime. Disable if you have multiple managers or need explicit binding. | **Events:** | Event | Payload | When It Fires | | ----------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | `OnConnected` | — | Initial connection established (`Connecting` → `Connected`). Does not fire on reconnection — see `OnReconnected`. | | `OnDisconnected` | — | Session enters `Disconnected` state. | | `OnReconnecting` | — | A reconnect attempt begins (session was `Connected`, connection dropped). | | `OnReconnected` | — | A reconnect attempt succeeded. Session is `Connected` again. | | `OnUsageLimitReached` | — | The API usage quota for the account has been exceeded. | | `OnSessionStateChanged` | `SessionStateChangedRelayData` | Any session state transition. Fires for every state change. | | `OnSessionError` | `SessionErrorRelayData` | An error event is received from the session. | #### `SessionStateChangedRelayData` | Property | Type | Description | | -------------------------- | -------------- | --------------------------------------------------------------------- | | `OldState` | `SessionState` | State before the transition. | | `NewState` | `SessionState` | State after the transition. | | `SessionId` | `string` | Current session identifier. Empty if no session is active. | | `ErrorCode` | `string` | Error code if the transition was caused by an error. Empty otherwise. | | `IsError` | `bool` | Computed: `NewState == Error`. | | `IsReconnecting` | `bool` | Computed: `NewState == Reconnecting`. | | `IsConnectionEstablished` | `bool` | Computed: `OldState == Connecting && NewState == Connected`. | | `IsReconnectionSuccessful` | `bool` | Computed: `OldState == Reconnecting && NewState == Connected`. | | `IsDisconnected` | `bool` | Computed: `NewState == Disconnected`. | #### `SessionErrorRelayData` | Property | Type | Description | | ------------------- | ------------------- | ---------------------------------------------------------- | | `ErrorCode` | `string` | Machine-readable error code. | | `Message` | `string` | Human-readable error description. | | `SessionId` | `string` | Session identifier at the time of the error. | | `IsRecoverable` | `bool` | Whether the SDK will attempt to recover automatically. | | `Stage` | `SessionErrorStage` | Where in the connection lifecycle the error occurred. | | `HttpStatusCode` | `int` | HTTP status code if the error originated from an API call. | | `HasHttpStatusCode` | `bool` | Whether `HttpStatusCode` contains a meaningful value. | `SessionErrorStage` values: `Unknown`, `Configuration`, `ConnectApi`, `Transport`, `SessionRecovery`, `Runtime`. **Code example — show a connection status indicator:** ```csharp public class ConnectionIndicator : MonoBehaviour { [SerializeField] private ConvaiSessionEventRelay _relay; [SerializeField] private GameObject _connectingOverlay; private void OnEnable() { _relay.OnConnected.AddListener(OnConnected); _relay.OnDisconnected.AddListener(OnDisconnected); _relay.OnReconnecting.AddListener(OnReconnecting); } private void OnDisable() { _relay.OnConnected.RemoveListener(OnConnected); _relay.OnDisconnected.RemoveListener(OnDisconnected); _relay.OnReconnecting.RemoveListener(OnReconnecting); } private void OnConnected() => _connectingOverlay.SetActive(false); private void OnDisconnected() => _connectingOverlay.SetActive(true); private void OnReconnecting() => _connectingOverlay.SetActive(true); } ``` *** ### `ConvaiCharacterEventRelay` Tracks events for a single `ConvaiCharacter`. Add one per character that needs to drive scene responses. **Inspector fields:** | Field | Description | | ---------------------- | ----------------------------------------------------------------------------------------------- | | `Character` | Reference to the `ConvaiCharacter` this relay monitors. | | `AutoResolveCharacter` | When enabled, the component searches for `ConvaiCharacter` on the same GameObject as the relay. | **Events:** | Event | Payload | When It Fires | | ---------------------- | --------------------------------- | ----------------------------------------------------------------- | | `OnTranscriptReceived` | `CharacterTranscriptRelayData` | Each transcript chunk arrives — both interim (partial) and final. | | `OnSpeechStarted` | — | The character begins speaking (audio starts playing). | | `OnSpeechStopped` | — | The character stops speaking (audio ends). | | `OnTurnCompleted` | `CharacterTurnCompletedRelayData` | The character's full response for one turn is complete. | | `OnCharacterReady` | — | The character is fully initialized and connected to the session. | | `OnEmotionChanged` | `CharacterEmotionRelayData` | A new emotion signal is received from Convai. | #### `CharacterTranscriptRelayData` | Property | Type | Description | | --------------- | -------- | -------------------------------------------------------------- | | `CharacterId` | `string` | The character's ID. | | `CharacterName` | `string` | The character's display name. | | `Text` | `string` | The transcript text. May be partial if `IsFinal` is false. | | `IsFinal` | `bool` | Whether this is the committed final transcript for this chunk. | #### `CharacterTurnCompletedRelayData` | Property | Type | Description | | ---------------- | -------- | ------------------------------------------------------------------ | | `CharacterId` | `string` | The character's ID. | | `CharacterName` | `string` | The character's display name. | | `WasInterrupted` | `bool` | Whether the turn ended because the user interrupted the character. | #### `CharacterEmotionRelayData` | Property | Type | Description | | --------------- | -------- | ------------------------------------------------------------------------------------------- | | `CharacterId` | `string` | The character's ID. | | `CharacterName` | `string` | The character's display name. | | `Emotion` | `string` | The emotion name (e.g., `"joy"`, `"fear"`, `"sadness"`). See the Emotion feature reference. | | `Intensity` | `int` | Emotion intensity (0–100). | **Code example — trigger an animation on emotion change:** ```csharp public class CharacterEmotionAnimator : MonoBehaviour { [SerializeField] private ConvaiCharacterEventRelay _relay; [SerializeField] private Animator _animator; private static readonly int EmotionHash = Animator.StringToHash("Emotion"); private void OnEnable() => _relay.OnEmotionChanged.AddListener(HandleEmotion); private void OnDisable() => _relay.OnEmotionChanged.RemoveListener(HandleEmotion); private void HandleEmotion(CharacterEmotionRelayData data) { _animator.SetTrigger(data.Emotion); _animator.SetFloat("EmotionIntensity", data.Intensity / 100f); } } ``` *** ### `ConvaiTranscriptEventRelay` Provides a scene-wide transcript feed. Unlike `ConvaiCharacterEventRelay`, this relay monitors all characters and the player through a single component. Use it to drive subtitle UI, session logs, or assessment systems. **Inspector fields:** | Field | Type | Default | Description | | ---------------------- | --------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Manager` | `ConvaiManager` | — | The `ConvaiManager` to monitor. | | `AutoResolveManager` | `bool` | — | Find `ActiveManager` automatically. | | `FinalOnly` | `bool` | `false` | When enabled, only final (committed) transcripts raise events. Interim partial transcripts are suppressed. | | `IgnoreInterimUpdates` | `bool` | `true` | Suppress interim transcript updates. Final updates still pass through. Disable this field if your UI needs to display partial text as the character speaks. | | `CharacterIdFilter` | `string` | `""` | If set, only transcripts from the character with this ID raise events. Leave empty for all characters. | **Events:** | Event | Payload | When It Fires | | ------------------------------------ | ------------------------------ | ------------------------------------------------------------------------ | | `OnCharacterTranscriptReceived` | `CharacterTranscriptRelayData` | Any character transcript (subject to filter and `IgnoreInterimUpdates`). | | `OnPlayerTranscriptReceived` | `PlayerTranscriptRelayData` | Any player transcript. | | `OnFinalCharacterTranscriptReceived` | `CharacterTranscriptRelayData` | Final character transcript only, regardless of `FinalOnly` setting. | | `OnFinalPlayerTranscriptReceived` | `PlayerTranscriptRelayData` | Final player transcript only. | #### `PlayerTranscriptRelayData` | Property | Type | Description | | --------------- | -------- | ------------------------------------------------------------------------------- | | `PlayerId` | `string` | The local player's identifier. | | `PlayerName` | `string` | The player's display name. | | `SpeakerId` | `string` | The speaker identifier (may differ from `PlayerId` in multi-participant rooms). | | `SpeakerName` | `string` | The speaker's display name. | | `ParticipantId` | `string` | Room participant identifier. | | `TurnId` | `string` | Identifies the turn this transcript chunk belongs to. | | `MessageId` | `string` | Unique identifier for this transcript message. | | `Text` | `string` | Transcript text. May be partial if `IsFinal` is false. | | `IsFinal` | `bool` | Whether this is the committed final transcript. | **Code example — multi-character transcript feed for a training log:** ```csharp public class TrainingTranscriptLog : MonoBehaviour { [SerializeField] private ConvaiTranscriptEventRelay _relay; [SerializeField] private TMP_Text _logText; private readonly System.Text.StringBuilder _log = new(); private void OnEnable() { _relay.OnFinalCharacterTranscriptReceived.AddListener(OnCharacterLine); _relay.OnFinalPlayerTranscriptReceived.AddListener(OnPlayerLine); } private void OnDisable() { _relay.OnFinalCharacterTranscriptReceived.RemoveListener(OnCharacterLine); _relay.OnFinalPlayerTranscriptReceived.RemoveListener(OnPlayerLine); } private void OnCharacterLine(CharacterTranscriptRelayData data) { _log.AppendLine($"[{data.CharacterName}]: {data.Text}"); _logText.text = _log.ToString(); } private void OnPlayerLine(PlayerTranscriptRelayData data) { _log.AppendLine($"[Learner]: {data.Text}"); _logText.text = _log.ToString(); } } ``` *** ### Subscription lifecycle Relay MonoBehaviour components manage their own subscriptions automatically. They subscribe when `OnEnable` runs and unsubscribe when `OnDisable` runs. When subscribing via C#, follow the same pattern: ```csharp private void OnEnable() => _relay.OnConnected.AddListener(MyHandler); private void OnDisable() => _relay.OnConnected.RemoveListener(MyHandler); ``` {% hint style="warning" %} Do not subscribe in `Start()` without a matching unsubscribe in `OnDestroy()`. Relay components can be disabled and re-enabled; a subscription from `Start()` without cleanup will result in duplicate handlers or null-reference errors after the relay is disabled. {% endhint %} *** ### `ConvaiNotificationEventBridge` `ConvaiNotificationEventBridge` is not a relay component. It is an internal service that bridges session error domain events to the notification UI system, with cooldown deduplication to prevent the same error notification from appearing repeatedly. | Property | Type | Default | Description | | ----------------- | ------- | ------- | ----------------------------------------------------------- | | `CooldownSeconds` | `float` | `10` | Minimum seconds between showing the same notification type. | Most projects never interact with this class directly. It is instantiated and managed by the SDK bootstrap. If you are building a custom notification system using `IConvaiNotificationService`, you may use `ConvaiNotificationEventBridge` to integrate session error events into your system. {% hint style="info" %} `ConvaiNotificationEventBridge` is not added to the scene via **Add Component**. It is instantiated programmatically during SDK startup. {% endhint %} *** ### Usage examples #### Example 1: Training simulation — connection overlay Show a "Connecting…" overlay while the session is not yet established. ```csharp [SerializeField] private ConvaiSessionEventRelay _sessionRelay; [SerializeField] private CanvasGroup _loadingOverlay; private void OnEnable() { _sessionRelay.OnConnected.AddListener(OnConnected); _sessionRelay.OnDisconnected.AddListener(OnDisconnected); _sessionRelay.OnReconnecting.AddListener(OnReconnecting); } private void OnDisable() { _sessionRelay.OnConnected.RemoveListener(OnConnected); _sessionRelay.OnDisconnected.RemoveListener(OnDisconnected); _sessionRelay.OnReconnecting.RemoveListener(OnReconnecting); } private void OnConnected() => _loadingOverlay.alpha = 0f; private void OnDisconnected() => _loadingOverlay.alpha = 1f; private void OnReconnecting() => _loadingOverlay.alpha = 0.5f; ``` **Expected outcome:** The overlay fades in when the session is not connected and fades out when the connection is established. *** #### Example 2: Medical trainer — emotion-triggered character response A patient character's facial expression and posture change based on the emotion detected by Convai. ```csharp [SerializeField] private ConvaiCharacterEventRelay _patientRelay; [SerializeField] private PatientExpressionController _expressionController; private void OnEnable() => _patientRelay.OnEmotionChanged.AddListener(ApplyEmotion); private void OnDisable() => _patientRelay.OnEmotionChanged.RemoveListener(ApplyEmotion); private void ApplyEmotion(CharacterEmotionRelayData data) { _expressionController.SetExpression(data.Emotion, data.Intensity / 100f); } ``` **Expected outcome:** The patient character's visual expression updates in real time as emotion signals arrive from Convai. *** #### Example 3: Shared transcript feed filtered to one character A corporate onboarding simulation has multiple NPC characters but only the main instructor's lines appear in the subtitle panel. On the `ConvaiTranscriptEventRelay` component in the Inspector: * Set `CharacterIdFilter` to the instructor character's ID (e.g., `"abc123"`). * Enable `FinalOnly` to show only committed transcript lines. * Wire `OnFinalCharacterTranscriptReceived` to your subtitle UI. **Expected outcome:** Only the instructor's completed sentences appear in the subtitle panel. Other characters in the scene do not affect the UI. *** ### Troubleshooting | Symptom | Likely Cause | Fix | | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Relay fires no events after the scene starts | `ConvaiManager` was not initialized before the relay's `OnEnable` ran | No action needed — the relay retries in `LateUpdate()` while enabled. Verify `ConvaiManager` is present and active in the scene. | | `ConvaiCharacterEventRelay` fires no events | `ConvaiCharacter` not found on the assigned GameObject | Verify `ConvaiCharacter` is on the **same** GameObject as the relay, or assign the reference explicitly. `AutoResolveCharacter` searches the same GameObject only — not parents. | | Interim transcript updates not arriving | `IgnoreInterimUpdates` is `true` by default | Set `IgnoreInterimUpdates = false` on `ConvaiTranscriptEventRelay` to receive partial transcript updates. | | Event handler fires multiple times for a single event | Handler subscribed in `Start()` without cleanup; relay was disabled and re-enabled | Move subscription to `OnEnable()` and unsubscribe in `OnDisable()`. | | `OnCharacterTranscriptReceived` not firing for an expected character | `CharacterIdFilter` is set to a different character ID | Clear `CharacterIdFilter` or set it to the correct character ID. | *** ### Next steps You now have the full reference for all relay components, event payloads, and subscription patterns. Proceed to the Features section to explore individual SDK capabilities. {% content-ref url="/pages/d5302389270319f4ca9bf82b6b3081d22b81bd8f" %} [Features](/api-docs/plugins-and-integrations/convai-unity-sdk/features) {% endcontent-ref %} # Features Features are the SDK's AI-powered capability modules. Each feature connects to Convai for a specific purpose — executing in-scene behaviors, expressing emotions, remembering users across sessions, following narrative graphs, or seeing the world through a camera. Features have dedicated module systems, ScriptableObject profiles, and backend integration. Select a feature to get started:
Actions
Let characters perform in-scene behaviors — move, pick up, animate, trigger events — in response to natural language commands.		/pages/8e2d39906eb40512aa9c4c6b05ffc3352eb87e51
Dynamic Context
Feed live runtime state — environment conditions, player status, simulation events — directly into a character's awareness mid-conversation.		/pages/HLRHVibG6Nqvot1DGfvc
Scene Metadata
Automatically register scene objects with Convai so characters understand the physical world around them without manual context injection.		/pages/a62866eb2b9826eb452e50eb8875192eb2464f3c
Emotion
Translate Convai's emotional AI signals into real-time facial animation via blendshapes and Animator parameters.		/pages/pzyAnwgRmeSToQBXFJYu
Long-Term Memory
Give characters cross-session recall — they remember individual users, facts, and past interactions over time.		/pages/Yg6rMK5Mnh9zStBOwoca
Narrative Design
Structure conversations around authored story graphs with section-based behavior changes and trigger-driven progression.		/pages/vN4t84KHA949ufTTNaVd
Vision
Stream live video from scene cameras or webcams to Convai, enabling characters to see and respond to real-time visual input.		/pages/c776d88f01e63c32eee310e37d0ed10bc12eb355
# Character actions Find quick-start guides, executor references, dispatcher configuration, scripting API, and usage examples for the Convai character actions system. Character actions enable NPC characters to respond to player requests with physical in-scene behaviors — navigation, object interaction, animations, and custom gameplay logic. This section covers everything from a first working setup to the complete scripting API.
How character actions work
Understand the pipeline, key concepts, required components, and built-in executors.		/pages/mqkxc4D6OCQA9OVVDCLv
Character actions quick start
Add a working Move To action to your NPC — no scripting required.		/pages/M9Lh8OZ24Oqv9SmxMjzb
Configure character actions
Full reference for ConvaiActionConfigSource — action definitions, targets, and connect-time overrides.		/pages/u3JAohXiqBNMG6zZ5m60
Action executors
Reference for all six executor components with Inspector field tables.		/pages/6RYF3LHemCYzxRfZpueZ
Dispatcher and batch policies
Configure how ConvaiActionDispatcher handles concurrent batches and step failures.		/pages/L2RNE4SlLqEb39SpjbDU
Write a custom action executor
Implement IConvaiActionExecutor to create project-specific behaviors for any action.		/pages/yKmOPjUYIRCR4eqPYD0A
Attention and reference grounding
Keep Convai's target resolution aligned with what the player is focused on at runtime.		/pages/RKSqmXPbQonS4iJeIfiO
Character actions scripting reference
Complete public API for all action system types, events, enums, and components.		/pages/73HufYIPX0JRnnAsGDbk
Character actions examples
Four progressive scenarios from no-code Inspector setup to scripted batch injection.		/pages/f7muso7lrEOfJU9hnYeK
Troubleshoot character actions
Diagnose action pipeline issues with ConvaiActionDebugProbe and a symptom/cause/fix reference.		/pages/LZcDaILUbaCK85OVcPJ6
# How character actions work Understand the Convai character actions pipeline — how the backend selects actions, how Unity resolves targets, and which components are required. The Convai character actions system lets NPC characters respond to player requests by performing physical behaviors in your scene. When a trainee says "retrieve the fire extinguisher," the character navigates to it. When a student says "point at the diagram," the character turns and faces it. The backend identifies what to do and who to target; Unity executes the behavior through a simple, extensible pipeline. ### How the action pipeline works Every action request travels through four stages: ```mermaid graph LR A["Player speaks or types"] --> B["Convai identifies action + target"] B --> C["ConvaiCharacter receives command batch"] C --> D["ConvaiActionDispatcher resolves and executes"] D --> E["Executor runs in-scene behavior"] ``` The Convai backend selects the action name and optional target from the affordances you registered at connect time. Unity resolves that target to a scene `GameObject` and runs the bound executor component. ### Key concepts | Concept | What it means | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Action affordances** | Which action names the backend is allowed to request. Authored in `ConvaiActionConfigSource` or overridden at connect time. | | **Action targets** | Which objects and characters the backend is allowed to reference. Also authored in `ConvaiActionConfigSource`. | | **Action events** | The ordered command batch the backend returns for a turn. Exposed via `ConvaiCharacter.OnActionsReceived`. | | **Local execution** | Optional Unity-side execution through `ConvaiActionDispatcher` and `IConvaiActionExecutor`. You can receive raw action events without the dispatcher if you want to handle them yourself. | ### Required components | Component | Required | Purpose | | ------------------------------- | ------------------- | --------------------------------------------------------------- | | `ConvaiCharacter` | Always | Receives action command batches from Convai | | `ConvaiActionConfigSource` | Yes | Authors connect-time affordances (actions, objects, characters) | | `ConvaiActionDispatcher` | Optional | Executes received batches automatically through bound executors | | One or more executor components | If using dispatcher | Performs the actual in-scene behavior | {% hint style="info" %} `ConvaiActionDispatcher` is optional. If you want to handle action batches in your own gameplay code, subscribe to `ConvaiCharacter.OnActionsReceived` directly and skip the dispatcher entirely. {% endhint %} ### Executors Six executor components ship with the Convai SDK: | Executor | Behavior | | ------------------------------- | ----------------------------------------------------------------------------------------- | | `LookAtTargetActionExecutor` | Smoothly rotates the NPC to face a target over a configurable duration | | `UnityEventActionExecutor` | Fires a `UnityEvent` — connects any action to Inspector-wired callbacks without scripting | | `TransformMoveToActionExecutor` | Instantly snaps the NPC to the target position — prototype use only | | `NavMeshMoveToActionExecutor` | Drives a `NavMeshAgent` to the target using pathfinding | | `AnimatorTriggerActionExecutor` | Maps action names to Animator triggers via a configurable binding list | | `PickUpActionExecutor` | Compound: navigate to target → trigger animation → attach object to hand | {% hint style="warning" %} `TransformMoveToActionExecutor` teleports the character instantly with no animation or pathfinding. Use it only for rapid prototyping. Replace it with `NavMeshMoveToActionExecutor` or a custom executor before shipping to players. {% endhint %} ### Next steps To get a working action set up in your scene, start with the quick-start guide. Once your first action runs end-to-end, read the configuration reference to understand the full `ConvaiActionConfigSource` options, then choose or build the right executor for your project. {% content-ref url="/pages/M9Lh8OZ24Oqv9SmxMjzb" %} [Character actions quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/quick-start) {% endcontent-ref %} {% content-ref url="/pages/u3JAohXiqBNMG6zZ5m60" %} [Configure character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/configuring-actions) {% endcontent-ref %} {% content-ref url="/pages/6RYF3LHemCYzxRfZpueZ" %} [Action executors](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/action-executors) {% endcontent-ref %} # Character actions quick start Add a working Move To action to your NPC using ConvaiActionConfigSource, ConvaiActionDispatcher, and TransformMoveToActionExecutor without writing any scripts. This guide walks you through connecting a "Move To" action so your NPC navigates to a scene object when the player asks. By the end, your character responds to natural language requests like "go to the crate" by physically moving to it in the scene — no code required. ### Prerequisites Before starting, verify: * [ ] A `ConvaiCharacter` component is already on your NPC's `GameObject` * [ ] Your scene has at least one target object the NPC should be able to reach ### Configure the action pipeline {% stepper %} {% step %} #### Add ConvaiActionConfigSource Select your NPC's `GameObject`. In the Inspector, click **Add Component** and search for **Convai Action Config Source** (`Convai/Convai Action Config Source`). You should see four new sections in the Inspector: **Action Definitions**, **Actionable Objects**, **Actionable Characters**, and **Initial Attention**.
Unity Inspector showing ConvaiActionConfigSource added to the NPC GameObject with the four configuration sections: Action Definitions, Actionable Objects, Actionable Characters, and Initial Attention

ConvaiActionConfigSource added to the NPC — the four sections are now available to configure which actions, objects, and characters the backend can reference at runtime.

{% endstep %} {% step %} #### Define the Move To action In the **Action Definitions** list, click the **+** button to add an entry, then fill in the three fields: | Field | Value | | ---------------------- | ----------------------------------------------------------- | | **Action Name** | `Move To` | | **Target Requirement** | `Object` | | **Executor** | *(leave empty for now — you'll assign it in the next step)* | Leave **Timeout Seconds** at `0` (no timeout).
Unity Inspector showing a Move To action definition in ConvaiActionConfigSource with Action Name set to Move To and Target Requirement set to Object

Move To action definition — Action Name matches the command string Convai sends; Target Requirement Object tells the dispatcher to resolve a scene object target before calling the executor.

{% endstep %} {% step %} #### Register a target object In the **Actionable Objects** list, click **+** and fill in: | Field | Value | | ------------------------ | ---------------------------------------------- | | **Name** | `Crate` | | **Description** | A wooden storage crate near the left workbench | | **GameObject Reference** | Drag your target object here | The **Description** is sent to Convai and helps the backend resolve vague references like "that box" or "the thing by the bench." Write it as a natural sentence that places the object in context.
Unity Inspector showing the Actionable Objects list on ConvaiActionConfigSource with a Crate entry including a descriptive name, description text, and a dragged-in GameObject reference

Crate registered as an Actionable Object — the Description is sent to Convai and used for natural-language reference resolution; the GameObject Reference stays local and is never transmitted.

{% endstep %} {% step %} #### Add TransformMoveToActionExecutor On the same NPC `GameObject`, click **Add Component** and search for **Transform Move To Action Executor** (`Convai/Samples/Transform Move To Action Executor`). Go back to the **Action Definitions** entry you created in the previous step. Drag the `TransformMoveToActionExecutor` component into the **Executor** field.
Unity Inspector showing TransformMoveToActionExecutor added to the NPC and its component reference assigned to the Executor field in the Move To action definition

TransformMoveToActionExecutor assigned as the Move To executor — this completes the action-to-behavior binding that the dispatcher uses when Convai selects the Move To action.

Executor field on the Move To action definition with TransformMoveToActionExecutor dragged in

Drag TransformMoveToActionExecutor into the Executor field on the Move To action definition.

{% hint style="warning" %} `TransformMoveToActionExecutor` is for prototyping only. It teleports the character instantly with no animation or pathfinding. Replace it with `NavMeshMoveToActionExecutor` or a custom executor before shipping to players. {% endhint %} {% endstep %} {% step %} #### Add ConvaiActionDispatcher On the same NPC `GameObject`, click **Add Component** and search for **Convai Action Dispatcher** (`Convai/Convai Action Dispatcher`). Leave **Batch Policy** at `Queue` and **Failure Policy** at `Stop Batch` — these are the correct defaults for most scenarios.
Unity Inspector showing ConvaiActionDispatcher added to the NPC GameObject with Batch Policy set to Queue and Failure Policy set to Stop Batch

ConvaiActionDispatcher added — Queue batch policy and Stop Batch failure policy are the correct defaults for most scenarios and require no further configuration for this quick start.

{% endstep %} {% endstepper %} ### Verify the setup Your NPC's `GameObject` should now have these four components: ``` ConvaiCharacter ConvaiActionConfigSource ← action definitions + target objects ConvaiActionDispatcher ← receives and executes batches TransformMoveToActionExecutor ← performs the move behavior ``` Enter Play Mode and say "go to the crate" or "move to the crate." Your NPC should teleport to the crate's position. If you added `ConvaiActionDebugProbe` (optional — see [Troubleshoot character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/debugging-and-troubleshooting)), you should also see this in the Console: ``` [ConvaiActionDebugProbe] Step succeeded #1: cmd='Move To crate', def='Move To', target=Object:Crate ``` If the NPC does not move, check [Troubleshoot character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/debugging-and-troubleshooting) for the diagnostic checklist. ### Runtime behavior **Action names are case-insensitive.** `Move To`, `move to`, and `MOVE TO` all match the same definition. Spaces are preserved — `Move To` and `MoveTO` do not match. **Configuration is sent once at session start.** If you add, rename, or remove actions or targets while in Play Mode, end the session and reconnect for the changes to take effect. ### Next steps {% content-ref url="/pages/6RYF3LHemCYzxRfZpueZ" %} [Action executors](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/action-executors) {% endcontent-ref %} {% content-ref url="/pages/yKmOPjUYIRCR4eqPYD0A" %} [Write a custom action executor](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/writing-custom-executors) {% endcontent-ref %} {% content-ref url="/pages/u3JAohXiqBNMG6zZ5m60" %} [Configure character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/configuring-actions) {% endcontent-ref %} # Configure character actions Reference for ConvaiActionConfigSource — action definitions, target objects, actionable characters, and scripted connect-time overrides. `ConvaiActionConfigSource` is the Inspector authoring surface for everything Convai needs to know about your NPC's action capabilities at connect time: which actions to allow, which scene objects the backend can reference, which characters are targetable, and which object has the NPC's initial attention. Add it to any `GameObject` that already has `ConvaiCharacter`. ### Component overview | Attribute | Value | | --------------- | ---------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Convai Action Config Source` | | **Namespace** | `Convai.Runtime.Components` | | **Constraints** | `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` | The component has four Inspector sections: | Section | Purpose | | ------------------------- | --------------------------------------------------------------- | | **Action Definitions** | Maps backend action names to Unity executor components | | **Actionable Objects** | Scene objects the backend may reference as action targets | | **Actionable Characters** | Other characters the backend may reference as action targets | | **Initial Attention** | The object name the NPC focuses on at the start of each session |
ConvaiActionConfigSource in the Unity Inspector showing all four sections: Action Definitions, Actionable Objects, Actionable Characters, and Initial Attention

ConvaiActionConfigSource Inspector — all four sections visible. Each section maps to a distinct part of the connect-time payload sent to Convai.

### Action definitions Each entry in the **Action Definitions** list binds one backend action name to a Unity executor component. #### Action definition fields | Field | Type | Description | | ------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------- | | `ActionName` | `string` | The name Convai sends when it selects this action. Case-insensitive at runtime; spaces are significant. | | `TargetRequirement` | `ConvaiActionTargetRequirement` | Whether this action requires a target and what kind. | | `Executor` | `MonoBehaviour` | The component that performs the behavior. Must implement `IConvaiActionExecutor`. | | `TimeoutSeconds` | `float` | Maximum seconds the executor may run before it is automatically canceled. `0` = no timeout. | #### Target requirement values | Value | Meaning | | ----------- | ------------------------------------------------------ | | `None` | Action does not reference a target object or character | | `Object` | Action requires a resolved object target | | `Character` | Action requires a resolved character target | | `Either` | Action accepts either an object or a character target | {% hint style="info" %} One executor component can serve multiple action definitions. Add separate entries with different `ActionName` values but the same `Executor` reference when the same behavior applies to multiple backend commands. {% endhint %} {% hint style="warning" %} Duplicate `ActionName` values in the same list are silently deduplicated at runtime. The first entry is kept; subsequent duplicates are discarded with a console warning. Names are compared case-insensitively. {% endhint %}
Unity Inspector showing a ConvaiActionConfigSource action definition entry with Action Name, Target Requirement, Executor, and Timeout Seconds fields filled in

A configured action definition — Action Name binds to the backend command string; Executor points to the Unity component that performs the behavior at runtime.

### Actionable objects Each entry in **Actionable Objects** registers a scene object as a valid target for the backend. #### Object definition fields | Field | Type | Description | | --------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Name` | `string` | The identifier Convai uses to reference this object in action commands. Case-insensitive matching at runtime. | | `Description` | `string` | Plain-language description sent to Convai. Used for natural language reference resolution ("the box by the wall"). Write as a full sentence describing type, color, location, and purpose. | | `GameObjectReference` | `GameObject` | The scene object to interact with at runtime. **Local-only — never sent to Convai.** | {% hint style="info" %} `GameObjectReference` is tagged `[JsonIgnore]`. Only `Name` and `Description` are serialized into the connect payload. Convai resolves targets by name; Unity maps that name to your `GameObject` locally. {% endhint %} **Writing effective descriptions:** | | Example | | ------------- | -------------------------------------------------------------------------------------------- | | **Too vague** | `An object in the scene` | | **Good** | `A red portable CO2 fire extinguisher mounted on the wall to the left of the main workbench` | | **Good** | `A yellow hard hat on the equipment shelf near the site entrance` | Descriptions are fixed at connect time. If a scene object's state changes mid-session (moved, replaced), the description Convai has does not update automatically. For dynamic scenes, use connect-time overrides (see below).
Unity Inspector showing the Actionable Objects list on ConvaiActionConfigSource with a registered scene object entry including Name, Description, and GameObject Reference fields

Actionable Objects list with a registered target — only Name and Description are serialized into the connect payload; GameObject Reference is local only and never sent to Convai.

### Actionable characters Each entry in **Actionable Characters** registers another NPC as a valid target for the backend. #### Character definition fields | Field | Type | Description | | --------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Name` | `string` | The identifier Convai uses to reference this character. | | `Bio` | `string` | Short description sent to Convai. Helps the backend understand who the character is for targeting decisions (e.g., "Site safety supervisor responsible for equipment checks"). | | `GameObjectReference` | `GameObject` | The character's `GameObject`. **Local-only — never sent to Convai.** |
Unity Inspector showing the Actionable Characters list on ConvaiActionConfigSource with a registered NPC entry including Name, Bio, and GameObject Reference fields

Actionable Characters list with a registered NPC — the Bio field helps the backend resolve natural-language character references such as "the supervisor" or "the engineer near the exit."

### Initial attention The **Initial Attention** field accepts a single object name. When the session starts, Convai treats that object as the NPC's current focus — it pre-seeds reference grounding before the first player turn. {% hint style="warning" %} If the name in **Initial Attention** does not match any entry in **Actionable Objects** (case-insensitive), the field is silently omitted from the connect payload and a console warning is logged. Verify the name matches exactly. {% endhint %} ### Session lifecycle The action configuration is sent to Convai once at session start and cannot be modified while a session is active. {% hint style="warning" %} Changes made to `ConvaiActionConfigSource` while in Play Mode do not take effect until you end the session and reconnect. {% endhint %} ### Dynamic configuration at connect time For procedurally generated scenes or multi-level games where action targets change between sessions, override the Inspector configuration via `RoomSessionConnectOptions` when calling `ConnectAsync`. Two independent override fields are available: | Field | Type | Effect | | --------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | `ActionConfigOverride` | `ConvaiActionConfig` | Replaces the full connect-time affordances sent to Convai (action names, objects, characters, initial attention) | | `ActionDefinitionsOverride` | `List` | Replaces the local Unity executor bindings for this session only | {% tabs %} {% tab title="Both overrides" %} Use when both the backend affordances and the local executor bindings should differ from the Inspector configuration: ```csharp using System.Collections.Generic; using Convai.Runtime.Actions; using Convai.Runtime.Room; using Convai.Shared.Actions; using UnityEngine; public sealed class DynamicActionSetup : MonoBehaviour { [SerializeField] private ConvaiManager _manager; [SerializeField] private NavMeshMoveToActionExecutor _mover; public async void ConnectWithOverrides() { var options = new RoomSessionConnectOptions { ActionConfigOverride = new ConvaiActionConfig { Actions = new List { "Move To", "Pick Up" }, Objects = new List { new() { Name = "Helmet", Description = "Yellow hard hat on the equipment shelf" }, new() { Name = "Locker", Description = "Green metal locker near the exit" } }, CurrentAttentionObject = "Helmet" }, ActionDefinitionsOverride = new List { new() { ActionName = "Move To", TargetRequirement = ConvaiActionTargetRequirement.Object, Executor = _mover } } }; await _manager.ConnectAsync(options); } } ``` {% endtab %} {% tab title="Config override only" %} Use when the backend affordances should change but the Inspector's local executor bindings remain correct: ```csharp var options = new RoomSessionConnectOptions { ActionConfigOverride = new ConvaiActionConfig { Actions = new List { "Move To" }, Objects = BuildObjectListFromCurrentLevel() } }; await _manager.ConnectAsync(options); ``` {% endtab %} {% endtabs %} {% hint style="info" %} `ActionDefinitionsOverride` is filtered against `ActionConfigOverride.Actions`. Only definitions whose `ActionName` appears in the config's action list are active for that session. Definitions for unlisted action names are silently ignored. {% endhint %} ### Usage examples #### Example 1 — Industrial safety drill (Inspector setup) **Scenario:** A fire safety training simulation where the NPC instructor can retrieve equipment when asked. **Setup in Inspector:** Action definitions: * `ActionName = Retrieve`, `TargetRequirement = Object`, `Executor = NavMeshMoveToActionExecutor` * `ActionName = Point At`, `TargetRequirement = Either`, `Executor = LookAtTargetActionExecutor` Actionable objects: * `Name = Extinguisher`, `Description = Red CO2 fire extinguisher on the wall bracket beside the pump station` * `Name = Alarm Panel`, `Description = Emergency alarm panel with a red pull handle near the main entrance` **Expected outcome:** When the trainee says "retrieve the extinguisher," the NPC navigates to it. When the trainee says "point at the alarm," the NPC faces it. #### Example 2 — Procedural level (scripted override) **Scenario:** Each level loads different equipment. Object targets are built from level data at runtime. ```csharp private List BuildObjectsFromLevel(LevelData level) { var objects = new List(); foreach (EquipmentEntry entry in level.Equipment) { objects.Add(new ConvaiActionObjectDefinition { Name = entry.Id, Description = entry.Description, GameObjectReference = entry.SceneObject }); } return objects; } ``` Pass the resulting list in `ActionConfigOverride.Objects` when calling `ConnectAsync`. ### Next steps {% content-ref url="/pages/6RYF3LHemCYzxRfZpueZ" %} [Action executors](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/action-executors) {% endcontent-ref %} {% content-ref url="/pages/L2RNE4SlLqEb39SpjbDU" %} [Dispatcher and batch policies](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/dispatcher-and-batch-policies) {% endcontent-ref %} # Action executors Reference for all six action executor components — look-at, event, NavMesh movement, animation trigger, and compound pickup executors. Executors are the components that perform in-scene behavior when the dispatcher runs an action step. Six executor components ship with the Convai SDK. This page documents every Inspector field and explains when to use each executor. #### LookAtTargetActionExecutor Smoothly rotates the NPC to face a resolved target over a configurable duration. Uses `Quaternion.Slerp` and respects cancellation. | Attribute | Value | | ------------------- | ------------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Actions → Look At Target Action Executor` | | **Namespace** | `Convai.Runtime.Actions` | | **Target required** | Yes — returns `Unhandled` if no target is resolved | **Inspector fields:** | Field | Type | Default | Description | | ------------- | ----------- | ------- | ----------------------------------------------------------------------------- | | `_rotateRoot` | `Transform` | `null` | The transform to rotate. If unassigned, uses the component's own `transform`. | | `_duration` | `float` | `0.5` | Seconds to complete the rotation. `0` snaps immediately. | **Behavior:** The executor interpolates from the current rotation toward the target's position over `_duration` seconds. If the root or target is destroyed mid-execution, it returns `Failed`. #### UnityEventActionExecutor Fires a `UnityEvent` and immediately returns `Succeeded`. No target resolution is required. Use this to connect any backend action to Inspector-wired callbacks without writing code — toggle doors, play sounds, open UI panels. | Attribute | Value | | ------------------- | ---------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Actions → Unity Event Action Executor` | | **Namespace** | `Convai.Runtime.Actions` | | **Target required** | No | **Inspector fields:** | Field | Type | Description | | ------------ | ------------ | -------------------------------------------------------------------------------------- | | `_onExecute` | `UnityEvent` | Invoked each time the action step runs. Wire any number of callbacks in the Inspector. | {% hint style="danger" %} **`TransformMoveToActionExecutor` is for prototyping only.** It teleports the character instantly with no animation or pathfinding. Replace it with `NavMeshMoveToActionExecutor` or a custom executor before shipping to users. {% endhint %} #### TransformMoveToActionExecutor Immediately snaps the NPC's transform to the resolved target's position plus an optional offset. Synchronous — completes in one frame. | Attribute | Value | | ------------------- | ---------------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Samples → Transform Move To Action Executor` | | **Namespace** | `Convai.Sample.Behaviors` | | **Target required** | Yes — returns `Failed` if no target is resolved | **Inspector fields:** | Field | Type | Default | Description | | ----------- | ----------- | ----------- | ----------------------------------------------------------------------------------------------- | | `_moveRoot` | `Transform` | `null` | The transform to move. If unassigned, moves the component's own `transform`. | | `_offset` | `Vector3` | `(0, 0, 0)` | World-space offset applied to the target position. Use to stop slightly in front of the target. | #### NavMeshMoveToActionExecutor Drives a `NavMeshAgent` to the resolved target's position and waits until the agent reaches stopping distance. The action step stays active until arrival, which means the dispatcher holds the next step until navigation completes. | Attribute | Value | | ------------------- | -------------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Samples → NavMesh Move To Action Executor` | | **Namespace** | `Convai.Sample.Behaviors` | | **Target required** | Yes — returns `Failed` if no target is resolved | **Inspector fields:** | Field | Type | Default | Description | | ------------------- | -------------- | ------------- | ------------------------------------------------------------------------------------------- | | `_agent` | `NavMeshAgent` | Auto-resolved | The `NavMeshAgent` to drive. If unassigned, resolved from the same `GameObject` on `Awake`. | | `_stoppingDistance` | `float` | `0.5` | Distance in world units at which the agent is considered to have arrived. | {% hint style="warning" %} The scene must have a baked NavMesh before this executor can navigate. Open **Window → AI → Navigation** and bake before entering Play Mode. If the agent starts off the NavMesh, `SetDestination` will fail silently and the executor will loop forever until the `TimeoutSeconds` expires. {% endhint %} #### AnimatorTriggerActionExecutor Maps backend action names to Animator trigger parameters via a configurable binding list. Fires the trigger and immediately returns `Succeeded` — it does not wait for the animation to finish. | Attribute | Value | | ------------------- | --------------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Samples → Animator Trigger Action Executor` | | **Namespace** | `Convai.Sample.Behaviors` | | **Target required** | No | **Inspector fields:** | Field | Type | Default | Description | | ----------- | ------------------------------------ | ------------- | --------------------------------------------------------------------------------------- | | `_animator` | `Animator` | Auto-resolved | The `Animator` to drive. If unassigned, resolved from the same `GameObject` on `Awake`. | | `_bindings` | `List` | Empty | Maps action names to trigger names. Each entry has two string fields (see below). | **AnimatorTriggerActionBinding fields:** | Field | Type | Description | | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------- | | `ActionName` | `string` | The action name to match (case-insensitive). Must match an entry in `ConvaiActionConfigSource`'s Action Definitions. | | `TriggerName` | `string` | The Animator trigger parameter to fire when the action is matched. Must match the exact trigger name in the Animator Controller. | **Example binding list:** | ActionName | TriggerName | | ---------- | --------------- | | `Wave` | `TriggerWave` | | `Salute` | `TriggerSalute` | | `Point At` | `TriggerPoint` | If no binding matches the incoming action name, the executor returns `Failed` with message `No binding for ''`. #### PickUpActionExecutor Compound executor that chains three behaviors: navigate to the target → fire an animation trigger → wait for the animation → attach the object to a hand transform. The step stays active until all three phases complete. | Attribute | Value | | ------------------- | ------------------------------------------------------------ | | **Menu path** | `Add Component → Convai → Samples → Pick Up Action Executor` | | **Namespace** | `Convai.Sample.Behaviors` | | **Target required** | Yes — returns `Failed` if no target is resolved | **Inspector fields:** | Field | Type | Default | Description | | -------------------- | ----------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------- | | `_mover` | `NavMeshMoveToActionExecutor` | Required | Drives navigation to the target. Must be assigned — returns `Failed` if null. | | `_animator` | `Animator` | `null` | The Animator to drive. Optional — skipped if null. | | `_pickUpTrigger` | `string` | `"PickUp"` | The Animator trigger parameter to fire after arriving at the target. | | `_attachPoint` | `Transform` | `null` | The transform the picked-up object is reparented to (e.g., hand bone). Optional — object is not reparented if null. | | `_animationDuration` | `float` | `1.0` | Seconds to wait after firing the animation trigger before reparenting the object. | **Execution sequence:** 1. `NavMeshMoveToActionExecutor.ExecuteAsync` navigates to the target. If it fails or is canceled, `PickUpActionExecutor` returns that result immediately. 2. `_animator.SetTrigger(_pickUpTrigger)` is called. 3. Waits `_animationDuration` seconds (cancellable). 4. Target `GameObject` is reparented to `_attachPoint` at local position/rotation zero. 5. Returns `Succeeded`. {% hint style="info" %} `PickUpActionExecutor` calls into `NavMeshMoveToActionExecutor` directly via `ExecuteAsync`. Both components must be on the same `GameObject` and a baked NavMesh must be present in the scene. {% endhint %} ### Choosing the right executor | Use case | Recommended executor | | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | NPC faces a target smoothly | `LookAtTargetActionExecutor` | | Any no-target action wired to gameplay callbacks | `UnityEventActionExecutor` | | Rapid prototyping without NavMesh | `TransformMoveToActionExecutor` | | Production NPC movement with pathfinding | `NavMeshMoveToActionExecutor` | | Play different animations for different actions | `AnimatorTriggerActionExecutor` | | Navigate + pick up + attach in one command | `PickUpActionExecutor` | | Custom movement stack, inventory, UI, physics | [Write a custom action executor](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/writing-custom-executors) | ### Usage examples #### Example 1 — Safety instructor with gesture and movement **Scenario:** A workplace safety training simulation. The instructor NPC uses two always-available executors to point at hazards and demonstrate equipment locations. **Inspector setup on the NPC:** * `LookAtTargetActionExecutor` — `_duration = 0.8` * `UnityEventActionExecutor` — `_onExecute` → calls `HazardHighlightManager.HighlightActive()` **ConvaiActionConfigSource definitions:** | ActionName | TargetRequirement | Executor | | ------------- | ----------------- | ---------------------------- | | `Point At` | `Either` | `LookAtTargetActionExecutor` | | `Flag Hazard` | `None` | `UnityEventActionExecutor` | **Expected outcome:** "Point at the gas valve" → the NPC rotates to face the gas valve over 0.8 seconds. "Flag the hazard" → the `UnityEvent` fires and highlights the active hazard in the UI. #### Example 2 — Equipment retrieval with animation **Scenario:** A medical training scenario. The instructor retrieves a defibrillator and hands it off. **Inspector setup:** * `NavMeshMoveToActionExecutor` — `_stoppingDistance = 0.6` * `PickUpActionExecutor` — `_mover = NavMeshMoveToActionExecutor`, `_pickUpTrigger = "GrabItem"`, `_attachPoint = RightHandBone`, `_animationDuration = 1.2` **ConvaiActionConfigSource definitions:** | ActionName | TargetRequirement | Executor | | ---------- | ----------------- | ---------------------- | | `Retrieve` | `Object` | `PickUpActionExecutor` | **Expected outcome:** "Retrieve the defibrillator" → the NPC navigates to the defibrillator, plays the grab animation for 1.2 seconds, then the defibrillator attaches to the right hand bone. ### Next steps {% content-ref url="/pages/L2RNE4SlLqEb39SpjbDU" %} [Dispatcher and batch policies](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/dispatcher-and-batch-policies) {% endcontent-ref %} {% content-ref url="/pages/yKmOPjUYIRCR4eqPYD0A" %} [Write a custom action executor](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/writing-custom-executors) {% endcontent-ref %} # Write a custom action executor Implement IConvaiActionExecutor on a MonoBehaviour to connect custom movement, inventory, UI, or physics behaviors to the Convai action pipeline. When the built-in executors don't match your project's movement system, interaction model, or gameplay rules, implement `IConvaiActionExecutor`. A custom executor is a standard C# `MonoBehaviour` with a single async method. The dispatcher treats it identically to any built-in executor — all policies, events, and cancellation behavior apply automatically. ### When to build a custom executor Build a custom executor when: * Your project uses a custom movement system (root motion, `CharacterController`, steering behaviors) * An action modifies inventory, UI state, quest flags, or physics objects * An action calls an external service or triggers a coroutine-based animation system * You need conditional logic — for example, an action that behaves differently depending on character state ### The IConvaiActionExecutor interface ```csharp public interface IConvaiActionExecutor { Task ExecuteAsync( ConvaiActionInvocation invocation, CancellationToken cancellationToken); } ``` Implement this interface on any `MonoBehaviour`. The dispatcher calls `ExecuteAsync` for each step and awaits the result before proceeding to the next step. Keep your task alive until the gameplay work is complete — returning early ends the step, even if the animation or movement is still running. {% hint style="info" %} Executors run on Unity's main thread. You can safely call Unity APIs (`transform`, `GetComponent`, `Instantiate`, etc.) anywhere in `ExecuteAsync`. Use `await Task.Yield()` to yield a frame without leaving the main thread. {% endhint %} ### The ConvaiActionInvocation object Every `ExecuteAsync` call receives a `ConvaiActionInvocation` with everything needed to perform the behavior: | Property | Type | Contains | | ---------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------- | | `Command` | `ConvaiActionCommand` | Raw backend command — `Name`, `Target`, `HasTarget` | | `Definition` | `ConvaiActionDefinition` | Local definition — `ActionName`, `TargetRequirement`, `Executor`, `TimeoutSeconds` | | `ResolvedTarget` | `ConvaiResolvedActionTarget` | Resolved target binding — `Kind`, `Name`, `ObjectBinding`, `CharacterBinding`, `GameObjectReference` | | `Character` | `ConvaiCharacter` | The executing NPC | | `BatchIndex` | `int` | Sequential index of this batch across the dispatcher's lifetime | | `StepIndex` | `int` | Index of this step within the current batch (0-based) | Access the target `GameObject` with: ```csharp GameObject targetGo = invocation.ResolvedTarget?.GameObjectReference; ``` Do not re-parse `invocation.Command.Name` or `invocation.Command.Target` to re-derive what to do. Use `invocation.Definition` and `invocation.ResolvedTarget` — they are already resolved and validated. ### Execution result types Return one of these factory methods from `ExecuteAsync`: | Factory method | When to use | | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `ConvaiActionExecutionResult.Succeeded()` | The behavior completed successfully | | `ConvaiActionExecutionResult.Failed(string message = null, Exception exception = null)` | A genuine error occurred (missing component, invalid state, gameplay failure) | | `ConvaiActionExecutionResult.Unhandled(string message = null)` | This executor intentionally declines to handle the invocation (wrong context or target type) | | `ConvaiActionExecutionResult.Canceled()` | The `CancellationToken` was signaled — return this when you observe cancellation in a loop | {% hint style="danger" %} Do **not** return `ConvaiActionExecutionResult.TimedOut()` manually. The dispatcher returns `TimedOut` automatically when `TimeoutSeconds` expires and the `CancellationToken` is triggered. If you return it yourself, the result is ambiguous and the dispatcher's timeout tracking is bypassed. {% endhint %} **`Failed` vs `Unhandled`:** Use `Failed` when you tried to perform the behavior and something went wrong. Use `Unhandled` when this executor should not handle this particular invocation at all — for example, if the target is the wrong type. The dispatcher fires `OnStepFailed` for `Failed` and `OnStepUnhandled` for `Unhandled`; both are treated as non-success for the `StopBatch` failure policy. ### Cancellation The `CancellationToken` is triggered when: 1. `BatchPolicy.ReplaceCurrent` activates (a new batch preempts the current one) 2. `TimeoutSeconds` on the action definition expires 3. The dispatcher is disabled or destroyed Always check the token in any loop or after each `await`: ```csharp while (!arrived) { cancellationToken.ThrowIfCancellationRequested(); // move one step await Task.Yield(); } ``` If your code catches `OperationCanceledException`, return `ConvaiActionExecutionResult.Canceled()` immediately: ```csharp try { await SomeAsyncOperation(cancellationToken); } catch (OperationCanceledException) { return ConvaiActionExecutionResult.Canceled(); } ``` Alternatively, let `ThrowIfCancellationRequested` propagate. The dispatcher wraps your `ExecuteAsync` in a try/catch and converts uncaught `OperationCanceledException` to `Canceled` automatically. ### Complete example: highlight object executor This executor enables an outline effect on the resolved target, waits three seconds, then disables it. ```csharp using System.Threading; using System.Threading.Tasks; using Convai.Runtime.Actions; using UnityEngine; [AddComponentMenu("MyProject/Actions/Highlight Object Executor")] public sealed class HighlightObjectExecutor : MonoBehaviour, IConvaiActionExecutor { [SerializeField] private float _highlightDuration = 3f; public async Task ExecuteAsync( ConvaiActionInvocation invocation, CancellationToken cancellationToken) { // 1. Get the target GameObject targetGo = invocation.ResolvedTarget?.GameObjectReference; if (targetGo == null) return ConvaiActionExecutionResult.Failed("No target resolved for Highlight action."); // 2. Find the required component var outline = targetGo.GetComponent(); if (outline == null) return ConvaiActionExecutionResult.Failed( $"Target '{invocation.ResolvedTarget.Name}' has no OutlineEffect component."); // 3. Execute the behavior outline.enabled = true; try { // 4. Wait, respecting cancellation await Task.Delay( (int)(_highlightDuration * 1000), cancellationToken); } catch (OperationCanceledException) { // Clean up on cancellation if (outline != null) outline.enabled = false; return ConvaiActionExecutionResult.Canceled(); } // 5. Clean up and return success if (outline != null) outline.enabled = false; return ConvaiActionExecutionResult.Succeeded(); } } ``` ### Compound actions Put the entire gameplay sequence inside one `ExecuteAsync`. The dispatcher treats one action definition as indivisible — it waits for your task to complete before starting the next step. This is the correct pattern for actions like pick-up, inspect, open-then-take, or any sequence that involves multiple sub-behaviors. ```csharp public async Task ExecuteAsync( ConvaiActionInvocation invocation, CancellationToken cancellationToken) { // Phase 1: Navigate ConvaiActionExecutionResult moveResult = await _mover.ExecuteAsync(invocation, cancellationToken); if (moveResult.Status != ConvaiActionExecutionStatus.Succeeded) return moveResult; // Phase 2: Interact cancellationToken.ThrowIfCancellationRequested(); _animator.SetTrigger("Interact"); // Phase 3: Wait for animation await Task.Delay(1200, cancellationToken); // Phase 4: Apply effect ApplyInteractionEffect(invocation.ResolvedTarget?.GameObjectReference); return ConvaiActionExecutionResult.Succeeded(); } ``` ### Executor design rules * **Use `invocation.ResolvedTarget`, not `invocation.Command.Target`.** The dispatcher has already resolved the name to a `GameObject` binding — don't re-parse the raw string. * **Return `Unhandled` when this executor is not appropriate.** A single executor component can be shared across multiple action definitions. Returning `Unhandled` signals the dispatcher to fire `OnStepUnhandled` without treating it as a hard failure. * **Set `TimeoutSeconds` in the action definition.** Use the timeout mechanism rather than implementing your own deadline logic inside the executor. * **Clean up on cancellation.** If your executor enables an effect, moves an object, or holds a resource, release it before returning `Canceled`. * **Do not hold state between invocations.** The same executor instance may be called for different targets across multiple batches. Do not assume the previous invocation's state is still valid. ### Next steps {% content-ref url="/pages/u3JAohXiqBNMG6zZ5m60" %} [Configure character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/configuring-actions) {% endcontent-ref %} {% content-ref url="/pages/73HufYIPX0JRnnAsGDbk" %} [Character actions scripting reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/actions-scripting-reference) {% endcontent-ref %} # Dispatcher and batch policies Configure ConvaiActionDispatcher's batch policy, failure policy, and lifecycle events, and inject action batches programmatically for testing or scripted sequences. `ConvaiActionDispatcher` is the runtime execution layer of the action system. It listens for command batches from Convai, resolves each action and target against the current session's configuration, and calls the bound executor components one step at a time. Two policies control what happens when new batches arrive during execution and when a step fails. ### Component overview | Attribute | Value | | --------------- | ---------------------------------------------------------------- | | **Menu path** | `Add Component → Convai → Convai Action Dispatcher` | | **Namespace** | `Convai.Runtime.Actions` | | **Constraints** | `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` | The dispatcher must be on the same `GameObject` as `ConvaiCharacter`. Only one dispatcher is allowed per character.
Unity Inspector showing the ConvaiActionDispatcher component with Batch Policy, Failure Policy, and lifecycle UnityEvent fields visible

ConvaiActionDispatcher in the Inspector — two policy dropdowns control queue and failure behavior; seven lifecycle UnityEvent fields expose the full batch and step execution pipeline.

### Inspector fields | Field | Type | Default | Description | | ------------------- | ---------------------------------- | ----------- | -------------------------------------------------------------------------- | | `_batchPolicy` | `ConvaiActionBatchPolicy` | `Queue` | How incoming batches behave while another batch is executing | | `_failurePolicy` | `ConvaiActionBatchFailurePolicy` | `StopBatch` | Whether a step failure aborts the remaining batch or allows it to continue | | `_onBatchStarted` | `UnityEvent` | — | Fires when a batch begins executing | | `_onStepStarted` | `ConvaiActionInvocationUnityEvent` | — | Fires at the start of each step | | `_onStepSucceeded` | `ConvaiActionInvocationUnityEvent` | — | Fires when a step executor returns `Succeeded` | | `_onStepFailed` | `ConvaiActionInvocationUnityEvent` | — | Fires when a step fails for any reason | | `_onStepUnhandled` | `ConvaiActionInvocationUnityEvent` | — | Fires when an executor returns `Unhandled` | | `_onBatchCompleted` | `UnityEvent` | — | Fires when all steps finish without being aborted | | `_onBatchAborted` | `UnityEvent` | — | Fires when the batch is cut short by the failure policy | `ConvaiActionInvocationUnityEvent` is a serializable `UnityEvent`. Wire it in the Inspector exactly like a standard `UnityEvent` — the event parameter carries the full invocation context (action name, target, character, batch and step index). ### Batch policy Batch policy controls what happens when Convai returns a new action batch while the dispatcher is still executing a previous one. | Policy | Enum value | Behavior | | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Queue` | `0` | New batches wait in a queue. The current batch finishes before the next starts. Suitable for most scenarios. | | `ReplaceCurrent` | `1` | Cancels the currently executing step and clears any queued batches. The new batch starts immediately. Use for interrupt-driven scenarios (e.g., "Stop, come here instead"). | | `DropIncoming` | `2` | Discards new batches until the current batch and all queued batches finish. Use when an in-progress sequence must not be interrupted (e.g., a safety demonstration that must complete). | {% hint style="info" %} `ReplaceCurrent` cancels the **currently running executor step** via the `CancellationToken` and clears all pending batches before starting the new one. Executors must respect the cancellation token for this to be instant — see [Write a custom action executor](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/writing-custom-executors). {% endhint %}
Unity Inspector showing the Batch Policy dropdown on ConvaiActionDispatcher expanded with Queue, ReplaceCurrent, and DropIncoming options

Batch Policy dropdown — Queue is the default and suits most scenarios; ReplaceCurrent handles interrupt-driven NPC behavior; DropIncoming protects sequences that must run to completion.

### Failure policy Failure policy controls what happens when an executor returns a non-success result (`Failed`, `Unhandled`, `Canceled`, or `TimedOut`). | Policy | Enum value | Behavior | | --------------- | ---------- | -------------------------------------------------------------------------------------------------- | | `StopBatch` | `0` | Remaining steps in the batch are skipped. `OnBatchAborted` fires. | | `ContinueBatch` | `1` | Execution continues with the next step regardless of failure. `OnBatchCompleted` fires at the end. | Use `ContinueBatch` when actions are independent — a failed "Point At" should not prevent a following "Wave." Use `StopBatch` (the default) for dependent sequences — a failed "Move To" should prevent a following "Pick Up" that would fail anyway.
Unity Inspector showing the Failure Policy dropdown on ConvaiActionDispatcher expanded with StopBatch and ContinueBatch options

Failure Policy dropdown — StopBatch (default) aborts the remaining steps and fires OnBatchAborted; ContinueBatch continues through failures and fires OnBatchCompleted at the end.

### Lifecycle events The dispatcher fires events at every meaningful stage of batch and step execution. Subscribe in the Inspector via UnityEvent fields, or subscribe in C# via the properties.
Unity Inspector showing the ConvaiActionDispatcher lifecycle UnityEvent fields: OnBatchStarted, OnStepStarted, OnStepSucceeded, OnStepFailed, OnStepUnhandled, OnBatchCompleted, and OnBatchAborted

Dispatcher lifecycle UnityEvent fields — wire these in the Inspector to respond to batch and step transitions without writing dispatcher-side C# code.

#### Event firing order ``` OnBatchStarted → OnStepStarted (for each step) → OnStepSucceeded (if executor returned Succeeded) or → OnStepFailed (if executor returned Failed, Canceled, or TimedOut) or → OnStepUnhandled (if executor returned Unhandled) OnBatchCompleted (all steps finished, or ContinueBatch allowed failures through) or OnBatchAborted (StopBatch policy cut the batch short after a failure) ``` #### Subscribing in C\# ```csharp using Convai.Runtime.Actions; using UnityEngine; public sealed class ActionFeedback : MonoBehaviour { [SerializeField] private ConvaiActionDispatcher _dispatcher; private void OnEnable() { _dispatcher.OnBatchStarted.AddListener(HandleBatchStarted); _dispatcher.OnStepSucceeded.AddListener(HandleStepSucceeded); _dispatcher.OnStepFailed.AddListener(HandleStepFailed); _dispatcher.OnBatchCompleted.AddListener(HandleBatchCompleted); _dispatcher.OnBatchAborted.AddListener(HandleBatchAborted); } private void OnDisable() { _dispatcher.OnBatchStarted.RemoveListener(HandleBatchStarted); _dispatcher.OnStepSucceeded.RemoveListener(HandleStepSucceeded); _dispatcher.OnStepFailed.RemoveListener(HandleStepFailed); _dispatcher.OnBatchCompleted.RemoveListener(HandleBatchCompleted); _dispatcher.OnBatchAborted.RemoveListener(HandleBatchAborted); } private void HandleBatchStarted() => Debug.Log("Batch started"); private void HandleStepSucceeded(ConvaiActionInvocation inv) => Debug.Log($"Step succeeded: {inv.Command.Name}"); private void HandleStepFailed(ConvaiActionInvocation inv) => Debug.LogWarning($"Step failed: {inv.Command.Name}"); private void HandleBatchCompleted() => Debug.Log("Batch completed"); private void HandleBatchAborted() => Debug.LogWarning("Batch aborted"); } ``` ### Manual batch injection `EnqueueActions(IReadOnlyList actions)` submits a batch to the dispatcher programmatically, respecting the active batch and failure policies. Use this for scripted demonstration sequences, automated test runs, or NPC behaviors triggered by game events rather than player speech. ```csharp using System.Collections.Generic; using Convai.Runtime.Actions; using Convai.Shared.Types; using UnityEngine; public sealed class DemoTrigger : MonoBehaviour { [SerializeField] private ConvaiActionDispatcher _dispatcher; public void RunSafetyDemo() { _dispatcher.EnqueueActions(new List { new ConvaiActionCommand("Move To", "Extinguisher"), new ConvaiActionCommand("Pick Up", "Extinguisher"), new ConvaiActionCommand("Move To", "Exit") }); } } ``` The dispatcher executes these steps sequentially. If the `BatchPolicy` is `Queue`, this batch waits behind any batch already in progress. ### Bypassing the dispatcher If you want to react to raw action commands without the dispatcher's target resolution and execution pipeline, subscribe to `ConvaiCharacter.OnActionsReceived` directly: ```csharp using System.Collections.Generic; using Convai.Runtime.Components; using Convai.Shared.Types; using UnityEngine; public sealed class ManualActionHandler : MonoBehaviour { [SerializeField] private ConvaiCharacter _character; private void OnEnable() => _character.OnActionsReceived += HandleActions; private void OnDisable() => _character.OnActionsReceived -= HandleActions; private void HandleActions(IReadOnlyList commands) { foreach (ConvaiActionCommand cmd in commands) Debug.Log($"Action: {cmd.Name}, Target: {cmd.Target}"); } } ``` {% hint style="warning" %} Bypassing the dispatcher means no automatic target resolution, no batch/failure policies, and no lifecycle events. This is appropriate for read-only observation or custom dispatch pipelines, but not for typical gameplay where the SDK should drive the behavior. {% endhint %} ### Dispatcher lifecycle behavior | Situation | Dispatcher behavior | | ------------------------------------ | ------------------------------------------------------------------ | | Dispatcher disabled | Active work is canceled; queue is cleared | | Dispatcher destroyed | Same as disabled | | Empty batch received | Silently ignored — no events fire | | Action name not in local definitions | Step fails: `OnStepFailed` fires; `StopBatch` aborts the batch | | Executor field not assigned | Step fails: `OnStepFailed` fires | | Target requirement not met | Step fails: `OnStepFailed` fires | | Executor returns `Unhandled` | `OnStepUnhandled` fires; treated as failure for `StopBatch` policy | ### Usage examples #### Example 1 — Training checklist integration **Scenario:** A corporate onboarding simulation. A checklist UI advances when the NPC completes each equipment demonstration. Wire `OnBatchCompleted` in the Inspector to `TrainingChecklistManager.AdvanceStep()`. Each time the NPC finishes a full sequence, the checklist advances automatically. ```csharp // TrainingChecklistManager.cs public void AdvanceStep() { _currentStep++; UpdateChecklistUI(); } ``` No additional code is required on the dispatcher side — wire the `OnBatchCompleted` UnityEvent in the Inspector. #### Example 2 — Fallback dialogue on navigation failure **Scenario:** When the NPC cannot reach a target (NavMesh path blocked), it should speak a fallback line rather than silently stopping. Subscribe to `OnStepFailed` and inject a dynamic context update: ```csharp private void HandleStepFailed(ConvaiActionInvocation invocation) { if (invocation.Command.Name == "Move To") { string targetName = invocation.Command.Target ?? "that location"; // Inject into dynamic context so the NPC acknowledges the failure naturally _character.DynamicContext.AddEvent($"Unable to reach {targetName} — path was blocked."); } } ``` ### Next steps {% content-ref url="/pages/yKmOPjUYIRCR4eqPYD0A" %} [Write a custom action executor](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/writing-custom-executors) {% endcontent-ref %} {% content-ref url="/pages/73HufYIPX0JRnnAsGDbk" %} [Character actions scripting reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/actions-scripting-reference) {% endcontent-ref %} # Attention and reference grounding Update NPC focus at runtime so Convai resolves vague player references such as "pick that up" or "go to it" to the correct registered scene object. Reference grounding is how Convai resolves vague player language — "grab that," "go to it," "look at the one on the left" — to a specific registered object or character. Two inputs drive grounding: the rich descriptions you write for each target in `ConvaiActionConfigSource`, and the current attention object you update at runtime as the player's focus changes. ### How grounding works When a player says "pick up that cylinder," Convai evaluates two things: 1. **Object descriptions** — the Name and Description text you registered at connect time. Convai uses these to match "cylinder" to your registered object. 2. **Current attention object** — which object the NPC is currently "focused on." When set, Convai weighs it heavily for ambiguous references like "that" or "it." Descriptions are fixed at connect time and cannot be updated mid-session. The current attention object can be changed at any point during an active conversation. ### Write effective object descriptions The `Description` field on each `ConvaiActionObjectDefinition` is the most important text for grounding accuracy. Write each description as a single natural sentence that includes: * **Object type** — what kind of thing it is * **Identifying attribute** — color, material, size, or label * **Location** — where it is relative to landmarks in the scene * **Purpose** — what it is used for | | Example | | ----------------------- | ------------------------------------------------------------------------------------------------------------- | | **Too vague — avoid** | `An object in the scene` | | **No location — avoid** | `A fire extinguisher` | | **Good** | `A red portable CO2 fire extinguisher mounted on the wall bracket to the left of the main pump control panel` | | **Good** | `A yellow hard hat on the equipment shelf immediately to the right of the site entrance gate` | Vague descriptions cause Convai to pick the wrong target or fail to resolve ambiguous references. {% hint style="warning" %} Descriptions are sent to Convai at connect time and cannot be changed while the session is active. If your scene changes at runtime (objects moved, replaced, or destroyed), end the session and reconnect with updated descriptions, or use `ActionConfigOverride` at connect time to build descriptions programmatically. See [Configure character actions — Dynamic configuration](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/configuring-actions#dynamic-configuration-at-connect-time). {% endhint %} ### Runtime attention API Update the NPC's current attention object at any point during an active conversation using methods on `ConvaiCharacter`: ```csharp // Set by object name character.SetCurrentAttentionObject("Extinguisher"); // Set by definition reference character.SetCurrentAttentionObject(myObjectDefinition); // Clear — NPC has no specific focus character.ClearCurrentAttentionObject(); ``` #### Method signatures ```csharp void SetCurrentAttentionObject(string objectName, string runLlm = "false") void SetCurrentAttentionObject(ConvaiActionObjectDefinition actionObject, string runLlm = "false") void ClearCurrentAttentionObject(string runLlm = "false") ``` #### The runLlm parameter The optional `runLlm` parameter controls whether the attention change immediately triggers a new LLM turn. The default `"false"` updates the grounding context silently. Pass `"true"` if you want Convai to react to the focus change with a natural language response. ```csharp // Silent update — NPC does not react aloud character.SetCurrentAttentionObject("GasValve"); // NPC may react aloud to the change in focus character.SetCurrentAttentionObject("GasValve", runLlm: "true"); ``` ### Silent failure conditions {% hint style="warning" %} These calls are silently ignored if the precondition is not met. A warning is logged to the Console in each case. {% endhint %} | Condition | Result | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Not in an active conversation | Call is ignored. Warning: `Cannot set attention object: not in conversation` | | Object name is empty or whitespace | Call is ignored. Warning: `Cannot set empty attention object` | | Object name not in active action-config | Call is ignored. Warning: `Cannot set attention object 'X': it is not present in the active action_config objects` | The object name must match an entry in `ConvaiActionConfigSource.Objects` (case-insensitive). It does not need to match the `GameObjectReference` name — it must match the `Name` field in the object definition. ### Attention scope {% hint style="info" %} The attention object affects **only the backend's reference resolution for future turns**. Setting the attention object does not: * Create a new actionable target * Change which objects are in the action config * Cause the NPC to physically look at or move toward the object * Affect any active in-progress action step {% endhint %} ### Initial attention at connect time To pre-seed the NPC's focus before the first player turn, set the **Initial Attention** field in `ConvaiActionConfigSource` to the name of an object in your **Actionable Objects** list. This is equivalent to calling `SetCurrentAttentionObject` at the moment of connection. The initial attention object must match an entry in **Actionable Objects** exactly (case-insensitive). If it does not match, the field is silently omitted from the connect payload and a warning is logged. ### Usage examples #### Example 1 — Cursor-based selection in a training simulation **Scenario:** An industrial inspection simulation. When the trainee's cursor hovers over a piece of equipment, update the instructor NPC's attention so "point at it" resolves correctly. ```csharp using Convai.Runtime.Components; using Convai.Shared.Actions; using UnityEngine; using UnityEngine.EventSystems; public sealed class EquipmentFocusTracker : MonoBehaviour, IPointerEnterHandler, IPointerExitHandler { [SerializeField] private ConvaiCharacter _instructor; [SerializeField] private ConvaiActionObjectDefinition _objectDefinition; public void OnPointerEnter(PointerEventData eventData) { _instructor.SetCurrentAttentionObject(_objectDefinition); } public void OnPointerExit(PointerEventData eventData) { _instructor.ClearCurrentAttentionObject(); } } ``` **Expected outcome:** When the trainee hovers the cursor over a gas valve, the instructor's grounding shifts to that valve. "Point at it" now reliably resolves to the hovered object. #### Example 2 — Physics-based proximity attention **Scenario:** A medical training scenario. The NPC instructor automatically focuses on whichever piece of equipment the student is standing near. ```csharp using Convai.Runtime.Components; using UnityEngine; public sealed class ProximityAttentionTrigger : MonoBehaviour { [SerializeField] private ConvaiCharacter _instructor; [SerializeField] private string _objectName; private void OnTriggerEnter(Collider other) { if (!other.CompareTag("Player")) return; _instructor.SetCurrentAttentionObject(_objectName); } private void OnTriggerExit(Collider other) { if (!other.CompareTag("Player")) return; _instructor.ClearCurrentAttentionObject(); } } ``` Place this component on a trigger volume around each piece of equipment. Set `_objectName` to match the object's `Name` in `ConvaiActionConfigSource`. When the student enters the trigger area, the NPC's grounding shifts to that equipment automatically. **Expected outcome:** When the student walks up to the defibrillator station, "show me how to use it" reliably resolves to the defibrillator without the student needing to name it explicitly. ### Next steps {% content-ref url="/pages/u3JAohXiqBNMG6zZ5m60" %} [Configure character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/configuring-actions) {% endcontent-ref %} {% content-ref url="/pages/73HufYIPX0JRnnAsGDbk" %} [Character actions scripting reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/actions-scripting-reference) {% endcontent-ref %} # Character actions scripting reference API reference for the Convai character actions system — dispatcher types, executor interface, config classes, invocation objects, result types, and enums. Complete API reference for all public types in the Convai character actions system. All types are in the `Convai.Runtime.Actions`, `Convai.Runtime.Components`, or `Convai.Shared.Actions` namespaces unless noted. ### ConvaiActionDispatcher `MonoBehaviour` — `Convai.Runtime.Actions` Menu path: `Add Component → Convai → Convai Action Dispatcher` Constraints: `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` #### Properties | Property | Type | Description | | ------------------ | ---------------------------------- | ------------------------------------------------------------------ | | `BatchPolicy` | `ConvaiActionBatchPolicy` | Current batch policy (read-only from code; set in Inspector) | | `FailurePolicy` | `ConvaiActionBatchFailurePolicy` | Current failure policy (read-only from code; set in Inspector) | | `OnBatchStarted` | `UnityEvent` | Fires when a batch begins execution | | `OnStepStarted` | `ConvaiActionInvocationUnityEvent` | Fires at the start of each action step | | `OnStepSucceeded` | `ConvaiActionInvocationUnityEvent` | Fires when an executor returns `Succeeded` | | `OnStepFailed` | `ConvaiActionInvocationUnityEvent` | Fires when a step fails (Failed, Canceled, or TimedOut) | | `OnStepUnhandled` | `ConvaiActionInvocationUnityEvent` | Fires when an executor returns `Unhandled` | | `OnBatchCompleted` | `UnityEvent` | Fires when all batch steps finish without the batch being aborted | | `OnBatchAborted` | `UnityEvent` | Fires when `StopBatch` policy cuts the batch short after a failure | #### Methods | Method | Signature | Description | | ---------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------- | | `EnqueueActions` | `void EnqueueActions(IReadOnlyList actions)` | Submits a batch to the dispatcher. Respects the active `BatchPolicy`. | ### ConvaiActionConfigSource `MonoBehaviour` — `Convai.Runtime.Components` Menu path: `Add Component → Convai → Convai Action Config Source` Constraints: `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` #### Properties | Property | Type | Description | | ------------------------ | ------------------------------------------------ | ---------------------------------------------------------- | | `Definitions` | `IReadOnlyList` | The authored action definitions list | | `Objects` | `IReadOnlyList` | The authored actionable objects list | | `Characters` | `IReadOnlyList` | The authored actionable characters list | | `InitialAttentionObject` | `string` | Object name to pre-seed as the NPC's focus at connect time | #### Methods | Method | Signature | Description | | ------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `BuildActionConfig` | `ConvaiActionConfig BuildActionConfig()` | Builds and returns the connect-time payload. Returns `null` if no valid definitions exist. | | `TryResolveObject` | `bool TryResolveObject(string objectName, out ConvaiActionObjectDefinition actionObject)` | Looks up a registered object by name (case-insensitive). Returns `true` if found. | ### ConvaiCharacter — action-relevant members `MonoBehaviour` — `Convai.Runtime.Components` #### Events | Event | Type | Description | | ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `OnActionsReceived` | `event Action>` | Fires when Convai returns an action batch for this character. Fires before the dispatcher processes it. | #### Properties | Property | Type | Description | | -------------- | -------------------- | ------------------------------------------------------------------------------------ | | `ActionConfig` | `ConvaiActionConfig` | Returns a clone of the active session's action config. May be `null` before connect. | #### Methods | Method | Signature | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GetActionConfigSource` | `ConvaiActionConfigSource GetActionConfigSource()` | Returns the `ConvaiActionConfigSource` on this `GameObject`, or `null`. | | `SetCurrentAttentionObject` | `void SetCurrentAttentionObject(string objectName, string runLlm = "false")` | Updates backend grounding to the named object. Requires an active conversation. Silently ignored if the object name is not in the active action config. | | `SetCurrentAttentionObject` | `void SetCurrentAttentionObject(ConvaiActionObjectDefinition actionObject, string runLlm = "false")` | Overload that accepts a definition reference. | | `ClearCurrentAttentionObject` | `void ClearCurrentAttentionObject(string runLlm = "false")` | Clears the current attention object. Requires an active conversation. | ### RoomSessionConnectOptions — action fields `Convai.Runtime.Room` | Field | Type | Description | | --------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | | `ActionConfigOverride` | `ConvaiActionConfig` | When set, replaces `ConvaiActionConfigSource.BuildActionConfig()` for this session. | | `ActionDefinitionsOverride` | `List` | When set, replaces the Inspector action definitions for this session. Filtered against `ActionConfigOverride.Actions` if both are set. | ### ConvaiActionCommand `Convai.Shared.Types` — Serializable sealed class Structured action command for one step, as returned by the backend. #### Properties | Property | Type | Description | | ----------- | -------- | ------------------------------------------------------------------------------------------- | | `Name` | `string` | Required. Action name selected by the backend (e.g., `"Move To"`). | | `Target` | `string` | Optional. Object or character name the backend resolved as the target. `null` if no target. | | `HasTarget` | `bool` | `true` when `Target` is non-empty. | #### Constructor ```csharp new ConvaiActionCommand("Move To", "Crate") // name + target new ConvaiActionCommand("Wave") // name only ``` ### ConvaiActionConfig `Convai.Shared.Actions` — Serializable sealed class Connect-time action affordances serialized into the session connect payload. #### Properties | Property | Type | Description | | ------------------------ | --------------------------------------- | ------------------------------------------------------------------------------------------- | | `Actions` | `List` | Action names allowed for this session. Only names are sent — executor bindings stay local. | | `Objects` | `List` | Objects the backend may reference as targets. `GameObjectReference` is never serialized. | | `Characters` | `List` | Characters the backend may reference as targets. `GameObjectReference` is never serialized. | | `CurrentAttentionObject` | `string` | Initial attention object name. Must match an entry in `Objects`. | ### ConvaiActionDefinition `Convai.Runtime.Actions` — Serializable sealed class Local Unity binding between a backend action name and an executor component. Only `ActionName` is sent to the backend. #### Fields | Field | Type | Description | | ------------------- | ------------------------------- | ----------------------------------------------------------------------------------- | | `ActionName` | `string` | The action name that maps to this definition. Case-insensitive matching at runtime. | | `TargetRequirement` | `ConvaiActionTargetRequirement` | What kind of target this action requires. | | `Executor` | `MonoBehaviour` | The component that performs the behavior. Must implement `IConvaiActionExecutor`. | | `TimeoutSeconds` | `float` | Maximum execution time in seconds. `0` = no timeout. | ### ConvaiActionObjectDefinition `Convai.Shared.Actions` — Serializable sealed class #### Properties | Property | Type | Serialized | Description | | --------------------- | ------------ | ----------------------- | --------------------------------------------------------------------- | | `Name` | `string` | Yes (`"name"`) | Identifier used in action commands. Case-insensitive matching. | | `Description` | `string` | Yes (`"description"`) | Natural language description sent to Convai for reference resolution. | | `GameObjectReference` | `GameObject` | **No** (`[JsonIgnore]`) | Local scene reference. Never sent to Convai. | ### ConvaiActionCharacterDefinition `Convai.Shared.Actions` — Serializable sealed class #### Properties | Property | Type | Serialized | Description | | --------------------- | ------------ | ----------------------- | ------------------------------------------------------------------ | | `Name` | `string` | Yes (`"name"`) | Identifier for this character target. | | `Bio` | `string` | Yes (`"bio"`) | Short description sent to Convai (e.g., "Site safety supervisor"). | | `GameObjectReference` | `GameObject` | **No** (`[JsonIgnore]`) | Local scene reference. Never sent to Convai. | ### ConvaiActionInvocation `Convai.Runtime.Actions` — Sealed class Typed execution context passed to executors and all dispatcher events. #### Properties | Property | Type | Description | | ---------------- | ---------------------------- | ---------------------------------------------------------------------------------------- | | `Command` | `ConvaiActionCommand` | The raw backend command for this step. | | `Definition` | `ConvaiActionDefinition` | The matched local action definition. `null` if no definition was found (step will fail). | | `ResolvedTarget` | `ConvaiResolvedActionTarget` | The resolved target binding. `null` if the action has no target or resolution failed. | | `Character` | `ConvaiCharacter` | The NPC executing this action. | | `BatchIndex` | `int` | Sequential index of the containing batch across the dispatcher's lifetime. | | `StepIndex` | `int` | 0-based index of this step within the current batch. | ### ConvaiResolvedActionTarget `Convai.Runtime.Actions` — Serializable sealed class Resolved target for one action step. #### Properties | Property | Type | Description | | --------------------- | --------------------------------- | ---------------------------------------------------------------------------------------- | | `Kind` | `ConvaiActionTargetKind` | Whether the resolved target is an Object, Character, or None. | | `Name` | `string` | The resolved name (from the backend command). | | `ObjectBinding` | `ConvaiActionObjectDefinition` | The matched object definition. `null` if `Kind != Object`. | | `CharacterBinding` | `ConvaiActionCharacterDefinition` | The matched character definition. `null` if `Kind != Character`. | | `GameObjectReference` | `GameObject` | The scene `GameObject` from the matching binding. The primary access point in executors. | ### ConvaiActionExecutionResult `Convai.Runtime.Actions` — Readonly struct Return type for `IConvaiActionExecutor.ExecuteAsync`. #### Properties | Property | Type | Description | | ----------- | ----------------------------- | --------------------------------------------------------------------- | | `Status` | `ConvaiActionExecutionStatus` | The outcome of this execution step. | | `Message` | `string` | Optional human-readable detail. Available in event handlers and logs. | | `Exception` | `Exception` | The exception that caused failure, if any. | #### Factory methods | Method | Signature | Use when | | ----------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `Succeeded` | `static ConvaiActionExecutionResult Succeeded()` | The behavior completed successfully. | | `Failed` | `static ConvaiActionExecutionResult Failed(string message = null, Exception exception = null)` | A genuine error occurred. | | `Canceled` | `static ConvaiActionExecutionResult Canceled()` | The `CancellationToken` was signaled. | | `TimedOut` | `static ConvaiActionExecutionResult TimedOut()` | **Do not call manually.** The dispatcher returns this automatically when `TimeoutSeconds` expires. | | `Unhandled` | `static ConvaiActionExecutionResult Unhandled(string message = null)` | This executor intentionally declines the invocation. | ### IConvaiActionExecutor `Convai.Runtime.Actions` — Interface The extension point for all action behavior. Implement on any `MonoBehaviour`. ```csharp public interface IConvaiActionExecutor { Task ExecuteAsync( ConvaiActionInvocation invocation, CancellationToken cancellationToken); } ``` ### Enumerations #### ConvaiActionBatchPolicy `Convai.Runtime.Actions` | Value | Integer | Description | | ---------------- | ------- | ---------------------------------------------------------------------------------- | | `Queue` | `0` | New batches wait until the current batch completes. Default. | | `ReplaceCurrent` | `1` | Cancels the active step and all pending batches; starts the new batch immediately. | | `DropIncoming` | `2` | Discards new batches until all current and queued work is finished. | #### ConvaiActionBatchFailurePolicy `Convai.Runtime.Actions` | Value | Integer | Description | | --------------- | ------- | -------------------------------------------------------------------------- | | `StopBatch` | `0` | A failed step aborts the remaining batch. `OnBatchAborted` fires. Default. | | `ContinueBatch` | `1` | Execution continues to the next step regardless. `OnBatchCompleted` fires. | #### ConvaiActionTargetRequirement `Convai.Runtime.Actions` | Value | Integer | Description | | ----------- | ------- | --------------------------------------------------------- | | `None` | `0` | Action does not require a target. | | `Object` | `1` | Action requires a resolved object target. | | `Character` | `2` | Action requires a resolved character target. | | `Either` | `3` | Action accepts either an object or a character as target. | #### ConvaiActionTargetKind `Convai.Runtime.Actions` | Value | Integer | Description | | ----------- | ------- | --------------------------------- | | `None` | `0` | No target resolved. | | `Object` | `1` | Target is a registered object. | | `Character` | `2` | Target is a registered character. | #### ConvaiActionExecutionStatus `Convai.Runtime.Actions` | Value | Integer | Dispatcher event fired | | ----------- | ------- | ---------------------- | | `Succeeded` | `0` | `OnStepSucceeded` | | `Failed` | `1` | `OnStepFailed` | | `Canceled` | `2` | `OnStepFailed` | | `TimedOut` | `3` | `OnStepFailed` | | `Unhandled` | `4` | `OnStepUnhandled` | ### ConvaiActionInvocationUnityEvent `Convai.Runtime.Actions` — Serializable class extending `UnityEvent` Wrapper type that makes `ConvaiActionInvocation` serializable as a UnityEvent parameter. Assign handlers in the Inspector like any standard UnityEvent. The event's single argument is the `ConvaiActionInvocation` for that step. ### ConvaiActionDebugProbe `Convai.Runtime.Actions` — Sealed `MonoBehaviour` Menu path: `Add Component → Convai → Debug → Convai Action Debug Probe` Constraints: `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` See [Troubleshoot character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/debugging-and-troubleshooting) for the full Inspector field reference and usage guide. #### Context menu actions | Command | Effect | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `Inject Test Batch` | Submits a `Move To` command targeting the first registered object to the dispatcher. Tests the pipeline without a live conversation. | | `Reset Probe State` | Resets all counters and text fields to zero/empty. | ### Next steps {% content-ref url="/pages/f7muso7lrEOfJU9hnYeK" %} [Character actions examples](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/usage-examples) {% endcontent-ref %} {% content-ref url="/pages/LZcDaILUbaCK85OVcPJ6" %} [Troubleshoot character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/debugging-and-troubleshooting) {% endcontent-ref %} # Character actions examples Progressive examples for the Convai character actions system — Inspector setup, event subscriptions, scripted batch injection, and navigation error recovery. These four examples progress from the simplest possible configuration to full scripting control. Each example is self-contained — you can follow any one of them without reading the others first. ### Example 1 — Fire safety retrieval (Inspector setup, no code) **Scenario:** A fire safety training simulation. The instructor NPC retrieves a fire extinguisher when the trainee asks. No scripting required. **Prerequisites:** NavMesh baked in the scene. #### Inspector configuration On the instructor NPC's `GameObject`, add these components: * `ConvaiCharacter` * `ConvaiActionConfigSource` * `ConvaiActionDispatcher` (leave both policies at defaults: Queue, StopBatch) * `NavMeshMoveToActionExecutor` — `_stoppingDistance = 0.6` In `ConvaiActionConfigSource`: **Action definitions:** | Action name | Target requirement | Executor | | ----------- | ------------------ | ----------------------------- | | `Retrieve` | `Object` | `NavMeshMoveToActionExecutor` | | `Point At` | `Either` | `LookAtTargetActionExecutor` | **Actionable objects:** | Name | Description | | -------------- | ----------------------------------------------------------------------------------------- | | `Extinguisher` | Red portable CO2 fire extinguisher on the wall bracket beside the main pump control panel | | `Alarm Panel` | Emergency alarm panel with a red pull handle mounted near the site entrance | **Expected outcome:** * "Retrieve the extinguisher" → the NPC navigates to the extinguisher and stops 0.6 units away. * "Point at the alarm panel" → the NPC rotates to face the alarm panel over 0.5 seconds. * "Retrieve the alarm" → Convai correctly resolves "alarm" to "Alarm Panel" based on the description. {% hint style="success" %} Open the Console and filter by `ConvaiActionDebugProbe` (if the probe is added). You should see: ``` [ConvaiActionDebugProbe] Step succeeded #1: cmd='Retrieve Extinguisher', def='Retrieve', target=Object:Extinguisher ``` {% endhint %} ### Example 2 — Onboarding checklist integration (event subscription) **Scenario:** A corporate onboarding simulation. As the NPC demonstrates each workstation, a checklist UI advances. Completing the full equipment tour advances the training stage. #### C# setup Wire `OnBatchCompleted` to the checklist manager. No additional code on the dispatcher side is required if you wire it in the Inspector. For code-driven wiring: ```csharp using Convai.Runtime.Actions; using UnityEngine; public sealed class OnboardingTourController : MonoBehaviour { [SerializeField] private ConvaiActionDispatcher _dispatcher; [SerializeField] private TrainingChecklistUI _checklist; private void OnEnable() { _dispatcher.OnBatchCompleted.AddListener(HandleTourStepCompleted); _dispatcher.OnBatchAborted.AddListener(HandleTourStepFailed); } private void OnDisable() { _dispatcher.OnBatchCompleted.RemoveListener(HandleTourStepCompleted); _dispatcher.OnBatchAborted.RemoveListener(HandleTourStepFailed); } private void HandleTourStepCompleted() { _checklist.MarkCurrentStepComplete(); _checklist.AdvanceToNextStep(); } private void HandleTourStepFailed() { _checklist.MarkCurrentStepIncomplete(); } } ``` **ConvaiActionConfigSource definitions:** | Action name | Target requirement | Executor | | ------------- | ------------------ | ----------------------------- | | `Walk To` | `Object` | `NavMeshMoveToActionExecutor` | | `Demonstrate` | `Object` | `LookAtTargetActionExecutor` | **Actionable objects:** Each workstation registered with its name and location description. **Expected outcome:** The trainee says "show me the filing system." The NPC walks to the filing cabinet, faces it, and `OnBatchCompleted` fires — the checklist advances to the next step automatically. ### Example 3 — Navigation failure with fallback dialogue (error recovery) **Scenario:** A construction site safety simulation. When the NPC cannot reach a hazard zone (path blocked), it acknowledges the obstacle rather than silently stopping. #### C# setup Subscribe to `OnStepFailed` and inject a dynamic context event so the NPC speaks a natural fallback: ```csharp using Convai.Runtime.Actions; using UnityEngine; public sealed class ActionFailureHandler : MonoBehaviour { [SerializeField] private ConvaiActionDispatcher _dispatcher; [SerializeField] private ConvaiCharacter _character; private void OnEnable() => _dispatcher.OnStepFailed.AddListener(HandleStepFailed); private void OnDisable() => _dispatcher.OnStepFailed.RemoveListener(HandleStepFailed); private void HandleStepFailed(ConvaiActionInvocation invocation) { if (invocation.Command.Name != "Move To") return; string targetName = string.IsNullOrEmpty(invocation.Command.Target) ? "that location" : invocation.Command.Target; // Tell Convai what happened so the NPC can acknowledge it naturally _character.DynamicContext.AddEvent( $"Movement to '{targetName}' failed — the path was blocked."); } } ``` **Expected outcome:** The NPC navigates toward the hazard zone, the `NavMeshAgent` fails to complete the path, the executor returns `Failed`, and `OnStepFailed` fires. The fallback event is injected, and the NPC says something like "I can't get to the chemical storage area — the path is blocked by the scaffolding." {% hint style="info" %} Set `FailurePolicy` to `StopBatch` (default) so subsequent steps in the same batch (e.g., "Demonstrate hazard") don't run when the navigation step failed. {% endhint %} ### Example 4 — Scripted demonstration sequence (programmatic injection) **Scenario:** A medical procedure training simulation. At a defined moment in the training script (triggered by a timeline event), the NPC automatically walks through an equipment demonstration without waiting for the trainee to ask. #### C# setup Use `ConvaiActionDispatcher.EnqueueActions` to inject a multi-step sequence from a timeline trigger or UI button: ```csharp using System.Collections.Generic; using Convai.Runtime.Actions; using Convai.Shared.Types; using UnityEngine; public sealed class DemonstrationTrigger : MonoBehaviour { [SerializeField] private ConvaiActionDispatcher _dispatcher; // Call this from a Unity Timeline signal, UI button, or game event public void RunDefibrillatorDemo() { _dispatcher.EnqueueActions(new List { new ConvaiActionCommand("Move To", "Equipment Cart"), new ConvaiActionCommand("Pick Up", "Defibrillator"), new ConvaiActionCommand("Move To", "Patient Bed"), new ConvaiActionCommand("Point At", "Patient Bed") }); } } ``` Wire `RunDefibrillatorDemo` to a `UnityEngine.Timeline` signal, a UI button `OnClick`, or any other trigger in your scene. **Expected outcome:** The instructor NPC navigates to the equipment cart, picks up the defibrillator, walks to the patient bed, and turns to face it — all without the trainee saying anything. `OnBatchCompleted` fires when the sequence finishes, which you can use to advance the training stage. {% hint style="info" %} `BatchPolicy = Queue` ensures this scripted sequence waits politely if the trainee is mid-conversation with an active action batch. Switch to `BatchPolicy = ReplaceCurrent` if the demonstration should interrupt any ongoing action. {% endhint %} ### Next steps {% content-ref url="/pages/LZcDaILUbaCK85OVcPJ6" %} [Troubleshoot character actions](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/debugging-and-troubleshooting) {% endcontent-ref %} {% content-ref url="/pages/73HufYIPX0JRnnAsGDbk" %} [Character actions scripting reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/actions-scripting-reference) {% endcontent-ref %} # Troubleshoot character actions Diagnose action pipeline issues using ConvaiActionDebugProbe's Inspector counters and test batch injection — with a complete symptom/cause/fix reference for common failures. The fastest path to diagnosing action pipeline issues is `ConvaiActionDebugProbe`. Add it to your NPC's `GameObject`, enter Play Mode, and watch its counters update in real time. This page covers the probe's full Inspector reference, its context menu tools, and a complete troubleshooting table for every common failure mode. ### ConvaiActionDebugProbe `MonoBehaviour` — `Convai.Runtime.Actions` Menu path: `Add Component → Convai → Debug → Convai Action Debug Probe` Constraints: `DisallowMultipleComponent`, `RequireComponent(ConvaiCharacter)` The probe auto-resolves `ConvaiCharacter` and `ConvaiActionDispatcher` from the same `GameObject` on `Awake`. Both are shown in the Inspector as read-only reference fields that confirm auto-resolution succeeded. #### Inspector fields | Field | Type | Description | | --------------------- | ------------------------ | --------------------------------------------------------------------------------------- | | `_character` | `ConvaiCharacter` | Auto-resolved. Tracks raw action batches from the backend. | | `_dispatcher` | `ConvaiActionDispatcher` | Auto-resolved. Tracks execution lifecycle events. | | `_logToConsole` | `bool` | When enabled, all probe events are printed to the Console. Disable for quieter testing. | | `_receivedBatchCount` | `int` | Total batches received from Convai via `OnActionsReceived`. | | `_startedStepCount` | `int` | Total steps the dispatcher has started executing. | | `_succeededStepCount` | `int` | Total steps that returned `Succeeded`. | | `_failedStepCount` | `int` | Total steps that returned `Failed`, `Canceled`, or `TimedOut`. | | `_unhandledStepCount` | `int` | Total steps that returned `Unhandled`. | | `_abortedBatchCount` | `int` | Total batches cut short by the `StopBatch` failure policy. | | `_lastReceivedBatch` | `string` (TextArea) | JSON of the most recent batch received from Convai. | | `_lastStepStarted` | `string` (TextArea) | Summary of the most recent step the dispatcher started. | | `_lastStepSucceeded` | `string` (TextArea) | Summary of the most recently succeeded step. | | `_lastUnhandledStep` | `string` (TextArea) | Summary of the most recently unhandled step. | #### Context menu actions Right-click the probe component header in the Inspector to access: | Command | Effect | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | **Inject Test Batch** | Submits a `Move To` command targeting the first registered object to the dispatcher. Tests the full pipeline without a live conversation. | | **Reset Probe State** | Resets all counters to `0` and clears all text fields. Use between test runs to keep counters meaningful. | #### Console log format When `_logToConsole` is enabled, the probe writes to the Console in these formats: ``` [ConvaiActionDebugProbe] Received action batch #1: [{"name":"Move To","target":"Extinguisher"}] [ConvaiActionDebugProbe] Dispatcher batch started. [ConvaiActionDebugProbe] Step started #1: cmd='Move To Extinguisher', def='Move To', target=Object:Extinguisher [ConvaiActionDebugProbe] Step succeeded #1: cmd='Move To Extinguisher', def='Move To', target=Object:Extinguisher [ConvaiActionDebugProbe] Dispatcher batch completed. ``` For failures: ``` [ConvaiActionDebugProbe] Step failed #1: cmd='Move To Cupboard', def='', target=None: [ConvaiActionDebugProbe] Dispatcher batch aborted #1. ``` ### Diagnostic checklist Use this checklist in order when actions are not executing: {% stepper %} {% step %} #### Verify the backend is sending actions Check `_receivedBatchCount` in the probe Inspector after speaking a command in Play Mode. * **Counter increments** → the backend returned an action batch; proceed to the next step. * **Counter stays at 0** → Convai did not return an action response. Possible causes: * `ConvaiActionConfigSource` has no action definitions (the backend does not know actions are available) * The Convai backend character is not configured to return actions for this character ID * The session did not connect successfully {% endstep %} {% step %} #### Verify the dispatcher is processing the batch If `_receivedBatchCount` increments but `_startedStepCount` stays at 0: * `ConvaiActionDispatcher` may be missing or disabled on the NPC's `GameObject` * Check that the dispatcher is on the **same `GameObject`** as `ConvaiCharacter` * Verify the dispatcher component is enabled in the Inspector (the checkbox next to the component name) {% endstep %} {% step %} #### Read the step failure message If `_failedStepCount` increments, expand `_lastStepStarted` and check the Console for a failure message. The dispatcher logs the exact reason: | Console message | Cause | | ------------------------------------------------------------ | ------------------------------------------------------------------- | | `No local action definition found for 'X'` | Action name mismatch — see next step | | `Action 'X' is missing a valid executor` | Executor field is empty — assign the executor component | | `Target requirement 'Object' not satisfied (resolved: None)` | Backend sent a target name that doesn't match any registered object | | {% endstep %} | | {% step %} #### Check for action name mismatches Action names are matched **case-insensitively** but **spaces are significant**. `Move To` and `move to` match. `Move To` and `MoveTo` do not. In `_lastReceivedBatch`, find the exact name the backend sent. Compare it to the `ActionName` field in `ConvaiActionConfigSource`. They must match character-for-character (ignoring case). {% endstep %} {% step %} #### Verify component references In `ConvaiActionConfigSource`, expand each **Action Definition** entry: * **Executor field empty** → the step will fail with "missing a valid executor." Drag the executor component reference into the Executor field. * **Executor does not implement `IConvaiActionExecutor`** → the step will fail. Custom executors must implement the interface. {% endstep %} {% step %} #### Test with Inject Test Batch Right-click `ConvaiActionDebugProbe` → **Inject Test Batch**. This submits a `Move To` command targeting your first registered object directly to the dispatcher, bypassing the backend. * **Step succeeds** → the pipeline works correctly; the issue is with how Convai is returning actions, not with your Unity setup. * **Step fails** → the issue is in local component configuration (executor, NavMesh, missing reference). Click **Reset Probe State** between test runs to keep counters readable. {% endstep %} {% endstepper %} ### Troubleshooting table | Symptom | Likely cause | Fix | | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `_receivedBatchCount` stays 0 after speaking | `ConvaiActionConfigSource` is missing or has no action definitions | Add `ConvaiActionConfigSource` with at least one action definition; the backend only returns actions if it knows actions are configured | | `_receivedBatchCount` increments but `_startedStepCount` stays 0 | `ConvaiActionDispatcher` missing, disabled, or on wrong `GameObject` | Add the dispatcher to the same `GameObject` as `ConvaiCharacter`; verify it is enabled | | `_failedStepCount` increments: "No local action definition found for 'X'" | Action name sent by backend does not match any `ActionName` in local definitions | Open `_lastReceivedBatch` to see the exact name; match it (case-insensitive, spaces matter) in `ConvaiActionConfigSource` | | `_failedStepCount` increments: "missing a valid executor" | `Executor` field in `ConvaiActionDefinition` is empty | Drag the executor component reference into the `Executor` field in `ConvaiActionConfigSource` | | `_failedStepCount` increments: "Target requirement not satisfied" | Target name from backend does not match any registered object or character | Open `_lastReceivedBatch` to see the target name; verify it matches a `Name` entry in **Actionable Objects** or **Actionable Characters** (case-insensitive) | | `_unhandledStepCount` increments | Executor returned `Unhandled` — executor declined to handle this invocation | Check executor logic; `Unhandled` means the executor chose not to run, not that something broke | | `_abortedBatchCount` increments | A step failed and `StopBatch` policy aborted the remaining steps | Fix the failing step (see above), or change `FailurePolicy` to `ContinueBatch` if steps are independent | | NPC teleports instead of navigating | `TransformMoveToActionExecutor` is in use | Replace with `NavMeshMoveToActionExecutor` or a custom executor using your movement system | | NPC starts moving then freezes | `NavMeshMoveToActionExecutor` agent is stuck or path is blocked | Bake NavMesh (**Window → AI → Navigation → Bake**); verify the NPC and target are both on NavMesh surface; set `TimeoutSeconds` on the action definition to prevent indefinite blocking | | NPC navigates but object is not picked up | `PickUpActionExecutor._mover` is null | Assign a `NavMeshMoveToActionExecutor` reference to `_mover` in the `PickUpActionExecutor` Inspector | | Actions configured in Inspector but not working after scene change | Configuration sent at connect time is now stale | End the session and reconnect; action configuration is only sent once at connect time | | Action works in editor but not in build | Executor components not included in build | Verify executor scripts are in the project's compile scope; check for `[assembly: ...]` exclusions | | `SetCurrentAttentionObject` call has no effect | Not in an active conversation, or object name not in active config | Call only after `ConnectAsync` completes; object name must match a registered entry in `ConvaiActionConfigSource.Objects` | ### Next steps {% content-ref url="/pages/73HufYIPX0JRnnAsGDbk" %} [Character actions scripting reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/actions-scripting-reference) {% endcontent-ref %} {% content-ref url="/pages/M9Lh8OZ24Oqv9SmxMjzb" %} [Character actions quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/character-actions/quick-start) {% endcontent-ref %} # Dynamic context Find all Dynamic Context guides — feed live state and events to characters from Inspector or C# and verify context-aware dialogue. Dynamic Context gives Convai characters real-time awareness of what is happening in the scene. Characters can reference live conditions — a trainee's current location, equipment collected, hazards triggered, or checkpoint status — and incorporate that information naturally into dialogue. Updates flow through two entry points: the `ConvaiDynamicContextCommand` Inspector component (no code required) and the `IConvaiDynamicContext` scripting interface on `ConvaiCharacter`. Both write to the same underlying tracker and produce identical network behavior.
How dynamic context works
Understand the states and events model, canonical context format, and pre-conversation queueing design.		/pages/Oyh83QIsfM02w8hOnerK
Dynamic context quick start
Add the command component to an NPC and verify context-aware dialogue in five steps.		/pages/c83fd507c05d6353540d730017d6b1681c7bb60d
Command component reference
Field-by-field reference for all six command types, reaction modes, and validation warnings.		/pages/o4soWlixkVqAjbqKQCzH
Static context at connection time
Configure initial dynamic info sent once at connection for facts that do not change during a session.		/pages/wuITWZtDYpGKPOC2GUTI
Sync behavior and timing
How and when Replace, Append, and Reset messages are transmitted for each context operation.		/pages/jLDVtNvrQQtepjrSEeAv
Dynamic context scripting API
Full API reference for IConvaiDynamicContext — all seven methods with signatures and queueing behavior.		/pages/Pwb170m39yjzkfAghzLG
Dynamic context usage examples
Four end-to-end scenarios covering safety drills, onboarding, guided tours, and emergency transitions.		/pages/3UWNhGolZFLe1BduX0bt
Troubleshoot dynamic context
Investigation checklist, symptom table, decision tree, and Console log reference for common issues.		/pages/7ij66qcsYVpiLPUZFGvh
### Next steps Start with [Dynamic context quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-quick-start) to get context-aware dialogue running. Then read [How dynamic context works](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/how-dynamic-context-works) to understand the states and events model before moving into the reference pages. # How dynamic context works Understand the states and events model, canonical context format, and how updates queue and flush before and during conversations. Dynamic Context gives characters a live, structured view of what is happening in the scene. Instead of relying only on the static system prompt configured on the Convai dashboard, a character can reference a trainee's current location, the equipment they have collected, or an alarm that just triggered — because that information was injected directly into the session as it occurred. This page explains the underlying model: what the primitives are, how the SDK assembles them into a canonical context string, and why updates queue safely before a conversation starts. ### States and events Dynamic Context is built on two primitive types. **States** are persistent, named key-value pairs. Each state has a name and a value. When you set a state, any previous value for that name is replaced. States are suitable for facts that change over time but have exactly one current value: the operator's current station, the hazard level in a zone, or whether a checklist item has been completed. **Events** are chronological, one-time occurrences. Unlike states, events accumulate in sequence and are never replaced or deduplicated. Each call to `AddEvent` appends a new line to the character's context. Events are suitable for things that happened during a session and that the character should be able to reference in order: "Trainee bypassed the manual lockout procedure", "Chemical alarm triggered at Bay 7". Both primitives feed into the character's awareness simultaneously. States provide a stable, queryable snapshot of current conditions; events provide a chronological record of what has happened. ### Canonical context format When the SDK sends an update to Convai, it assembles a canonical context string from all tracked states and events: ``` {StateName} is {Value} {AnotherState} is {Value} Event text line one Event text line two ``` States appear first, in the order they were **first set** — updating a state's value does not change its position. Events follow in call order after all states. The reason states preserve insertion order across updates is to give the character a stable, predictable view of the world. If `Station` was the first thing set, it always appears first in the character's context, regardless of how many times the value has changed. This makes the context easier for the model to interpret consistently. **Example:** ```csharp context.SetState("Station", "Bay 3"); // position 1 context.SetState("HazardLevel", "High"); // position 2 context.AddEvent("Operator bypassed interlock"); context.SetState("Station", "Bay 7"); // updates value; position stays at 1 ``` Canonical context after all four calls: ``` Station is Bay 7 HazardLevel is High Operator bypassed interlock ``` You supply only names, values, and event text. The SDK assembles and delivers the canonical string automatically. ### Two entry points Dynamic Context has two entry points that write to the same underlying tracker and produce identical network behavior. **Inspector — `ConvaiDynamicContextCommand`** Add this `MonoBehaviour` to the NPC's GameObject and configure the command type, fields, and reaction mode in the Inspector. Call `Execute()` from a `UnityEvent`, trigger collider, timeline marker, or UI button. No scripting required. One component encapsulates one command; for multiple commands per NPC, place each on a child GameObject. Use this entry point when: * Context changes are tied to scene events that already fire `UnityEvent` callbacks * Non-programmers need to configure or modify context triggers * You want to prototype quickly without writing glue code **Scripting — `IConvaiDynamicContext`** Access `character.DynamicContext` to get the `IConvaiDynamicContext` interface and call methods directly from C#. This gives full control over timing, batching, and reaction mode. ```csharp IConvaiDynamicContext context = _character.DynamicContext; context.SetState("Station", "Bay 7"); context.AddEvent("Operator bypassed interlock"); ``` Use this entry point when: * Context updates depend on runtime logic or data that cannot be expressed as static Inspector fields * Multiple states must change atomically (use `SetStates`) * You need to read state values back (`TryGetStateValue`) * The update source is an external system such as a state machine or analytics pipeline ### Pre-conversation queueing Updates made before a conversation begins are automatically queued by all tracked methods — `SetState`, `SetStates`, `AddEvent`, `RemoveState`, and `Reset`. When the session connects, the SDK delivers a single Replace message containing the full canonical context at that moment. This means you can set initial context freely from `Awake` or `Start` without timing concerns. The SDK handles delivery. ```csharp void Start() { // Safe — all three queue and flush as one Replace when ConnectAsync fires _character.DynamicContext.SetState("Facility", "Offshore Platform Alpha"); _character.DynamicContext.SetState("Scenario", "Fire Drill"); _character.DynamicContext.AddEvent("Session initialized"); } ``` The reason this collapses into one Replace rather than replaying individual messages is efficiency: the character receives one authoritative snapshot rather than a stream of incremental updates that could arrive out of order or create redundant LLM turns. {% hint style="warning" %} `Apply()` is the one exception: it does not queue. If called before a conversation starts, the update is discarded. Use `SetState`, `AddEvent`, or other tracked methods for pre-conversation context. See [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) for details. {% endhint %} ### When to use which command type | Goal | Use | | ------------------------------------------------------- | ----------------------------------------------------------- | | Track one current condition | `SetState` | | Track several conditions that change at the same moment | `SetStates` (one canonical rebuild, one network round-trip) | | Record that something happened | `AddEvent` | | Remove a condition that no longer applies | `RemoveState` | | Clear all runtime context (for a new scenario phase) | `Reset` | | Send externally constructed context text | `Apply()` — advanced; does not queue | ### Next steps {% content-ref url="/pages/c83fd507c05d6353540d730017d6b1681c7bb60d" %} [Dynamic context quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-quick-start) {% endcontent-ref %} {% content-ref url="/pages/o4soWlixkVqAjbqKQCzH" %} [Command component reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/command-component-reference) {% endcontent-ref %} {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} # Dynamic context quick start Add ConvaiDynamicContextCommand to an NPC, configure a SetState command, and confirm the character acknowledges live scene conditions. This guide walks you through the minimum setup to verify that your Convai character acknowledges live in-scene conditions. You will add the `ConvaiDynamicContextCommand` component, configure a `SetState` command, wire it to a UI button, and confirm the character references the state you sent. ### Prerequisites Before starting, verify: * [ ] A `ConvaiCharacter` is in the scene and responds to speech in Play Mode {% stepper %} {% step %} #### Add the command component Select the NPC's GameObject in the Hierarchy. In the Inspector, click **Add Component** and search for **Convai Dynamic Context Command**, or navigate to **Convai → Dynamic Context → Convai Dynamic Context Command**. The component appears with three sections: **Target**, **Command**, and **Events**.
Unity Inspector showing ConvaiDynamicContextCommand added to the NPC GameObject, with Target, Command, and Events sections visible

ConvaiDynamicContextCommand added to the NPC — three sections appear: Target resolves the character, Command defines the context operation, and Events exposes execution callbacks.

{% endstep %} {% step %} #### Verify character resolution In the **Target** section, confirm that **Auto Resolve Character** is enabled (the default). The component finds the `ConvaiCharacter` on the same GameObject automatically. If `ConvaiCharacter` is on a different GameObject, disable **Auto Resolve Character** and drag the correct `ConvaiCharacter` into the **Character** field. {% endstep %} {% step %} #### Configure the command In the **Command** section: * Set **Command Type** to **Set State** * Set **State Name** to `Location` * Set **State Value** to `Fire Exit Corridor` * Leave **Reaction Mode** at **Auto** (the default) The component is now configured to set a tracked state named `Location` to `Fire Exit Corridor` whenever `Execute()` is called. {% endstep %} {% step %} #### Wire the trigger In the **Events** section of a UI Button in your scene (create a temporary one if needed), locate **On Click ()**. Click **+** to add a listener, drag the NPC's GameObject into the object field, and select **ConvaiDynamicContextCommand → Execute ()** from the function dropdown.
Unity Inspector showing a UI Button's On Click event wired to ConvaiDynamicContextCommand.Execute() on the NPC GameObject

Execute() wired to the button's On Click event — pressing the button at runtime delivers the configured context update to Convai and triggers the character's reaction according to the configured Reaction Mode.

{% endstep %} {% step %} #### Test in Play Mode Enter Play Mode and start a conversation with the character. Click the button you wired in the previous step, then ask the character where you are. The character should reference the location — for example: *"You're at the Fire Exit Corridor. Make sure you know the evacuation procedure before proceeding."* If the character does not respond with location awareness, open the Unity Console and check for a `[ConvaiDynamicContextCommand]` warning. See [Troubleshoot dynamic context](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/troubleshoot-dynamic-context) for a full diagnosis checklist. {% endstep %} {% endstepper %} {% hint style="success" %} Your character is now context-aware. The `Location` state is tracked locally and delivered to Convai when the conversation is active. Any future `SetState` call for the same name updates the value and notifies the character automatically. {% endhint %} ### Test without custom code The SDK includes a pre-built test UI for exploring the full Dynamic Context system without writing any integration code. **Prefab path:** Packages/space.vars.sdk\_package\_id/Prefabs/SampleDynamicContextUI.prefab Drop it into your scene, assign your `ConvaiCharacter`, enter Play Mode, and use the **Set State** button to send known values. If the character responds correctly through the Sample UI but not through your own integration, the issue is in your code — not in the Dynamic Context system itself. ### Next steps {% content-ref url="/pages/o4soWlixkVqAjbqKQCzH" %} [Command component reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/command-component-reference) {% endcontent-ref %} {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} {% content-ref url="/pages/3UWNhGolZFLe1BduX0bt" %} [Dynamic context usage examples](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-usage-examples) {% endcontent-ref %} # Command component reference Complete field-by-field reference for ConvaiDynamicContextCommand — all six command types, reaction modes, validation warnings, and multi-command child GameObject patterns. `ConvaiDynamicContextCommand` is the no-code entry point for Dynamic Context. It is a `MonoBehaviour` that encapsulates one context operation and exposes its entire configuration through the Unity Inspector. The component is marked `[DisallowMultipleComponent]` — Unity prevents adding a second instance to the same GameObject. To drive multiple commands from one NPC, place each additional command on a **child GameObject** (see [Multiple commands per NPC](#multiple-commands-per-npc)). Add the component via **Convai → Dynamic Context → Convai Dynamic Context Command**.
Unity Inspector showing the ConvaiDynamicContextCommand component with Target, Command, and Events sections collapsed on an NPC GameObject

ConvaiDynamicContextCommand Inspector — the Target section resolves which character receives the command, Command defines the operation type and its parameters, and Events exposes On Executed and On Execution Skipped callbacks.

### Target section Controls which `ConvaiCharacter` the command operates on. | Field | Type | Default | Description | | ---------------------- | ----------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Character | `ConvaiCharacter` | None | Explicit character reference. Takes precedence over auto-resolve when assigned. Use when the command is on a different GameObject than the NPC. | | Auto Resolve Character | `bool` | `true` | When enabled, the component calls `GetComponent()` on the **same GameObject** at execution time. | **Resolution order:** If **Character** is assigned, it is used regardless of the **Auto Resolve Character** setting. If **Character** is empty and **Auto Resolve Character** is enabled, the component searches the same GameObject. If neither resolves a character, `Execute()` is skipped and **On Execution Skipped** fires. ### Command section #### Command type The **Command Type** field controls what operation `Execute()` performs. One component = one command type. **Set State** Updates a single tracked state. If the state does not exist, it is created. If the value is identical to the current value, no update is sent (idempotent). | Field | Description | | ----------- | --------------------------------------------------------------------------- | | State Name | The state identifier. Must be non-empty and non-whitespace. Case-sensitive. | | State Value | The value to assign. May be empty string. | **Set States** Updates multiple tracked states atomically in one call. Preferred over multiple sequential `SetState` commands when several values change simultaneously — produces one canonical rebuild rather than multiple. | Field | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------- | | State Entries | List of Name / Value pairs. Must have at least one entry. All names must be non-empty and unique within the list. | **Add Event** Appends a chronological event entry. Events are never replaced or deduplicated — each call adds a new line in call order, after all states in the canonical context. | Field | Description | | ---------- | -------------------------------------------------------- | | Event Text | The event description sent to Convai. Must be non-empty. | **Remove State** Removes a tracked state by name and sends an updated canonical context to Convai. If the state is not tracked, `Execute()` completes without sending any update. | Field | Description | | ---------- | --------------------------------------------------- | | State Name | The name of the state to remove. Must be non-empty. | **Reset** Clears all tracked states and events from the character's Dynamic Context and sends a Reset message to Convai. Takes no additional fields. {% hint style="warning" %} **Reset** clears the runtime Dynamic Context layer only. Initial Dynamic Info Text (set on `ConvaiCharacter`) is not re-injected, and system prompt facts on the Convai dashboard are not affected. The character's in-session LLM memory is also not cleared. See [Static context at connection time](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/static-context-at-connection-time) for a full explanation of what Reset does and does not clear. {% endhint %} **Raw Update** Sends a typed context update directly to the transport layer, bypassing the local tracker. For advanced use cases that construct context text externally. | Field | Description | | -------- | ---------------------------------------------------------------------------------------------------------------------------- | | Raw Text | The context text to send. Required unless Raw Mode is `Reset`. | | Raw Mode | `Append` — adds to existing context. `Replace` — replaces entire context. `Reset` — clears all context; Raw Text is ignored. | {% hint style="danger" %} **Raw Update does not queue pre-conversation updates.** If `Execute()` is called before a conversation is active, the update is discarded. Additionally, values sent via Raw Update are not readable via `TryGetStateValue` on the scripting API. For context that must be delivered regardless of conversation state, use **Set State** or **Add Event** instead. {% endhint %} ### Reaction mode The **Reaction Mode** field controls whether the character generates an immediate response after the context update is delivered. | Value | Behavior | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | | `Auto` | Convai decides whether the update warrants an immediate response. **Default for `ConvaiDynamicContextCommand`.** | | `ReactImmediately` | Always triggers an immediate LLM turn after the update. Use for updates that require the character to acknowledge the change. | | `SyncOnly` | Updates context silently. The character incorporates the new information into its next natural turn. No immediate response is generated. | {% hint style="warning" %} **Component default differs from scripting API defaults.** `ConvaiDynamicContextCommand` defaults **Reaction Mode** to `Auto` for all command types — including Set State and Set States. The scripting API methods `SetState` and `SetStates` default to `SyncOnly`. If you switch between Inspector and scripting control, verify the reaction mode is what you expect. {% endhint %} ### Events section | Event | When it fires | | -------------------- | ------------------------------------------------------------------------------------------------------------------------ | | On Executed | After `Execute()` completes successfully — the command ran and the context update was dispatched. | | On Execution Skipped | When `Execute()` is called but validation fails or character resolution fails. The Unity Console shows the exact reason. | Wire **On Execution Skipped** to a temporary `Debug.Log` during development to catch misconfiguration without having to poll the Console manually. ### Multiple commands per NPC `[DisallowMultipleComponent]` prevents more than one `ConvaiDynamicContextCommand` on the same GameObject. To send multiple independent commands from one NPC: 1. Create a child GameObject under the NPC. 2. Add `ConvaiDynamicContextCommand` to the child. 3. In the **Target** section, **disable Auto Resolve Character** — auto-resolve only searches the same GameObject. 4. Drag the NPC's `ConvaiCharacter` into the **Character** field explicitly. 5. Repeat for each additional command. Each child command is independent — configure its command type, fields, and reaction mode separately. ### Validation warnings When `Execute()` is skipped due to a configuration or validation failure, the component logs a warning to the Unity Console prefixed with `[ConvaiDynamicContextCommand]`. The **On Execution Skipped** event also fires. | Console message | Cause | Fix | | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `No ConvaiCharacter found on this GameObject. Assign one or disable Auto Resolve Character.` | **Auto Resolve Character** is enabled but no `ConvaiCharacter` is on the same GameObject. | Move the command to the NPC's GameObject, or disable **Auto Resolve** and assign **Character** explicitly. | | `Assign a ConvaiCharacter or enable Auto Resolve Character.` | **Auto Resolve Character** is disabled and **Character** is not assigned. | Assign the `ConvaiCharacter` reference, or enable **Auto Resolve Character**. | | `Set State requires a non-empty state name.` | **State Name** is blank or whitespace. | Enter a non-empty state name. | | `Set States requires at least one state entry.` | **State Entries** list is empty. | Add at least one Name / Value entry. | | `Each state entry requires a non-empty name.` | One or more **State Entries** have a blank name. | Fill in all entry names. | | `State entries contain duplicate name '{name}'.` | Two or more entries share the same state name. The actual duplicate name replaces `{name}` in the Console output. | Remove or rename the duplicate entry. | | `Add Event requires non-empty event text.` | **Event Text** is blank or whitespace. | Enter the event description. | | `Remove State requires a non-empty state name.` | **State Name** is blank or whitespace. | Enter the name of the state to remove. | | `Raw Update requires text unless mode is Reset.` | **Raw Text** is empty and **Raw Mode** is not `Reset`. | Enter context text, or set **Raw Mode** to `Reset`. | ### Next steps {% content-ref url="/pages/3UWNhGolZFLe1BduX0bt" %} [Dynamic context usage examples](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-usage-examples) {% endcontent-ref %} {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} # Static context at connection time Configure InitialDynamicInfoText and InitialDynamicInfoKeepInContext on ConvaiCharacter to send fixed scenario facts once at the start of each conversation. Every Convai character has two fields — **Initial Dynamic Info Text** and **Initial Dynamic Info Keep In Context** — that inject a fixed block of context into the session request at the moment the conversation connects. This context is delivered to Convai once, before the first response, and is separate from runtime Dynamic Context updates. Use this mechanism for facts that are true before the conversation begins and will not change during the session: the facility name, the character's role, the training scenario type, or the starting conditions of a drill. Use runtime Dynamic Context (`ConvaiDynamicContextCommand` or `IConvaiDynamicContext`) for everything that evolves as the session progresses. ### Inspector configuration Both fields are on the `ConvaiCharacter` component under the **Dynamic Info (Connection Request)** header.
Unity Inspector showing the Dynamic Info (Connection Request) section on ConvaiCharacter with Initial Dynamic Info Text and Initial Dynamic Info Keep In Context fields

Dynamic Info fields on ConvaiCharacter — Initial Dynamic Info Text is delivered once at session start; enable Keep In Context to retain these facts across all LLM turns for the duration of the session.

| Field | Type | Default | Description | | ------------------------------------ | -------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Initial Dynamic Info Text | `string` | *(empty)* | Free-text block sent as part of the session connection request. No format constraints — write plain sentences or key-value lines. | | Initial Dynamic Info Keep In Context | `bool` | `false` | When `true`, Convai retains this text across all LLM turns for the duration of the session. When `false`, the text informs only the first response. | {% hint style="warning" %} **Initial Dynamic Info Keep In Context defaults to `false`.** When disabled, Convai uses the initial context text to inform the character's first response only — it is not retained across turns. If you expect the character to reference initial facts throughout a long session, enable this field. Leaving it disabled is a common cause of characters appearing to "forget" scenario context after the first exchange. {% endhint %} #### Example configuration For a fire suppression certification drill, set **Initial Dynamic Info Text** to: ``` Facility: Offshore Platform Alpha Scenario: Fire Suppression Certification Drill Trainee role: Operator under assessment ``` Enable **Initial Dynamic Info Keep In Context**. The character will reference these facts throughout the conversation. You do not need to re-send them via runtime Dynamic Context — they persist for the life of the session. ### Relationship to runtime Dynamic Context Initial context and runtime Dynamic Context are complementary — not alternatives. | | Initial Dynamic Info | Runtime Dynamic Context | | ------------------------- | -------------------------------------------- | ----------------------------------------------- | | **When sent** | Once, at `ConnectAsync` | During the session, on demand | | **Content** | Fixed facts set at design time | Live state and events set at runtime | | **Suitable for** | Facility name, scenario type, character role | Trainee location, equipment state, hazard level | | **Modifiable at runtime** | No — sent once per connection | Yes — via `SetState`, `AddEvent`, `Reset` | Both mechanisms feed into the character's awareness simultaneously. Initial context provides a stable foundation; runtime Dynamic Context layers live updates on top. ```csharp // These complement the initial context — not replace it _character.DynamicContext.SetState("Station", "Chemical Storage Bay"); _character.DynamicContext.SetState("HazardLevel", "Extreme"); _character.DynamicContext.AddEvent("Trainee bypassed manual lockout procedure"); ``` ### What `Reset()` does not clear Calling `Reset()` on the runtime Dynamic Context layer clears all tracked states and events. It does **not** affect initial dynamic info: * **Initial Dynamic Info Text** was sent at connection time and cannot be recalled or re-sent by any runtime call. * **System prompt facts** on the Convai dashboard are not part of the Dynamic Context layer. No SDK call affects them at runtime. * **In-session LLM memory** — the character retains conversational context across turns within the same session. `Reset()` clears the Dynamic Context tracker and sends a Reset message to Convai, but it does not clear the model's in-session conversational memory. {% hint style="info" %} To change the initial context for a new session, end the current conversation, update the **Initial Dynamic Info Text** field, and reconnect. Initial context is sent once per `ConnectAsync` call. {% endhint %} ### Scripting access Both fields are readable via C# properties on `ConvaiCharacter`: ```csharp string initialText = _character.InitialDynamicInfoText; bool keepInContext = _character.InitialDynamicInfoKeepInContext; ``` These properties are read-only at runtime. Set the values in the Inspector before the scene is played, or via `SerializedObject` in a custom Editor tool. ### Next steps {% content-ref url="/pages/o4soWlixkVqAjbqKQCzH" %} [Command component reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/command-component-reference) {% endcontent-ref %} {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} # Sync behavior and timing Understand how the SDK transmits Replace, Append, and Reset messages for each Dynamic Context operation, including queueing and Apply() behavior. Every tracked Dynamic Context operation produces one or two RTVI `context-update` messages sent to Convai. This page documents the exact message sequence for each scenario, the pre-conversation queuing behavior, and how the SDK flushes pending context when a conversation starts. ### Canonical context format Before describing sync scenarios, it helps to understand what the SDK sends. The canonical context is a newline-separated string assembled from all tracked states and events: ``` {StateName} is {Value} {AnotherState} is {Value} Event text line one Event text line two ``` States appear first, in the order they were **first set** — not the order of the most recent update. Updating a state's value does not change its position in the output. Events follow in chronological order (call order). **Example — state insertion order is preserved across updates:** ```csharp context.SetState("Station", "Bay 3"); // position 1 context.SetState("HazardLevel", "High"); // position 2 context.AddEvent("Operator bypassed interlock"); context.SetState("Station", "Bay 7"); // updates value; position stays at 1 ``` Canonical output after all four calls: ``` Station is Bay 7 HazardLevel is High Operator bypassed interlock ``` ### Sync scenarios during active conversations #### Adding a new state **SDK call:** `SetState("Station", "Bay 3")` — `Station` has never been set. **Messages sent:** One Append. ``` mode: Append text: "Station is Bay 3" ``` The server appends this line to its existing context view. #### Updating an existing state **SDK call:** `SetState("Station", "Bay 7")` — `Station` was previously `"Bay 3"`. **Messages sent:** Two messages in sequence. **Message 1 — Replace (full canonical context with updated value):** ``` mode: Replace text: "Station is Bay 7\nHazardLevel is High\nOperator bypassed interlock" ``` **Message 2 — Append (delta for natural dialogue reference):** ``` mode: Append text: "Station changed from Bay 3 to Bay 7" ``` The Replace gives the character an authoritative complete picture of the current state. The Append gives it a natural way to reference the transition in dialogue: *"I see you've moved from Bay 3 to Bay 7."* {% hint style="info" %} Two messages for one `SetState` call on an existing state is expected behavior. If you are monitoring network traffic during debugging, expect this pattern for every existing-state modification. {% endhint %} #### Removing a state **SDK call:** `RemoveState("Station")`. **Messages sent:** One Replace containing the canonical context without the removed state. ``` mode: Replace text: "HazardLevel is High\nOperator bypassed interlock" ``` #### Batch update with `SetStates` **SDK call:** `SetStates({ "Station": "Bay 7", "HazardLevel": "Extreme" })` — `Station` existed (`"Bay 3"`), `HazardLevel` is new. Because at least one existing state was modified, two messages are sent: **Message 1 — Replace (full canonical context, all values updated):** ``` mode: Replace text: "Station is Bay 7\nHazardLevel is Extreme\n..." ``` **Message 2 — Append (all changes summarized):** ``` mode: Append text: "Station changed from Bay 3 to Bay 7\nHazardLevel is Extreme" ``` **All-new states only:** If every state in `SetStates` is new (none existed before), only one Append is sent — no Replace. The Append contains all new state lines joined by newline. #### Adding an event **SDK call:** `AddEvent("Operator bypassed interlock")`. **Messages sent:** One Append. ``` mode: Append text: "Operator bypassed interlock" ``` Events never trigger a Replace. The server appends the event text to its context view. #### Resetting all context **SDK call:** `Reset()`. **Messages sent:** One Reset-mode message. ``` mode: Reset text: null ``` The server clears its Dynamic Context view. The local tracker is also cleared — all states and events are removed. ### Pre-conversation queuing All tracked methods — `SetState`, `SetStates`, `AddEvent`, `RemoveState`, `Reset` — queue automatically when no conversation is active. When the session connects, the SDK flushes the pending queue in one of two ways: * **Pending sync (states or events queued):** A single Replace message containing the full canonical context at the moment of connection. All incremental changes are collapsed into one authoritative snapshot — they are not replayed as individual Append and Replace messages. * **Pending reset (Reset was called while offline):** A Reset message. If both a reset and subsequent changes are pending, the reset takes priority and clears the queue. **Example — all three calls queue safely before `ConnectAsync`:** ```csharp void Start() { // Safe to call before ConnectAsync — all three queue and flush at connection _character.DynamicContext.SetState("Facility", "Offshore Platform Alpha"); _character.DynamicContext.SetState("Scenario", "Fire Drill"); _character.DynamicContext.AddEvent("Session initialized"); } ``` The character receives one Replace at connection time: ``` Facility is Offshore Platform Alpha Scenario is Fire Drill Session initialized ``` ### `Apply()` exception `Apply()` bypasses the tracker entirely and does **not** queue pre-conversation. * If the character is **not** in an active conversation: the update is discarded and a warning is emitted through the Convai logger. Enable Convai debug logging to see it in the Unity Console. No queue is built — the update is lost. * If the character **is** in an active conversation: the message is sent directly to transport using the `ConvaiContextUpdateMode` you specify. * The local state tracker is not updated. `TryGetStateValue` returns `false` for keys sent via `Apply()`. {% hint style="danger" %} Do not use `Apply()` for context that must be delivered before a conversation starts. Use `SetState`, `AddEvent`, or other tracked methods instead — they queue automatically and flush on connection. {% endhint %} ### Next steps {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} {% content-ref url="/pages/7ij66qcsYVpiLPUZFGvh" %} [Troubleshoot dynamic context](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/troubleshoot-dynamic-context) {% endcontent-ref %} # Dynamic context scripting API API reference for IConvaiDynamicContext — all seven method signatures, default parameters, pre-conversation queueing behavior, and Apply() caveats. `IConvaiDynamicContext` is the C# surface for programmatic Dynamic Context control. Access it through the `DynamicContext` property on `ConvaiCharacter`: ```csharp IConvaiDynamicContext context = _character.DynamicContext; ``` The property is lazy-initialized and safe to cache for the lifetime of the component. No additional setup is required. ### Method reference #### `SetState` ```csharp void SetState(string name, string value, ConvaiContextReactionMode reaction = ConvaiContextReactionMode.SyncOnly) ``` Sets or updates one tracked state entry. If the state does not exist, it is created and appended to the canonical context in the order it was first set. If the value is identical to the current value, the call is a no-op — no update is sent. **Parameters** | Parameter | Type | Default | Description | | ---------- | --------------------------- | ---------- | ----------------------------------------------------------------------- | | `name` | `string` | — | State identifier. Must be non-empty and non-whitespace. Case-sensitive. | | `value` | `string` | — | State value. May be empty string. Cannot be `null`. | | `reaction` | `ConvaiContextReactionMode` | `SyncOnly` | Controls whether the character generates an immediate response. | **Network behavior** | Condition | Messages sent | | ----------------------------- | ----------------------------------------------------------------------------------- | | New state | One Append: `"{name} is {value}"` | | Existing state, value changed | Replace (full canonical context) + Append: `"{name} changed from {old} to {value}"` | | Identical value | None — idempotent no-op | **Pre-conversation:** queues automatically; delivered as a single Replace at connection time. #### `SetStates` ```csharp void SetStates(IReadOnlyDictionary states, ConvaiContextReactionMode reaction = ConvaiContextReactionMode.SyncOnly) ``` Sets or updates multiple tracked state entries atomically. Prefer over sequential `SetState` calls when several values change simultaneously — produces one canonical rebuild rather than multiple. **Parameters** | Parameter | Type | Default | Description | | ---------- | ------------------------------------- | ---------- | --------------------------------------------------------------- | | `states` | `IReadOnlyDictionary` | — | Map of state names to values. Must have at least one entry. | | `reaction` | `ConvaiContextReactionMode` | `SyncOnly` | Controls whether the character generates an immediate response. | **Network behavior** | Condition | Messages sent | | ------------------------------- | ------------------------------------------------------------------ | | All states are new | One Append listing all new state lines | | Any existing state changed | Replace (full canonical context) + Append (all changes summarized) | | All values identical to current | None — no-op | **Pre-conversation:** queues automatically. ```csharp _character.DynamicContext.SetStates( new Dictionary { { "Station", "Bay 7" }, { "HazardLevel", "Extreme" } }, ConvaiContextReactionMode.ReactImmediately ); ``` #### `AddEvent` ```csharp void AddEvent(string text, ConvaiContextReactionMode reaction = ConvaiContextReactionMode.Auto) ``` Appends a chronological event entry. Events accumulate in call order after all states in the canonical context. Unlike states, events are never replaced or deduplicated — each call adds a new line. **Parameters** | Parameter | Type | Default | Description | | ---------- | --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `text` | `string` | — | Event description. Must be non-empty and non-whitespace. | | `reaction` | `ConvaiContextReactionMode` | `Auto` | Controls whether the character generates an immediate response. Default is `Auto` — note the difference from `SetState` and `SetStates`, which default to `SyncOnly`. | **Network behavior:** One Append message containing the event text. **Pre-conversation:** queues automatically. #### `RemoveState` ```csharp void RemoveState(string name) ``` Removes a tracked state by name and sends an updated canonical context to Convai. If the state is not present in the tracker, the call is a no-op — no message is sent and no warning is logged. **Parameters** | Parameter | Type | Description | | --------- | -------- | ----------------------------------------------- | | `name` | `string` | Name of the state to remove. Must be non-empty. | **Network behavior:** One Replace message containing the canonical context with the state removed. `RemoveState` has no reaction mode — removal never triggers an immediate LLM response. **Pre-conversation:** queues automatically. #### `Reset` ```csharp void Reset() ``` Clears all tracked states and events and sends a Reset message to Convai. Takes no parameters. **Network behavior:** One Reset-mode message. The character's Dynamic Context view on the Convai side is cleared. **Pre-conversation:** queues automatically. {% hint style="warning" %} `Reset()` clears the runtime Dynamic Context layer only. It does not affect Initial Dynamic Info Text (sent once at connection time) or facts in the character's system prompt on the Convai dashboard. The character's in-session conversational memory is also not cleared. See [Static context at connection time](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/static-context-at-connection-time) for the full scope of what `Reset()` does and does not clear. {% endhint %} #### `TryGetStateValue` ```csharp bool TryGetStateValue(string name, out string value) ``` Reads the current value of a tracked state from the local tracker. No network call is made. **Parameters** | Parameter | Type | Description | | --------- | ------------ | ------------------------------------------------------- | | `name` | `string` | State name to look up. | | `value` | `out string` | Set to the current value if found; `null` if not found. | **Returns:** `true` if the state exists in the local tracker; `false` if it was never set, has been removed, or was sent via `Apply()` (which bypasses the tracker). ```csharp if (_character.DynamicContext.TryGetStateValue("HazardLevel", out string level)) Debug.Log($"Current hazard level: {level}"); else Debug.Log("HazardLevel state not set."); ``` #### `Apply` ```csharp void Apply(ConvaiDynamicContextUpdate update) ``` Sends a raw typed update directly to the transport layer, bypassing the local tracker. For advanced use cases that construct context text externally — for example, integrating with an external state machine that produces its own canonical text. **Parameters** | Parameter | Type | Description | | --------- | ---------------------------- | ------------------- | | `update` | `ConvaiDynamicContextUpdate` | The update to send. | **`ConvaiDynamicContextUpdate` constructor:** ```csharp new ConvaiDynamicContextUpdate( string text, ConvaiContextUpdateMode mode = ConvaiContextUpdateMode.Append, ConvaiContextReactionMode reaction = ConvaiContextReactionMode.Auto) ``` | Parameter | Type | Default | Description | | ---------- | --------------------------- | -------- | -------------------------------------------------------- | | `text` | `string` | — | Context text to send. Required unless `mode` is `Reset`. | | `mode` | `ConvaiContextUpdateMode` | `Append` | How Convai applies the text. | | `reaction` | `ConvaiContextReactionMode` | `Auto` | Whether the character responds immediately. | {% hint style="danger" %} **`Apply()` does not queue.** If the character is not in an active conversation when `Apply()` is called, the update is discarded and a warning is emitted through the Convai logger. Enable Convai debug logging to see it in the Unity Console. No queue is built — the update is lost permanently. Values sent via `Apply()` are **not** recorded in the local tracker. `TryGetStateValue` returns `false` for keys sent this way. For all standard context management, use the tracked methods (`SetState`, `SetStates`, `AddEvent`, `RemoveState`, `Reset`). `Apply()` is an escape hatch for external systems, not the primary API. {% endhint %} ### Enum reference #### `ConvaiContextReactionMode` | Value | Description | | ------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `Auto` | Convai decides whether the update warrants an immediate character response. | | `ReactImmediately` | Always triggers an immediate LLM turn after the update. Use when the character must acknowledge the change. | | `SyncOnly` | Context is updated silently. The character incorporates it into the next natural turn. No immediate response is generated. | #### `ConvaiContextUpdateMode` Used by `Apply()` and `ConvaiDynamicContextCommand` with **Raw Update** command type. | Value | Description | | --------- | --------------------------------------------------------------------------------- | | `Append` | Adds the text to the existing Dynamic Context without replacing prior content. | | `Replace` | Replaces the entire Dynamic Context with the provided text. | | `Reset` | Clears all Dynamic Context. The `text` parameter is ignored when mode is `Reset`. | ### Default reaction mode reference | Method | Default reaction | | ---------------------------------------- | ------------------------- | | `SetState` | `SyncOnly` | | `SetStates` | `SyncOnly` | | `AddEvent` | `Auto` | | `RemoveState` | *(no reaction parameter)* | | `Reset` | *(always `SyncOnly`)* | | `Apply()` / `ConvaiDynamicContextUpdate` | `Auto` | {% hint style="warning" %} `ConvaiDynamicContextCommand` (the Inspector component) defaults **Reaction Mode** to `Auto` for all command types — including `SetState` and `SetStates`. This differs from the scripting API defaults above. When switching between Inspector and scripting, verify the reaction mode is what you expect. {% endhint %} ### Next steps {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} {% content-ref url="/pages/o4soWlixkVqAjbqKQCzH" %} [Command component reference](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/command-component-reference) {% endcontent-ref %} {% content-ref url="/pages/7ij66qcsYVpiLPUZFGvh" %} [Troubleshoot dynamic context](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/troubleshoot-dynamic-context) {% endcontent-ref %} # Dynamic context usage examples Four Dynamic Context examples covering a safety drill, an onboarding walkthrough, a guided tour with timeline events, and a multi-state emergency transition. The following examples progress from a single-state Inspector setup to multi-state scripting scenarios. Each example includes the scenario context, concrete setup, and the expected runtime outcome. {% hint style="info" %} All examples assume `ConvaiManager` is in the scene with a valid API key configured, and the target NPC has a `ConvaiCharacter` component with a Character ID assigned and is able to hold a conversation. {% endhint %} ### Safety drill: station tracking **Context:** A fire suppression certification drill. A trainer NPC guides operators through suppression stations. The character must always know the operator's current station to give station-specific instructions and hazard warnings. #### Setup (Inspector) 1. Add `ConvaiDynamicContextCommand` to the trainer NPC's GameObject. 2. Set **Command Type** to **Set State**. 3. Set **State Name** to `Station`. 4. Set **State Value** to `Fire Suppression Bay`. 5. Set **Reaction Mode** to **React Immediately** — the character should acknowledge each station transition. 6. Add a trigger collider to the Fire Suppression Bay zone. Wire its `OnTriggerEnter` event to the command's `Execute()` method. Repeat with a separate child-GameObject command for each additional station, each with the appropriate **State Value**. In each child command's **Target** section, disable **Auto Resolve Character** and assign the NPC's `ConvaiCharacter` explicitly — auto-resolve only searches the same GameObject. #### Expected outcome When the operator enters the Fire Suppression Bay, `Execute()` fires. The character receives the updated `Station` state and immediately responds: > *"You've arrived at the Fire Suppression Bay. With the current extreme hazard rating, confirm your PPE is on before touching any equipment."* The `Station` state persists in the tracker. If the operator asks "Where am I?" at any point, the character answers with the current station value. ### Onboarding walkthrough: batch state update **Context:** A corporate onboarding simulation. An HR representative NPC adapts its guidance based on which items a new employee has collected and which checkpoints they have cleared. Two conditions are met simultaneously — they should be sent in one atomic update. #### Setup (Scripting) ```csharp using System.Collections.Generic; using Convai.Runtime.Components; using Convai.Runtime.DynamicContext; using UnityEngine; public class OnboardingProgressTracker : MonoBehaviour { [SerializeField] private ConvaiCharacter _hrCharacter; public void OnAccessCardAndBriefingComplete() { // Batch update — one canonical rebuild, one network round-trip _hrCharacter.DynamicContext.SetStates( new Dictionary { { "AccessCard", "Collected" }, { "SecurityBriefing", "Completed" } }, ConvaiContextReactionMode.ReactImmediately ); } public void CheckIfReadyForFloorAccess() { // Read local tracker — no network call bool hasCard = _hrCharacter.DynamicContext.TryGetStateValue("AccessCard", out string cardState) && cardState == "Collected"; bool hasBriefing = _hrCharacter.DynamicContext.TryGetStateValue("SecurityBriefing", out string briefingState) && briefingState == "Completed"; if (hasCard && hasBriefing) Debug.Log("Employee is ready for floor access."); } } ``` #### Expected outcome `OnAccessCardAndBriefingComplete()` sends one atomic update. The HR character responds immediately: > *"You've collected your access card and completed the security briefing — you're cleared for floor access. Head to Workstation 4B next."* `TryGetStateValue` reads from the local tracker with no network round-trip. It returns the current value if the state was set, or `false` if it was never set or has been removed. ### Guided tour: multiple commands and timeline events **Context:** A museum guided tour. A docent NPC tracks which exhibit is currently active and records visitor interactions as chronological events. The docent uses that history to give personalized recommendations. #### Setup (Inspector — multiple child commands) Because `ConvaiDynamicContextCommand` allows only one instance per GameObject, each command lives on a child GameObject of the NPC. **Child GameObject 1 — "SetActiveExhibit"** * Command Type: `Set State` * State Name: `ActiveExhibit` * State Value: `Ancient Rome Collection` * Reaction Mode: `SyncOnly` — the exhibit name updates silently; tour narrative drives pacing * Target → Auto Resolve Character: disabled; Character field: NPC's `ConvaiCharacter` **Child GameObject 2 — "RecordVisitorQuestion"** * Command Type: `Add Event` * Event Text: `Visitor asked about the Colosseum reconstruction` * Reaction Mode: `Auto` * Target → Auto Resolve Character: disabled; Character field: NPC's `ConvaiCharacter` **Child GameObject 3 — "RecordPhotoTaken"** * Command Type: `Add Event` * Event Text: `Visitor photographed the gladiator exhibit` * Reaction Mode: `Auto` * Target → Auto Resolve Character: disabled; Character field: NPC's `ConvaiCharacter` Wire each child command's `Execute()` to timeline markers, interaction zones, or UI buttons. Wire the **On Executed** event on each child to drive UI feedback — highlight exhibit cards, update tour progress — without additional scripting. #### Expected outcome As the visitor progresses, the docent's canonical context accumulates: ``` ActiveExhibit is Ancient Rome Collection Visitor asked about the Colosseum reconstruction Visitor photographed the gladiator exhibit ``` The docent references both the current exhibit and the visitor's specific interactions: > *"Since you photographed the gladiator exhibit, you might enjoy the additional display on Roman military equipment in the next room."* ### Emergency response: multi-state transition **Context:** An industrial safety simulation. A supervisor NPC must respond to a simultaneous shift from routine inspection to emergency mode — three conditions change at once, and the character must acknowledge all of them immediately. #### Setup (Scripting) ```csharp using System.Collections.Generic; using Convai.Runtime.Components; using Convai.Runtime.DynamicContext; using UnityEngine; public class EmergencyResponseController : MonoBehaviour { [SerializeField] private ConvaiCharacter _supervisorCharacter; public void TriggerChemicalLeak() { // All three states change simultaneously — one atomic update, one canonical rebuild _supervisorCharacter.DynamicContext.SetStates( new Dictionary { { "OperationMode", "Emergency" }, { "HazardType", "Chemical Leak — Bay 7" }, { "EvacuationStatus", "In Progress" } }, ConvaiContextReactionMode.ReactImmediately ); // Log the triggering event after the state batch _supervisorCharacter.DynamicContext.AddEvent( "Chemical leak alarm triggered at Bay 7 — automated ventilation engaged", ConvaiContextReactionMode.SyncOnly ); } } ``` #### Expected outcome The supervisor character receives three state updates and one event. The `ReactImmediately` mode on `SetStates` triggers an immediate response acknowledging all simultaneous changes: > *"Chemical leak at Bay 7 — all personnel evacuate the east wing immediately. Bay 7 ventilation is engaged. Do not re-enter until the all-clear is given."* Using `SetStates` for three simultaneous transitions produces one canonical rebuild rather than three sequential ones, ensuring the character receives a coherent picture rather than three partial updates. ### Next steps {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} # Troubleshoot dynamic context Diagnose Dynamic Context issues with a five-step checklist, a symptom table, a character-not-responding decision tree, and the full Console log reference. Most Dynamic Context problems fall into one of three categories: incorrect timing (calling `Apply()` before a conversation starts), a misconfigured component (missing `ConvaiCharacter` reference or empty required field), or a misunderstanding of what `Reset()` clears. Work through the first-line investigation checklist below — most issues resolve at step 2 or 3. ### First-line investigation {% stepper %} {% step %} #### Check the Unity Console for warnings `ConvaiDynamicContextCommand` logs a warning with the full validation message every time `Execute()` is skipped. Open the Console (**Window → General → Console**) and look for messages tagged `[ConvaiDynamicContextCommand]`. If you see a warning, find the exact message in the [Console log reference](#console-log-reference) table below and follow the listed fix. {% endstep %} {% step %} #### Use the Sample UI to isolate the issue Before debugging your own integration, verify that the Dynamic Context system itself is working by using the SDK's built-in test UI. **Prefab path:** Packages/space.vars.sdk\_package\_id/Prefabs/SampleDynamicContextUI.prefab Drop it into your scene, assign your `ConvaiCharacter`, enter Play Mode, and use the **Set State** button to send a known value. If the character responds correctly through the Sample UI, the issue is in your integration code — not in the Dynamic Context system itself. {% endstep %} {% step %} #### Verify the character reference is resolved Select the `ConvaiDynamicContextCommand` component in the Inspector. If the **Target** section shows a yellow warning, the component cannot find a `ConvaiCharacter`. * **Auto Resolve Character enabled:** confirm that `ConvaiDynamicContextCommand` and `ConvaiCharacter` are on the **same GameObject**. * **Auto Resolve Character disabled:** confirm that the **Character** field has a reference assigned. {% endstep %} {% step %} #### Check whether the character was in a conversation Context updates sent via `Apply()` are discarded if the character is not in an active conversation. A warning is emitted through the Convai logger — enable Convai debug logging to see it in the Unity Console. If you are calling `Apply()` in `Awake()`, `Start()`, or before the session connects, switch to `SetState` or `AddEvent` instead. These methods queue automatically and flush when the conversation begins. {% endstep %} {% step %} #### Check the reaction mode If a context update reached the character but it did not respond immediately, verify the **Reaction Mode** setting. * **`SyncOnly`** — context is stored silently. The character incorporates it into its next natural turn. No immediate response is generated. This is expected behavior. * **`Auto`** — Convai decides whether to respond. For guaranteed immediate response, use `ReactImmediately`. * **`ReactImmediately`** — always triggers an immediate LLM turn after the update. {% endstep %} {% endstepper %} ### Common issues | Symptom | Likely cause | Fix | | ---------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | | Character does not reference context updates at all | `Apply()` called before conversation started | Switch to `SetState` or `AddEvent` — they queue automatically | | Character does not respond immediately after update | Reaction mode is `SyncOnly` | Change **Reaction Mode** to `Auto` or `ReactImmediately` | | **On Execution Skipped** fires instead of **On Executed** | Validation failure — see Console warning | Check the [Console log reference](#console-log-reference) for the exact message and fix | | Inspector shows yellow warning on the component | No `ConvaiCharacter` resolved | Enable **Auto Resolve** and place both components on the same GameObject, or assign **Character** explicitly | | `TryGetStateValue` returns `false` after `Apply()` | `Apply()` bypasses the local tracker | Use `SetState` if the value needs to be queryable via `TryGetStateValue` | | Two `context-update` messages sent for one `SetState` call | Existing state was updated — expected behavior | Not a bug. See [Two messages for one update](#two-messages-for-one-update) | | Character still references facts after `Reset()` | Initial context or system prompt is not cleared by `Reset()` | See [Reset did not clear everything](#reset-did-not-clear-everything) | | Character references initial scenario facts only in the first response | `InitialDynamicInfoKeepInContext` is `false` (the default) | Enable **Initial Dynamic Info Keep In Context** on `ConvaiCharacter` | | Cannot add a second Command component to the same GameObject | `[DisallowMultipleComponent]` restriction | Place additional commands on child GameObjects | | `SetStates` list produces a validation warning | Duplicate state names or empty entries in list | Remove duplicates; ensure every entry has a non-empty name | ### Character does not reference context updates When the character appears completely unaware of a state or event you sent, work through the following in order. **Verify the method used.** `Apply()` is discarded if the character is not in an active conversation. A warning is emitted through the Convai logger (visible in the Console when debug logging is enabled), but no queue is built. Substitute `SetState` or `AddEvent` instead — these queue automatically and flush when the session opens. **Verify `Execute()` was not skipped.** Wire the `ConvaiDynamicContextCommand` **On Execution Skipped** event to a temporary `Debug.Log` call during development. If it fires, the Console shows the exact validation message that caused the skip. **Verify the reaction mode.** If reaction mode is `SyncOnly`, the character received the update but will not generate an immediate response — it references the new state in its next natural turn. Switch to `Auto` or `ReactImmediately` if immediate acknowledgement is required. ### `Apply()` has no effect `Apply()` has two behaviors that differ from every other `IConvaiDynamicContext` method: * **No pre-conversation queue.** If the character is not in an active conversation when `Apply()` is called, the update is discarded immediately. A warning is emitted through the Convai logger — enable Convai debug logging to see it in the Unity Console. No queue is built. Use `SetState`, `AddEvent`, or another tracked method if the call may happen before the session starts. * **No tracker update.** `Apply()` sends directly to transport without touching the local state tracker. A subsequent call to `TryGetStateValue` for a key sent via `Apply()` returns `false`. If the value needs to be readable via `TryGetStateValue`, use `SetState` instead. {% hint style="warning" %} `Apply()` is an advanced escape hatch for external systems that construct their own context text. For all standard context management, prefer the tracked methods — `SetState`, `SetStates`, `AddEvent`, `RemoveState`, and `Reset` — which queue automatically and keep the local tracker consistent. {% endhint %} ### Reset did not clear everything `Reset()` operates on the **runtime Dynamic Context layer only**. Three sources of character knowledge are outside its scope: **Initial Dynamic Info Text.** The content of **Initial Dynamic Info Text** on `ConvaiCharacter` is sent once at connection time as part of the session request. `Reset()` does not re-send it and cannot clear it. Ending and restarting the session is the only way to change what was sent at connection time. **System prompt.** Facts baked into the character's system prompt on the Convai dashboard are not part of the Dynamic Context layer. No SDK call affects them at runtime. **In-session LLM memory.** The character's language model retains conversational context across turns within the same session. `Reset()` clears the Dynamic Context tracker and sends a Reset message to Convai, but it does not clear the model's in-session conversational memory. ### Two messages for one update When you call `SetState` on a state that already exists with a different value, two `context-update` RTVI messages are sent in sequence: 1. A **Replace** message carrying the full canonical context with the updated value in place. 2. An **Append** message carrying the human-readable delta: `"{name} changed from {oldValue} to {newValue}"`. This is intentional and non-configurable. The Replace gives the character an authoritative complete picture; the Append gives it a natural way to reference the transition in dialogue. If you are monitoring network traffic during debugging, expect this pattern for every existing-state modification. See [Updating an existing state](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing#updating-an-existing-state) in Sync behavior and timing for the full sequence. ### Cannot add multiple command components to the same GameObject `ConvaiDynamicContextCommand` is marked `[DisallowMultipleComponent]`. Unity prevents a second instance from being added to the same GameObject. Place each additional command on a **child GameObject** of the NPC. In the **Target** section of each command, disable **Auto Resolve Character** and assign the NPC's `ConvaiCharacter` explicitly — auto-resolve only searches the same GameObject, not parents or children. ### Character not responding The following decision tree covers the full troubleshooting surface for context updates that appear to have no effect. ```mermaid flowchart TD A[Character not responding to context update] --> B{Which entry point?} B -- ConvaiDynamicContextCommand --> C{Did OnExecuted fire?} C -- No --> D[Check Console for\nvalidation warning\nSee Console Log Reference] C -- Yes --> E{Reaction mode?} B -- IConvaiDynamicContext script --> F{Which method?} F -- Apply --> G{In active conversation?} G -- No --> H[Apply discarded\nUse SetState or AddEvent] G -- Yes --> E F -- SetState / AddEvent / etc --> E E -- SyncOnly --> I[Expected — no immediate response\nSwitch to Auto or ReactImmediately] E -- Auto or ReactImmediately --> J{Did update reach backend?} J -- Yes --> K[Check character system prompt\nand dashboard configuration] J -- Unsure --> L[Use Sample UI prefab\nto verify delivery in isolation] ``` ### Console log reference The following messages appear in the Unity Console during Dynamic Context operations. All `ConvaiDynamicContextCommand` warnings are prefixed with `[ConvaiDynamicContextCommand]`. | Message | Source | Meaning | Fix | | -------------------------------------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `No ConvaiCharacter found on this GameObject. Assign one or disable Auto Resolve Character.` | `Execute()` validation | **Auto Resolve** is on but no `ConvaiCharacter` is on the same GameObject. | Move the command to the NPC's GameObject, or disable **Auto Resolve** and assign **Character** explicitly. | | `Assign a ConvaiCharacter or enable Auto Resolve Character.` | `Execute()` validation | **Auto Resolve** is off and **Character** is not assigned. | Assign the `ConvaiCharacter` reference, or enable **Auto Resolve**. | | `Set State requires a non-empty state name.` | `Execute()` validation | **State Name** field is blank or whitespace. | Enter a non-empty state name. | | `Set States requires at least one state entry.` | `Execute()` validation | **State Entries** list is empty. | Add at least one Name / Value entry. | | `Each state entry requires a non-empty name.` | `Execute()` validation | One or more **State Entries** have a blank name. | Fill in all entry names. | | `State entries contain duplicate name '{name}'.` | `Execute()` validation | Two entries in **State Entries** share the same state name. The actual name replaces `{name}` in the Console. | Remove or rename the duplicate. | | `Add Event requires non-empty event text.` | `Execute()` validation | **Event Text** field is blank or whitespace. | Enter the event text. | | `Remove State requires a non-empty state name.` | `Execute()` validation | **State Name** field is blank or whitespace. | Enter the name of the state to remove. | | `Raw Update requires text unless mode is Reset.` | `Execute()` validation | **Raw Text** is empty and **Raw Mode** is not `Reset`. | Enter text, or set **Raw Mode** to `Reset`. | ### Next steps {% content-ref url="/pages/jLDVtNvrQQtepjrSEeAv" %} [Sync behavior and timing](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/sync-behavior-and-timing) {% endcontent-ref %} {% content-ref url="/pages/Pwb170m39yjzkfAghzLG" %} [Dynamic context scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/dynamic-context/dynamic-context-scripting-api) {% endcontent-ref %} # Emotion Find guides for adding emotionally responsive facial animation to Convai characters — from quick setup to scripting API and troubleshooting. The Emotion system translates Convai AI emotional signals into live facial animation, driving blendshapes, Animator parameters, or both simultaneously. This section covers everything from a first working setup to custom taxonomy authoring and the complete scripting API.
How the emotion system works
Understand the pipeline, key concepts, and required component placement.		/pages/VKDUIdxDSqYZKjcrf1FW
Emotion quick start
Attach the Emotion Controller, assign the bundled profile, and see your NPC react emotionally to conversation — no custom assets required.		/pages/PyOgkztYoFXIQ0l3uTfN
Emotion profile
Configure smoothing speed, micro-expression bursts, neutral alternation, and output bindings in one portable asset.		/pages/4kPK7vkbjtuDa9qfc75O
Emotion output bindings
Map smoothed emotion scores to facial blendshapes and Animator float parameters, with per-slot weight and LipSync control.		/pages/mqwNGzveKocvvR9sOhts
Emotion taxonomy
Understand the built-in Plutchik vocabulary, how server aliases are resolved, and how to author a custom taxonomy.		/pages/FOiJ57aQtQ6qN404z3rE
Emotion scripting API
Read live emotion state, inject overrides, lock expressions, and react to emotion events — from Inspector relays to typed C# subscriptions.		/pages/OM0Nzbb46ldsnhmYvu6a
Emotion examples
Complete scenarios covering hazard triggers, locked greetings, distress branching, analytics logging, and Editor debugging.		/pages/5Bo6oC1qF0jfrzgrtD56
Troubleshoot emotion
Step-by-step fixes for the most common problems — from expressions that will not move to LipSync conflicts and production build issues.		/pages/N6zov1sJHXMKI44gRL56
### Next steps Start with [Emotion quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/quick-start) to get expressions running on your first NPC. Then read [How the emotion system works](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/how-the-emotion-system-works) to understand the pipeline and component roles before moving into configuration and reference pages. # How the emotion system works Understand the emotion pipeline — how Convai sends emotion signals, how the SDK resolves and smooths them, and which components to place and where. The Convai emotion system translates server emotion signals into live facial animation through a four-stage pipeline. This page explains how each stage works, what the required components do, and where to place them in your scene. ### How the emotion pipeline works Every emotion signal travels through four stages: ```mermaid flowchart TD A([Convai Backend]) -->|RTVI bot-emotion message\nemotion label + scale 1–3| B[RTVIBotEmotionMessage] B --> C[CharacterEmotionChanged\ndomain event] C --> D[ConvaiEmotionController] D --> E{Taxonomy\nresolution} E -->|alias → canonical label| F[EmotionScoreAccumulator\nsmoothing · micro-burst] F --> G[NeutralAlternator\nperiodic fade-to-neutral] G --> H{Output bindings} H --> I[BlendshapeEmotionBinding\nfacial blendshapes] H --> J[AnimatorParameterEmotionBinding\nAnimator float params] I & J --> K([EmotionReading\nread by your scripts]) ``` The backend sends a short emotion label (for example `"happy"`) and an intensity on a 1–3 scale. The **taxonomy** resolves that label to its canonical form (`"joy"`), normalises the intensity to a 0–1 score, and hands it to the **score accumulator**, which applies exponential smoothing and an optional micro-expression burst. The **neutral alternator** periodically blends the expression back toward neutral to prevent a frozen face during long turns. The smoothed scores are then written to blendshapes and Animator parameters through configurable **output bindings**. ### Key concepts | Concept | What it is | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ConvaiEmotionController` | The MonoBehaviour that owns the entire pipeline for one NPC. Add one per character. | | `ConvaiEmotionProfile` | A ScriptableObject asset that holds every tunable parameter: smoothing, micro-burst, neutral alternation, and output slot definitions. | | `EmotionTaxonomyAsset` | A ScriptableObject that defines the emotion vocabulary — canonical labels, server aliases, and mouth influence hints. The built-in default is Plutchik's nine emotions including neutral. | | Output bindings | `BlendshapeEmotionBinding` and `AnimatorParameterEmotionBinding` map each canonical emotion label to mesh blendshape names or Animator float parameters. | | `EmotionReading` | An immutable snapshot of the current emotional state: dominant label, dominant score, all scores, and mouth influence hint for LipSync. Available every frame via `ConvaiEmotionController.Current`. | | Micro-burst | A short overshoot applied when a new emotion arrives, giving expressions a punchy entry before settling to their sustained level. | | Neutral alternation | A timer that periodically fades the active expression toward neutral and back, preventing the character's face from locking into a single pose during long turns. | | `ConvaiCharacterEventRelay` | An Inspector-friendly component that exposes emotion change callbacks as Unity Events — no code required. | ### Component placement | Component | Where to place it | Notes | | --------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `ConvaiEmotionController` | On the NPC's root GameObject, alongside the Embodiment component | One per character | | `ConvaiEmotionProfile` | Anywhere in your `Assets/` folder as a ScriptableObject asset | Shared across multiple NPC prefabs if needed | | `EmotionTaxonomyAsset` | Anywhere in your `Assets/` folder | Optional — omit to use the built-in Plutchik set | | `ConvaiCharacterEventRelay` | On any GameObject in the scene | Auto-resolves `ConvaiCharacter` on the same GameObject; drag a different character if needed | ### Next steps {% content-ref url="/pages/PyOgkztYoFXIQ0l3uTfN" %} [Emotion quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/quick-start) {% endcontent-ref %} {% content-ref url="/pages/4kPK7vkbjtuDa9qfc75O" %} [Emotion profile](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/emotion-profile) {% endcontent-ref %} {% content-ref url="/pages/OM0Nzbb46ldsnhmYvu6a" %} [Emotion scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/scripting-api) {% endcontent-ref %} # Emotion quick start Build a working emotion pipeline on a Convai NPC — attach the Emotion Controller, assign the bundled sample profile, and verify expressions in Play Mode. We will attach the Emotion Controller to an NPC, assign the bundled sample profile, and see the character's face react to live AI emotion signals in Play Mode. No custom assets are required for the initial setup. ### Prerequisites Before starting, verify: * [ ] A `ConvaiCharacter` is in the scene and responds to speech in Play Mode ### Set up the Emotion Controller {% stepper %} {% step %} #### Add the Emotion Controller Select your NPC's root GameObject in the Hierarchy. In the Inspector, click **Add Component** and search for **Emotion Controller**, or navigate to **Convai → Embodiment → Emotion Controller**. The component appears with a **Profile** field that is currently empty.
Unity Inspector showing ConvaiEmotionController added to the NPC root GameObject with the Profile field empty, ready for a profile asset to be assigned

ConvaiEmotionController added to the NPC root — the Profile field is empty until an EmotionProfile asset is assigned in step 3. No blendshape mapping is active yet.

{% endstep %} {% step %} #### Locate the bundled sample profile In the Project window, navigate to: {% code overflow="wrap" %} ``` Packages / Convai SDK for Unity / SamplesShared / Resources / Embodiment / Modules / Emotion ``` {% endcode %} You will find two assets: | Asset | Purpose | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `ConvaiSamplesShared_EmotionProfile` | Pre-configured expression slots for Reallusion characters, with smoothing, micro-burst, and neutral alternation already tuned. | | `ConvaiSamplesShared_EmotionTaxonomy` | The default emotion vocabulary, already referenced by the profile above. | | {% endstep %} | | {% step %} #### Assign the profile Drag `ConvaiSamplesShared_EmotionProfile` from the Project window into the **Profile** field on the `ConvaiEmotionController` component.
Unity Inspector showing ConvaiEmotionController with ConvaiSamplesShared_EmotionProfile assigned to the Profile field

ConvaiSamplesShared_EmotionProfile assigned — the controller is now configured with pre-tuned expression slots for Reallusion characters and will begin driving blendshapes as soon as Play Mode starts.

{% hint style="warning" %} The bundled asset is **read-only** — it lives inside the package. To adjust any settings, duplicate it first (**Ctrl+D** on Windows / **Cmd+D** on macOS), move the copy into your own `Assets/` folder, and assign the copy instead. {% endhint %} {% endstep %} {% step %} #### Enter Play Mode and speak Press **Play**. Talk to the character using your configured microphone. As the AI responds, watch the `ConvaiEmotionController` in the Inspector — the **Current** reading updates live. {% endstep %} {% endstepper %} {% hint style="success" %} **Expected result:** The NPC's facial expression changes as the conversation develops. The **Current → Dominant Label** field shows the active emotion and **Current → Dominant Score** shows its smoothed intensity. If you are using a Reallusion character with the default rig, blendshapes activate on the character's face immediately. {% endhint %} ### How it works When you spoke to the character, `ConvaiEmotionController` received the backend's emotion signal, resolved it through the taxonomy (mapping `"happy"` to `"joy"`), smoothed the intensity score over time, and wrote the score to the character's facial blendshapes every frame. For a full explanation of every stage, see [How the emotion system works](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/how-the-emotion-system-works). The bundled profile is configured for Reallusion characters. For other rigs, duplicate the profile and update the blendshape names in each slot to match your character's shapes. See [Emotion profile](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/emotion-profile) and [Emotion output bindings](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/output-bindings) for how to configure slots for any rig. ### Next steps The quick start runs end-to-end with the bundled profile. These pages cover tuning and extending the setup. {% content-ref url="/pages/VKDUIdxDSqYZKjcrf1FW" %} [How the emotion system works](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/how-the-emotion-system-works) {% endcontent-ref %} {% content-ref url="/pages/4kPK7vkbjtuDa9qfc75O" %} [Emotion profile](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/emotion-profile) {% endcontent-ref %} {% content-ref url="/pages/mqwNGzveKocvvR9sOhts" %} [Emotion output bindings](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/output-bindings) {% endcontent-ref %} {% content-ref url="/pages/OM0Nzbb46ldsnhmYvu6a" %} [Emotion scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/emotion/scripting-api) {% endcontent-ref %} --- [Next Page](/api-docs/llms-full.txt/1)