What is the Convai Unreal Engine plugin

Understand what the Convai Unreal Engine plugin adds to a project: Blueprint components, a real-time voice pipeline, and opt-in feature add-ons.

The Convai Unreal Engine plugin connects Unreal Engine 5 projects to Convai so in-scene actors can speak, listen, reason, and act in real time. A player speaks into a microphone or sends text; the plugin captures input, streams it to Convai for speech recognition and language understanding, generates a response with text-to-speech, and plays it back on the character with synchronized lip sync, facial emotion, and optional in-scene actions. The plugin targets Unreal developers building training simulations, interactive experiences, and games.

What it includes

The plugin ships with a complete conversation pipeline and a set of opt-in feature add-ons.

Conversation pipeline

Always active once connected:

• Real-time voice input — microphone capture through UConvaiAudioCaptureComponent, streamed to Convai over WebRTC
• Language understanding and generation — Convai processes speech and responds in character
• Text-to-speech playback — voice generated by Convai, played back through UConvaiChatbotComponent

Feature add-ons

Opt-in, each configured through Blueprint or the Details panel:

• Character actions — characters execute structured in-scene commands (Move To, Follow, custom) dispatched from Convai
• Dynamic context — push live world state and events into character knowledge at runtime
• Emotion — Convai infers emotion from conversation and drives blendshape expressions on the character
• Long-term memory — characters remember each player across sessions using an end-user ID
• Narrative design — trigger scripted conversation branches and sections by name from Blueprint
• Scene metadata — tag level actors so characters are aware of and can act on them
• Vision — stream camera frames to Convai so characters can describe what they see
• Gaze attention — route the object in the player's gaze direction as context to the active character
• Lip sync — Convai-provided face animation data drives blendshape mouth movement in sync with speech audio, with selectable Viseme Based, MetaHuman Blendshapes, ARKit Blendshapes, and CC4 Extended Blendshapes modes for different rigs

Editor tooling

The ConvaiEditor module adds an in-editor configuration window for API key setup, the character dashboard browser, and Blueprint graph utilities such as the Create Convai Action Handler right-click entry.

Relationship to Convai

Convai hosts the language model, voice synthesis, narrative design engine, long-term memory store, and character configuration. The plugin is a client-side integration layer: it captures audio or text from the player, streams it to Convai over WebRTC, and delivers the response back to the character's audio and animation pipeline.

The plugin runs client-side integration logic inside the game process, including session routing, audio playback, action handling, dynamic context, gaze attention, and scene metadata. It does not bundle a local language model or Convai decision engine. Character configuration and conversation decisions remain in Convai, so dashboard-side changes do not require recompiling the Unreal project.

Voice → Convai → character flow

UConvaiPlayerComponent captures microphone audio and sends it through a player session proxy. UConvaiChatbotComponent holds a character session proxy. UConvaiSubsystem owns the underlying Convai client and routes data between those proxies and Convai. The chatbot receives response data and coordinates audio playback, lip-sync processing, and Blueprint action handlers.

Blueprint-first design

The main gameplay workflows are exposed to Blueprint graphs. C++ access is available but secondary; most projects can build character logic through Blueprint components, events, and callable nodes, then use C++ only for project-specific extensions.

Requirements

Requirement	Minimum
Unreal Engine	5.0
Platforms	Win64, Android
Network	Internet connection to Convai
API key	Account at https://convai.com

Android requires microphone permission handling. The plugin bundles the AndroidPermission engine plugin as a dependency and requests RECORD_AUDIO permission automatically when it connects to Convai.

The Convai Unreal Engine plugin is available on Fab. Plugin releases are also published on GitHub.

For the full platform and engine version support matrix, see Compatibility and requirements.

Compatibility and requirements

Next steps

Install the plugin and add your first AI character.

Getting started

To understand the module and component structure before building, see the architecture page.

Plugin architecture

For a one-screen overview of every feature and links to the corresponding guides, see the feature map.

Feature map