Vision quick start

Add scene vision to a Convai character Blueprint and verify that captured frames affect the character's response in Play In Editor.

We will add an Environment Webcam component to an existing Convai character, assign a render target, and start capture from Event BeginPlay. At the end, the character should respond to something visible in front of the capture component during Play In Editor.

Prerequisites

• The Convai Unreal Engine plugin version 4.0.0-beta.21 is installed and enabled. See Install the plugin if needed.
• Your API key is configured. See Configure the API key.
• Your level has a character with a Convai Chatbot component (UConvaiChatbotComponent) and a valid CharacterID. See Add your first Convai character if you have not completed that setup.
• A Convai Player component (UConvaiPlayerComponent) is present so you can start a conversation in Play In Editor.

Vision uses scene capture from the character's viewpoint, not a physical device webcam. The shipped component is Environment Webcam (UEnvironmentWebcam).

1

Open the character Blueprint

In the Content Browser, double-click the Blueprint for your Convai character to open the Blueprint Editor.

2

Add the Environment Webcam component

In the Components panel, click Add and search for EnvironmentWebcam. Select Environment Webcam to add it to the same Actor as the Convai Chatbot component.

3

Position the component

In the Viewport, select Environment Webcam. Move it near the character's eyes and rotate it so its +X axis faces the part of the level the character should see.

4

Create a render target asset

In the Content Browser, right-click in an empty area, open the Convai submenu, and choose Vision Render Target (the context menu label in the current plugin build). Name the asset, for example RT_ConvaiVision, and save it.

The Convai render target action creates a UTextureRenderTarget2D with RTF_RGBA8, a black clear color, and a default size of 512 x 512.

5

Assign the render target

Select Environment Webcam in the character Blueprint. In the Details panel, assign the asset to Convai Render Target under Convai | Vision.

6

Start capture from BeginPlay

Open the Event Graph. Drag Environment Webcam from the Components panel into the graph. From the component pin, call Start (category Convai | Vision). Connect Event BeginPlay to Start.

7

Compile and save

Click Compile, then Save the Blueprint.

Verify vision is working

Place a distinct object in front of Environment Webcam. Enter Play In Editor, start a conversation, and ask about that object, such as its color or location.

Check these signals in order:

1. On the Convai Chatbot component, call Supports Vision. It should return true.
2. On Environment Webcam, call Get State. It should return Capturing.
3. Open the render target asset in the Content Browser while Play In Editor is running. The preview should show scene content, not a blank image.
4. In the Output Log, look for SendImage: Sending raw image from ConvaiChatbotComponentLog.

Vision is working when Supports Vision returns true, Get State returns Capturing, the render target preview updates during Play In Editor, and the character's response references the visible object.

If any check fails, see Troubleshoot vision.

Next steps

Vision frame sourcesVision usage examplesHow vision works