Frame Sources

Select and configure the right frame source for your platform. Covers all Inspector fields, capture presets, platform requirements, and source state diagnostics.

Choosing and Configuring a Vision Frame Source

A frame source is the component responsible for capturing images and making them available to the publisher as a Y-flipped RenderTexture. The Convai SDK ships three built-in frame sources — one for Unity scene cameras, one for physical webcams, and one for the Meta Quest passthrough camera — each optimised for its target platform. This page covers how to add, configure, and troubleshoot all three.

Selecting a Frame Source

Frame source
Best for
Platforms

CameraVisionFrameSource

Streaming any Unity scene camera — main camera, security camera, overhead view

PC, Mac, Android, iOS, console

WebcamVisionFrameSource

Streaming a physical camera device attached to the player's machine or mobile device

PC, Mac, Android, iOS

QuestVisionFrameSource

Streaming the real-world passthrough feed on a Meta Quest headset

Meta Quest (requires Meta XR SDK)

Add a frame source via Add Component and type the class name, or navigate the component menu under Convai → Vision.


CameraVisionFrameSource

CameraVisionFrameSource captures a Unity Camera's output into a RenderTexture and provides it to the publisher on every frame. It automatically selects the correct capture backend for the active render pipeline (built-in or SRP/URP) when Camera Capture Mode is set to Auto.

Component menu path: Convai/Vision/Camera Vision Frame Source

Inspector Reference

Capture Settings

Field
Type
Default
Description

Capture Preset

CapturePreset

Balanced

Selects a preconfigured resolution and frame-rate combination. Set to Custom to enter values manually.

Capture Width

int

Output width in pixels. Active only when preset is Custom.

Capture Height

int

Output height in pixels. Active only when preset is Custom.

Target Fps

int

Target capture frame rate. Active only when preset is Custom.

Camera Capture Mode

CameraCaptureMode

Auto

Selects the render-pipeline capture strategy. Leave at Auto unless directed otherwise.

Camera

Field
Type
Default
Description

Target Camera

Camera

(auto-resolved)

The camera to capture. If left blank, resolved to Camera.main at runtime.

Debug

Field
Type
Default
Description

Source Id

string

"camera"

An identifier used in domain events and multi-source scenarios.

Enable Diagnostic Frame Health Probe

bool

false

Performs a synchronous per-frame pixel readback to validate frame content. Use only when diagnosing black-frame issues; incurs GPU readback cost every frame.

Capture Preset Values

Preset
Width
Height
FPS
Use case

LowOverhead

640

480

10

High-volume deployments, mobile, or bandwidth-constrained environments

Balanced

1280

720

15

General purpose — default for most scenarios

HighDetail

1920

1080

30

Scenes where fine visual detail is important to AI comprehension

Custom

(set manually)

(set manually)

(set manually)

Full control over dimensions and frame rate

Camera Capture Mode Values

Mode
When to use

Auto

Default. Detects the active render pipeline and selects the appropriate backend automatically.

BuiltInHooks

Forces use of Camera.onPreRender / Camera.onPostRender callbacks. Only compatible with the Built-in Render Pipeline.

SrpNative

Forces the SRP/URP explicit render path (TargetCamera.Render() in LateUpdate). Use when Auto fails to detect SRP correctly.

ExplicitRenderCompatibility

A compatibility fallback that explicitly renders the camera. Useful for highly customised render setups.


WebcamVisionFrameSource

WebcamVisionFrameSource captures a physical camera device using Unity's WebCamTexture API and converts the output to a RenderTexture. It handles device selection, permission requests (on Android and iOS), automatic rotation correction, and resolution clamping.

Component menu path: Convai/Vision/Webcam Vision Frame Source

Inspector Reference

Webcam Settings

Field
Type
Default
Description

Webcam Device Name

string

""

Name of the webcam device to open. An empty string selects the first available device.

Requested Width

int

640

Requested capture width sent to the driver. The actual resolution may differ.

Requested Height

int

480

Requested capture height sent to the driver.

Requested Fps

int

15

Requested frame rate sent to the driver.

Max Output Width

int

1280

Maximum width of the output RenderTexture. Frames wider than this are scaled down. Set to 0 to disable scaling.

Max Output Height

int

720

Maximum height of the output RenderTexture.

Source Identification

Field
Type
Default
Description

Source Id

string

"webcam"

Identifier used in domain events and multi-source scenarios.

Listing Available Devices

To enumerate connected webcam devices at runtime:

Switching Devices at Runtime

To switch to a different webcam without stopping the session:

Permission Flow (Android / iOS)

On Android and iOS the component requests camera permission asynchronously when StartCapture() is called. The State property transitions through:

Idle → AwaitingPermission → Starting → Ready

If the user denies permission, State becomes Failed and ErrorKind is set to PermissionDenied. You can monitor this via IVisionFrameSourceStatusProvider.StatusChanged — see Advanced Topics for details.

On Android and iOS the system camera permission dialog appears the first time StartCapture() runs. Ensure your app's manifest or Info.plist declares the camera usage description before shipping.


QuestVisionFrameSource

QuestVisionFrameSource streams the real-world passthrough feed from a Meta Quest headset, giving Convai characters a live view of the physical environment the user is standing in. The component binds to Meta's PassthroughCameraAccess API via reflection so that the SDK does not take a hard compile-time dependency on the Meta XR package.

Component menu path: Convai/Vision/Quest Vision Frame Source

Inspector Reference

Quest Camera Access

Field
Type
Default
Description

Passthrough Camera Access

MonoBehaviour

(auto-discovered)

Reference to a PassthroughCameraAccess component in the scene. Leave blank to auto-find.

Output Settings

Field
Type
Default
Description

Source Id

string

"quest-passthrough"

Identifier used in domain events.

Max Output Width

int

1280

Maximum width of the output RenderTexture.

Max Output Height

int

720

Maximum height of the output RenderTexture.

Target Frame Rate

int

15

Target frames per second for capture.

Flip Y

bool

true

Flips the passthrough texture vertically. Required for correct video orientation; disable only if the Meta SDK changes its coordinate conventions.

The binding to PassthroughCameraAccess is established via reflection at runtime. If the Meta XR package is updated and PassthroughCameraAccess changes its API, the component will log an error and enter Failed state. Check for SDK updates in that case.


Source State Reference

All three frame sources implement IVisionFrameSourceStatusProvider, which exposes a State property and a StatusChanged event.

VisionSourceState

State
Meaning

Idle

Capture has not been started.

AwaitingPermission

Waiting for the user to grant camera permission (Android / iOS).

Starting

Capture is initialising — device is opening, RenderTextures are being created.

Ready

Capture is running and frames are being produced.

Degraded

Capture is running but frame health checks are detecting issues (e.g., consecutive blank frames).

Stopped

Capture was stopped normally.

Failed

Capture failed and cannot continue. Check ErrorKind and StatusMessage for details.

VisionSourceErrorKind

Error kind
Cause

None

No error.

Timeout

The source did not produce a usable frame within the expected time window.

PermissionDenied

Camera permission was denied by the user or the OS.

UnsupportedPlatform

The source is not supported on this platform (e.g., QuestVisionFrameSource on PC).

DeviceUnavailable

The requested camera device could not be opened.

InvalidConfiguration

A field value is out of range or inconsistent (check StatusMessage).

Unknown

An unexpected error occurred.

For scripting against these states and responding to StatusChanged events, see Advanced Topics.

Conclusion

Each frame source targets a specific platform and capture scenario — use CameraVisionFrameSource for Unity scene cameras, WebcamVisionFrameSource for physical devices on desktop and mobile, and QuestVisionFrameSource for Meta Quest passthrough. With your frame source configured and in the Ready state, proceed to Publishing & Policies to control how and when frames are sent to the Convai backend.

Last updated

Was this helpful?