Frame Sources

Select and configure the right frame source for your platform. Covers all Inspector fields, capture presets, platform requirements, and source state diagnostics.

Choosing and Configuring a Vision Frame Source

A frame source is the component responsible for capturing images and making them available to the publisher as a Y-flipped RenderTexture. The Convai SDK ships three built-in frame sources — one for Unity scene cameras, one for physical webcams, and one for the Meta Quest passthrough camera — each optimised for its target platform. This page covers how to add, configure, and troubleshoot all three.

Selecting a Frame Source

Frame source	Best for	Platforms
CameraVisionFrameSource	Streaming any Unity scene camera — main camera, security camera, overhead view	PC, Mac, Android, iOS, console
WebcamVisionFrameSource	Streaming a physical camera device attached to the player's machine or mobile device	PC, Mac, Android, iOS
QuestVisionFrameSource	Streaming the real-world passthrough feed on a Meta Quest headset	Meta Quest (requires Meta XR SDK)

Add a frame source via Add Component and type the class name, or navigate the component menu under Convai → Vision.

---

CameraVisionFrameSource

CameraVisionFrameSource captures a Unity Camera's output into a RenderTexture and provides it to the publisher on every frame. It automatically selects the correct capture backend for the active render pipeline (built-in or SRP/URP) when Camera Capture Mode is set to Auto.

Component menu path: Convai/Vision/Camera Vision Frame Source

Inspector Reference

Capture Settings

Field	Type	Default	Description
Capture Preset	CapturePreset	Balanced	Selects a preconfigured resolution and frame-rate combination. Set to Custom to enter values manually.
Capture Width	int	—	Output width in pixels. Active only when preset is Custom.
Capture Height	int	—	Output height in pixels. Active only when preset is Custom.
Target Fps	int	—	Target capture frame rate. Active only when preset is Custom.
Camera Capture Mode	CameraCaptureMode	Auto	Selects the render-pipeline capture strategy. Leave at Auto unless directed otherwise.

Camera

Field	Type	Default	Description
Target Camera	Camera	(auto-resolved)	The camera to capture. If left blank, resolved to Camera.main at runtime.

Debug

Field	Type	Default	Description
Source Id	string	"camera"	An identifier used in domain events and multi-source scenarios.
Enable Diagnostic Frame Health Probe	bool	false	Performs a synchronous per-frame pixel readback to validate frame content. Use only when diagnosing black-frame issues; incurs GPU readback cost every frame.

Capture Preset Values

Preset	Width	Height	FPS	Use case
LowOverhead	640	480	10	High-volume deployments, mobile, or bandwidth-constrained environments
Balanced	1280	720	15	General purpose — default for most scenarios
HighDetail	1920	1080	30	Scenes where fine visual detail is important to AI comprehension
Custom	(set manually)	(set manually)	(set manually)	Full control over dimensions and frame rate

Camera Capture Mode Values

Mode	When to use
Auto	Default. Detects the active render pipeline and selects the appropriate backend automatically.
BuiltInHooks	Forces use of Camera.onPreRender / Camera.onPostRender callbacks. Only compatible with the Built-in Render Pipeline.
SrpNative	Forces the SRP/URP explicit render path (TargetCamera.Render() in LateUpdate). Use when Auto fails to detect SRP correctly.
ExplicitRenderCompatibility	A compatibility fallback that explicitly renders the camera. Useful for highly customised render setups.

Leave Camera Capture Mode at Auto unless you have a specific reason to override it. Selecting the wrong backend for your render pipeline will result in a black or blank video feed.

Enable Diagnostic Frame Health Probe performs a synchronous GPU-to-CPU pixel readback every frame. Enable it only during debugging and disable it before shipping. In production, the SDK performs lighter-weight frame health checks internally.

---

WebcamVisionFrameSource

WebcamVisionFrameSource captures a physical camera device using Unity's WebCamTexture API and converts the output to a RenderTexture. It handles device selection, permission requests (on Android and iOS), automatic rotation correction, and resolution clamping.

Component menu path: Convai/Vision/Webcam Vision Frame Source

Inspector Reference

Webcam Settings

Field	Type	Default	Description
Webcam Device Name	string	""	Name of the webcam device to open. An empty string selects the first available device.
Requested Width	int	640	Requested capture width sent to the driver. The actual resolution may differ.
Requested Height	int	480	Requested capture height sent to the driver.
Requested Fps	int	15	Requested frame rate sent to the driver.
Max Output Width	int	1280	Maximum width of the output RenderTexture. Frames wider than this are scaled down. Set to 0 to disable scaling.
Max Output Height	int	720	Maximum height of the output RenderTexture.

Source Identification

Field	Type	Default	Description
Source Id	string	"webcam"	Identifier used in domain events and multi-source scenarios.

Listing Available Devices

To enumerate connected webcam devices at runtime:

string[] deviceNames = WebcamVisionFrameSource.GetAvailableDeviceNames();
foreach (string name in deviceNames)
    Debug.Log(name);

Switching Devices at Runtime

To switch to a different webcam without stopping the session:

// Requires a reference to the WebcamVisionFrameSource component
WebcamVisionFrameSource webcam = GetComponent<WebcamVisionFrameSource>();
IConvaiOperation<Unit> op = webcam.SwitchWebcamAsync("Front Camera");

Permission Flow (Android / iOS)

On Android and iOS the component requests camera permission asynchronously when StartCapture() is called. The State property transitions through:

Idle → AwaitingPermission → Starting → Ready

If the user denies permission, State becomes Failed and ErrorKind is set to PermissionDenied. You can monitor this via IVisionFrameSourceStatusProvider.StatusChanged — see Advanced Topics for details.

On Android and iOS the system camera permission dialog appears the first time StartCapture() runs. Ensure your app's manifest or Info.plist declares the camera usage description before shipping.

---

QuestVisionFrameSource

QuestVisionFrameSource streams the real-world passthrough feed from a Meta Quest headset, giving Convai characters a live view of the physical environment the user is standing in. The component binds to Meta's PassthroughCameraAccess API via reflection so that the SDK does not take a hard compile-time dependency on the Meta XR package.

Component menu path: Convai/Vision/Quest Vision Frame Source

Inspector Reference

Quest Camera Access

Field	Type	Default	Description
Passthrough Camera Access	MonoBehaviour	(auto-discovered)	Reference to a PassthroughCameraAccess component in the scene. Leave blank to auto-find.

Output Settings

Field	Type	Default	Description
Source Id	string	"quest-passthrough"	Identifier used in domain events.
Max Output Width	int	1280	Maximum width of the output RenderTexture.
Max Output Height	int	720	Maximum height of the output RenderTexture.
Target Frame Rate	int	15	Target frames per second for capture.
Flip Y	bool	true	Flips the passthrough texture vertically. Required for correct video orientation; disable only if the Meta SDK changes its coordinate conventions.

QuestVisionFrameSource requires the Meta XR SDK to be installed. On non-Quest platforms the component enters the Failed state with ErrorKind set to UnsupportedPlatform and produces no frames.

The binding to PassthroughCameraAccess is established via reflection at runtime. If the Meta XR package is updated and PassthroughCameraAccess changes its API, the component will log an error and enter Failed state. Check for SDK updates in that case.

---

Source State Reference

All three frame sources implement IVisionFrameSourceStatusProvider, which exposes a State property and a StatusChanged event.

VisionSourceState

State	Meaning
Idle	Capture has not been started.
AwaitingPermission	Waiting for the user to grant camera permission (Android / iOS).
Starting	Capture is initialising — device is opening, RenderTextures are being created.
Ready	Capture is running and frames are being produced.
Degraded	Capture is running but frame health checks are detecting issues (e.g., consecutive blank frames).
Stopped	Capture was stopped normally.
Failed	Capture failed and cannot continue. Check ErrorKind and StatusMessage for details.

VisionSourceErrorKind

Error kind	Cause
None	No error.
Timeout	The source did not produce a usable frame within the expected time window.
PermissionDenied	Camera permission was denied by the user or the OS.
UnsupportedPlatform	The source is not supported on this platform (e.g., QuestVisionFrameSource on PC).
DeviceUnavailable	The requested camera device could not be opened.
InvalidConfiguration	A field value is out of range or inconsistent (check StatusMessage).
Unknown	An unexpected error occurred.

For scripting against these states and responding to StatusChanged events, see Advanced Topics.

Conclusion

Each frame source targets a specific platform and capture scenario — use CameraVisionFrameSource for Unity scene cameras, WebcamVisionFrameSource for physical devices on desktop and mobile, and QuestVisionFrameSource for Meta Quest passthrough. With your frame source configured and in the Ready state, proceed to Publishing & Policies to control how and when frames are sent to the Convai backend.