> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/frame-sources.md).

# Vision frame sources

A frame source captures images from your scene and exposes them as a Y-flipped `RenderTexture` for `ConvaiVisionPublisher` to stream. The Convai SDK ships three built-in frame sources: `CameraVisionFrameSource` for Unity scene cameras, `WebcamVisionFrameSource` for physical devices, and `QuestVisionFrameSource` for Meta Quest passthrough.

| Frame source              | Best for                                                                             | Platforms                                |
| ------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------- |
| `CameraVisionFrameSource` | Streaming any Unity scene camera — main camera, security camera, overhead view       | PC, Mac, Android, iOS, console           |
| `WebcamVisionFrameSource` | Streaming a physical camera device attached to the player's machine or mobile device | PC, Mac, Android, iOS                    |
| `QuestVisionFrameSource`  | Streaming the real-world passthrough feed on a Meta Quest 3 or 3S headset            | Meta Quest 3 / 3S (requires Meta XR SDK) |

Add a frame source via **Add Component** and type the class name, or navigate the component menu under **Convai → Vision**.

### CameraVisionFrameSource

`CameraVisionFrameSource` captures a Unity `Camera`'s output into a `RenderTexture` and provides it to the publisher on every frame. It automatically selects the correct capture backend for the active render pipeline (Built-in or SRP/URP) when **Camera Capture Mode** is set to `Auto`.

**Component menu path:** `Convai/Vision/Camera Vision Frame Source`

<figure><img src="/files/Dw1gfZjfoDxOxOEQgVVo" alt="CameraVisionFrameSource Inspector"><figcaption><p>CameraVisionFrameSource Inspector.</p></figcaption></figure>

**Capture Settings**

| Field                   | Type                | Default    | Description                                                                                              |
| ----------------------- | ------------------- | ---------- | -------------------------------------------------------------------------------------------------------- |
| **Capture Preset**      | `CapturePreset`     | `Balanced` | Selects a preconfigured resolution and frame-rate combination. Set to `Custom` to enter values manually. |
| **Capture Width**       | `int`               | —          | Output width in pixels. Active only when preset is `Custom`.                                             |
| **Capture Height**      | `int`               | —          | Output height in pixels. Active only when preset is `Custom`.                                            |
| **Target Fps**          | `int`               | —          | Target capture frame rate. Active only when preset is `Custom`.                                          |
| **Camera Capture Mode** | `CameraCaptureMode` | `Auto`     | Selects the render-pipeline capture strategy. Leave at `Auto` unless the feed is black.                  |

**Camera**

| Field             | Type     | Default           | Description                                                                 |
| ----------------- | -------- | ----------------- | --------------------------------------------------------------------------- |
| **Target Camera** | `Camera` | *(auto-resolved)* | The camera to capture. If left blank, resolved to `Camera.main` at runtime. |

**Debug**

| Field                                    | Type     | Default    | Description                                                                                                                                                                              |
| ---------------------------------------- | -------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Source Id**                            | `string` | `"camera"` | Identifier used in domain events and multi-source scenarios.                                                                                                                             |
| **Enable Diagnostic Frame Health Probe** | `bool`   | `false`    | Performs a synchronous per-frame pixel readback to validate frame content. Use only when diagnosing black-frame issues; incurs a GPU readback cost every frame. Disable before shipping. |

{% hint style="warning" %}
If **Target Camera** is blank and no camera in the scene is tagged **MainCamera**, the frame source enters `Failed` state at runtime with `ErrorKind = InvalidConfiguration`. Always assign a camera explicitly or ensure one camera has the **MainCamera** tag.
{% endhint %}

**Capture preset values**

| Preset        | Width            | Height           | FPS              | Use case                                                               |
| ------------- | ---------------- | ---------------- | ---------------- | ---------------------------------------------------------------------- |
| `LowOverhead` | 640              | 480              | 10               | High-volume deployments, mobile, or bandwidth-constrained environments |
| `Balanced`    | 1280             | 720              | 15               | General purpose — default for most scenarios                           |
| `HighDetail`  | 1920             | 1080             | 30               | Scenes where fine visual detail is critical to AI comprehension        |
| `Custom`      | *(set manually)* | *(set manually)* | *(set manually)* | Full control over dimensions and frame rate                            |

<figure><img src="/files/rbQR5xFnxKImh6WkJZsN" alt="Capture preset options in the Inspector dropdown"><figcaption><p>Capture preset options in the Inspector dropdown.</p></figcaption></figure>

**Camera capture mode values**

| Mode                          | When to use                                                                                                                                                                                                                                            |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `Auto`                        | Default. On Built-in Render Pipeline, uses render hooks (`Camera.onPreRender` / `Camera.onPostRender`). On SRP/URP, uses the explicit render path (`TargetCamera.Render()` in `LateUpdate`). Select this first and override only if the feed is black. |
| `BuiltInHooks`                | Forces Built-in Render Pipeline capture hooks. Only compatible with the Built-in RP — selecting this on SRP/URP produces a black feed.                                                                                                                 |
| `ExplicitRenderCompatibility` | Forces an explicit per-frame camera render. Use when `Auto` produces a black feed on SRP/URP, or with highly customised render setups. Adds a bounded extra camera render at the configured capture FPS.                                               |
| `SrpNative`                   | **Not available in this SDK build.** See note below.                                                                                                                                                                                                   |

{% hint style="danger" %}
**`SrpNative` is not functional in the current SDK build.** Selecting it causes `CameraVisionFrameSource` to enter `Failed` state immediately with `ErrorKind = UnsupportedPlatform`. If `Auto` is producing a black feed on your SRP/URP project, use `ExplicitRenderCompatibility` instead.
{% endhint %}

### WebcamVisionFrameSource

`WebcamVisionFrameSource` captures a physical camera device using Unity's `WebCamTexture` API and converts the output to a `RenderTexture`. It handles device selection, permission requests (on Android and iOS), automatic rotation correction, and resolution clamping.

**Component menu path:** `Convai/Vision/Webcam Vision Frame Source`

<figure><img src="/files/7qpBXvLSmO2YxPj1dDHj" alt="WebcamVisionFrameSource Inspector"><figcaption><p>WebcamVisionFrameSource Inspector.</p></figcaption></figure>

**Webcam Settings**

| Field                  | Type     | Default | Description                                                                                                         |
| ---------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| **Webcam Device Name** | `string` | `""`    | Name of the webcam device to open. An empty string selects the first available device.                              |
| **Requested Width**    | `int`    | `640`   | Requested capture width sent to the driver. Actual resolution may differ.                                           |
| **Requested Height**   | `int`    | `480`   | Requested capture height sent to the driver.                                                                        |
| **Requested Fps**      | `int`    | `15`    | Requested frame rate sent to the driver.                                                                            |
| **Max Output Width**   | `int`    | `1280`  | Maximum width of the output `RenderTexture`. Frames wider than this are scaled down. Set to `0` to disable scaling. |
| **Max Output Height**  | `int`    | `720`   | Maximum height of the output `RenderTexture`.                                                                       |

**Source Identification**

| Field         | Type     | Default    | Description                                                  |
| ------------- | -------- | ---------- | ------------------------------------------------------------ |
| **Source Id** | `string` | `"webcam"` | Identifier used in domain events and multi-source scenarios. |

**Listing available devices**

To enumerate connected webcam devices at runtime:

```csharp
string[] deviceNames = WebcamVisionFrameSource.GetAvailableDeviceNames();
foreach (string name in deviceNames)
    Debug.Log(name);
```

**Switching devices at runtime**

To switch to a different webcam without stopping the session:

```csharp
WebcamVisionFrameSource webcam = GetComponent<WebcamVisionFrameSource>();
await webcam.SwitchWebcamAsync("Front Camera");
```

**Permission flow (Android / iOS)**

On Android and iOS the component requests camera permission asynchronously when `StartCapture()` is called. The `State` property transitions through:

`Idle → AwaitingPermission → Starting → Ready`

If the user denies permission, `State` becomes `Failed` and `ErrorKind` is set to `PermissionDenied`. Monitor this via `IVisionFrameSourceStatusProvider.StatusChanged` — see [Vision scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/scripting-api.md) for the state monitor pattern.

{% hint style="info" %}
On Android and iOS the system camera permission dialog appears the first time `StartCapture()` runs. Ensure your app's `AndroidManifest.xml` declares `android.permission.CAMERA` and your iOS `Info.plist` includes `NSCameraUsageDescription` before submitting to app stores.
{% endhint %}

### QuestVisionFrameSource

`QuestVisionFrameSource` streams the real-world passthrough feed from a Meta Quest 3 or 3S headset, giving Convai characters a live view of the physical environment. The component binds to Meta's `PassthroughCameraAccess` API via reflection so the SDK does not take a hard compile-time dependency on the Meta XR package.

**Component menu path:** `Convai/Vision/Quest Vision Frame Source`

<figure><img src="/files/Jum2faxSrZtIUaBltbWQ" alt="QuestVisionFrameSource Inspector"><figcaption><p>QuestVisionFrameSource Inspector.</p></figcaption></figure>

{% hint style="warning" %}
`QuestVisionFrameSource` requires **Meta Quest 3 or 3S** running Horizon OS with the Passthrough Camera API. Quest 2 and Quest Pro do not support `PassthroughCameraAccess`. In the Editor or on other platforms, the component enters `Failed` state with `ErrorKind = UnsupportedPlatform` and produces no frames.
{% endhint %}

{% hint style="danger" %}
**AndroidManifest.xml permissions required.** Your manifest must declare both `horizonos.permission.HEADSET_CAMERA` and `android.permission.CAMERA`. Without these declarations, passthrough capture fails silently on device and the frame source enters `Failed` state. The device does not show a permission dialog — it simply denies access.
{% endhint %}

**Quest Camera Access**

| Field                         | Type            | Default             | Description                                                                                |
| ----------------------------- | --------------- | ------------------- | ------------------------------------------------------------------------------------------ |
| **Passthrough Camera Access** | `MonoBehaviour` | *(auto-discovered)* | Reference to a `PassthroughCameraAccess` component in the scene. Leave blank to auto-find. |

**Output Settings**

| Field                 | Type     | Default               | Description                                                                                                                                         |
| --------------------- | -------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Source Id**         | `string` | `"quest-passthrough"` | Identifier used in domain events.                                                                                                                   |
| **Max Output Width**  | `int`    | `1280`                | Maximum width of the output `RenderTexture`.                                                                                                        |
| **Max Output Height** | `int`    | `720`                 | Maximum height of the output `RenderTexture`.                                                                                                       |
| **Target Frame Rate** | `int`    | `15`                  | Target frames per second for capture.                                                                                                               |
| **Flip Y**            | `bool`   | `true`                | Flips the passthrough texture vertically. Required for correct video orientation — disable only if the Meta SDK changes its coordinate conventions. |

The binding to `PassthroughCameraAccess` is established via reflection at runtime. If the Meta XR package is updated and `PassthroughCameraAccess` changes its API, the component logs an error and enters `Failed` state. Check for SDK updates in that case.

### Source state reference

All three frame sources implement `IVisionFrameSourceStatusProvider`, which exposes a `State` property and a `StatusChanged` event. Use these to monitor capture health from scripts.

**VisionSourceState**

| State                | Meaning                                                                                           |
| -------------------- | ------------------------------------------------------------------------------------------------- |
| `Idle`               | Capture has not been started.                                                                     |
| `AwaitingPermission` | Waiting for the user to grant camera permission (Android / iOS).                                  |
| `Starting`           | Capture is initializing — device is opening, `RenderTexture`s are being created.                  |
| `Ready`              | Capture is running and frames are being produced.                                                 |
| `Degraded`           | Capture is running but frame health checks are detecting issues (e.g., consecutive blank frames). |
| `Stopped`            | Capture was stopped normally.                                                                     |
| `Failed`             | Capture failed and cannot continue. Check `ErrorKind` and `StatusMessage` for details.            |

**VisionSourceErrorKind**

| Error kind             | Cause                                                                                                                      |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `None`                 | No error.                                                                                                                  |
| `Timeout`              | The source did not produce a usable frame within the expected time window.                                                 |
| `PermissionDenied`     | Camera permission was denied by the user or the OS.                                                                        |
| `UnsupportedPlatform`  | The source is not supported on this platform (e.g., `QuestVisionFrameSource` on PC, or `SrpNative` capture mode selected). |
| `DeviceUnavailable`    | The requested camera device could not be opened.                                                                           |
| `InvalidConfiguration` | A field value is out of range or inconsistent (check `StatusMessage`).                                                     |
| `Unknown`              | An unexpected error occurred.                                                                                              |

For scripting against these states and responding to `StatusChanged` events, see [Vision scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/scripting-api.md).

### Platform compatibility matrix

| Feature                   | PC / Mac        | Android / iOS       | WebGL                      | Meta Quest 3 / 3S        |
| ------------------------- | --------------- | ------------------- | -------------------------- | ------------------------ |
| `CameraVisionFrameSource` | ✅               | ✅                   | ❌ (no frame source needed) | ✅                        |
| `WebcamVisionFrameSource` | ✅               | ✅ (permission flow) | ❌                          | ❌                        |
| `QuestVisionFrameSource`  | ❌               | ❌                   | ❌                          | ✅ (Meta XR SDK required) |
| WebGL canvas capture      | ❌               | ❌                   | ✅ (automatic)              | ❌                        |
| `VisionDebugPreview`      | ✅ (Editor only) | ✅ (Editor only)     | ⚠️ blank                   | ✅ (Editor only)          |
| Max publish FPS           | 30              | 30                  | 15                         | 30                       |

On WebGL no frame source component is required or used. `ConvaiVisionPublisher` captures the visible browser canvas automatically via `canvas.captureStream()`. See [Publish policies](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/publishing-and-policies.md) for WebGL-specific behavior and the HTTPS requirement.

### Next steps

{% content-ref url="/pages/8e81321c92ed2b28e658ae5686d3cac89efa9213" %}
[Publish policies](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/publishing-and-policies.md)
{% endcontent-ref %}

{% content-ref url="/pages/a605ab5aa55eec202ad6748667a5629e34141045" %}
[Custom frame sources](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/custom-frame-sources.md)
{% endcontent-ref %}

{% content-ref url="/pages/35fce4a7255d7ee06c08b0cc4b37b598f89f6068" %}
[Vision scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/scripting-api.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/frame-sources.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.