> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/custom-frame-sources.md).

# 自定义帧源

实现 `IVisionFrameSource` 发布任何自定义视频管线——视频文件、自定义渲染纹理或屏幕捕获工具——而无需修改发布层。一旦你的组件进入场景， `ConvaiVisionPublisher` 就会自动发现并流式传输它。

### 接口契约

`IVisionFrameSource` 是通过以下方式进行视频流传输所需的最小契约 `ConvaiVisionPublisher` 以及通过以下方式进行实时画面显示所需的最小契约 `VisionDebugPreview`.

```csharp
public interface IVisionFrameSource
{
    bool IsCapturing { get; }
    long FrameCount { get; }
    (int Width, int Height) FrameDimensions { get; }
    float TargetFrameRate { get; }
    string SourceId { get; }
    RenderTexture CurrentRenderTexture { get; }
    bool IsFrameReady { get; }
    event Action FrameReady;
    void StartCapture();
    void StopCapture();
}
```

你的实现必须是一个 `MonoBehaviour`. `ConvaiVisionPublisher` 使用以下方式发现帧源 `GetComponent` 和 `GetComponentsInChildren`，它们只适用于 Unity 组件。

### Y 轴翻转要求

`CurrentRenderTexture` 必须为 **自上而下方向** （Y 轴相对于 Unity 默认的自下而上方向进行了翻转）。LiveKit 和标准视频格式都期望图像顶部的 Y=0。跳过这一步会导致接收端看到上下颠倒的画面。

使用 `Graphics.Blit` 调用，在将你的源纹理写入输出时应用翻转 `RenderTexture`:

```csharp
// sourceTexture：你的原始 Unity RenderTexture（自下而上）
// _outputRt：你通过 CurrentRenderTexture 暴露的 RenderTexture（自上而下）
Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f));
```

该 `scale.y = -1` 和 `offset.y = 1` 这两个参数组合起来会翻转垂直轴。将 `_outputRt` 到 `CurrentRenderTexture`.

{% hint style="warning" %}
v4.4.0 移除了 LiveKit 纹理回读中一个内部的双重垂直翻转，该翻转会在发布期间运行。在 v4.4.0 之前，自定义的 `IVisionFrameSource` 除了上面显示的翻转之外，还需要再额外做一次补偿性翻转，以抵消该内部翻转并得到正确的自上而下图像。现在请移除任何此类额外翻转。上面显示的单次翻转是 v4.4.0 之后唯一应用的方向修正。保留额外翻转会重新引入上下颠倒的画面。请参阅 [发布策略](/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/publishing-and-policies.md) 完整的迁移说明。
{% endhint %}

### 最小实现

下面的骨架实现了所有必需成员，并且正确处理了 `FrameReady` 。请在 `CaptureFrame` 方法中填入你的实际捕获逻辑。

```csharp
using System;
using Convai.Runtime.Vision.Sources;
using UnityEngine;

public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource
{
    [SerializeField] private int _width = 1280;
    [SerializeField] private int _height = 720;
    [SerializeField] private float _targetFps = 15f;
    [SerializeField] private string _sourceId = "custom";

    private RenderTexture _outputRt;
    private long _frameCount;
    private float _captureInterval;
    private float _nextCaptureTime;

    // IVisionFrameSource

    public bool IsCapturing { get; private set; }
    public long FrameCount => _frameCount;
    public (int Width, int Height) FrameDimensions => IsCapturing ? (_width, _height) : (0, 0);
    public float TargetFrameRate => _targetFps;
    public string SourceId => _sourceId;
    public RenderTexture CurrentRenderTexture => _outputRt;
    public bool IsFrameReady => _frameCount > 0;
    public event Action FrameReady;

    public void StartCapture()
    {
        if (IsCapturing) return;

        _outputRt = new RenderTexture(_width, _height, 24, RenderTextureFormat.ARGB32)
        {
            name = $"CustomFrameSource_{_sourceId}"
        };
        _outputRt.Create();

        _frameCount = 0;
        _captureInterval = _targetFps > 0f ? 1f / _targetFps : 1f / 15f;
        _nextCaptureTime = Time.realtimeSinceStartup;
        IsCapturing = true;
    }

    public void StopCapture()
    {
        if (!IsCapturing) return;

        IsCapturing = false;

        if (_outputRt != null)
        {
            _outputRt.Release();
            Destroy(_outputRt);
            _outputRt = null;
        }
    }

    private void Update()
    {
        if (!IsCapturing) return;

        float now = Time.realtimeSinceStartup;
        if (now < _nextCaptureTime) return;

        _nextCaptureTime = now + _captureInterval;
        CaptureFrame();
    }

    private void OnDestroy() => StopCapture();

    private void CaptureFrame()
    {
        // 替换为你的实际源纹理
        RenderTexture sourceTexture = GetYourSourceTexture();
        if (sourceTexture == null) return;

        // 向输出 RenderTexture 应用 Y 轴翻转
        Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f));

        _frameCount++;
        FrameReady?.Invoke();
    }

    private RenderTexture GetYourSourceTexture()
    {
        // 返回你自定义管线中的 RenderTexture
        return null;
    }
}
```

`FrameReady` 必须在 **Unity 主线程**. `ConvaiVisionPublisher` 和 `VisionDebugPreview` 都假定所有 `IVisionFrameSource` 回调都在主线程上执行。如果你的捕获逻辑运行在后台线程中，请使用一个在 `Update`中检查的标志将事件触发切回主线程，如上面的骨架所示。

### 公开生命周期状态

实现 `IVisionFrameSourceStatusProvider` 以及 `IVisionFrameSource` 一起用于公开更丰富的生命周期状态——权限流程、延迟初始化或结构化错误信息。这样发布者就能在不轮询的情况下响应就绪状态变化。

```csharp
public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource, IVisionFrameSourceStatusProvider
{
    // --- IVisionFrameSourceStatusProvider ---

    public VisionSourceState State { get; private set; } = VisionSourceState.Idle;
    public VisionSourceErrorKind ErrorKind { get; private set; } = VisionSourceErrorKind.None;
    public string StatusMessage { get; private set; } = string.Empty;
    public bool HasUsableFrame => FrameCount > 0;
    public event Action StatusChanged;

    private void SetState(VisionSourceState state, VisionSourceErrorKind error = VisionSourceErrorKind.None, string message = "")
    {
        State = state;
        ErrorKind = error;
        StatusMessage = message;
        StatusChanged?.Invoke();
    }

    public void StartCapture()
    {
        SetState(VisionSourceState.Starting);
        // ... 初始化捕获 ...
        SetState(VisionSourceState.Ready);
    }

    public void StopCapture()
    {
        // ... 释放资源 ...
        SetState(VisionSourceState.Stopped);
    }

    // ... 其余 IVisionFrameSource 实现
}
```

### 自定义 Inspector

`IVisionFrameSource` 是自定义视频管线唯一受支持的扩展点。SDK 自带的帧源 Inspector 只是内部实现细节，不是扩展面。

{% hint style="warning" %}
**在 SDK 4.5.0 中发生破坏性变更。** `ConvaiVisionBaseEditor` ——内置 `CameraVisionFrameSource`, `WebcamVisionFrameSource`，以及 `QuestVisionFrameSource` Inspector 背后的共享基类——从 `public` 到 `internal`发生了变化。基于它派生自定义 Inspector 的项目将不再能编译。请改为为你的 `CustomEditor` 组件编写一个独立的 `IVisionFrameSource` ，而不是继承 SDK 的编辑器基类。
{% endhint %}

### 自动发现

一旦你的组件进入场景， `ConvaiVisionPublisher` 会按以下顺序自动发现它：

1. 该 **来源** 在 Inspector 中的字段（显式赋值）。
2. `GetComponent<CameraVisionFrameSource>()` 在同一个 GameObject 上（内置优先级）。
3. `GetComponentsInChildren<MonoBehaviour>(true)` ——首先 `IVisionFrameSource` 在同一个 GameObject 或其子对象上找到的。

如果在步骤 3 中找到多个帧源，发布器会记录警告并选择第一个。请显式分配 **来源** 字段以避免歧义。

### 下一步

{% content-ref url="/pages/885a0e6676256e7738207107695532701ef9b1ec" %}
[视觉调试预览](/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/debug-preview.md)
{% endcontent-ref %}

{% content-ref url="/pages/9ee175b12da238182d707abd778abdc3a8706c80" %}
[视觉脚本 API](/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/scripting-api.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/custom-frame-sources.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.