> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/custom-frame-sources.md).

# 自定义帧来源

实现 IVisionFrameSource，将自定义视频管线发布到 Convai，包括 Y 翻转要求、生命周期状态模式和自动发现规则。

实现 `IVisionFrameSource` 即可发布任何自定义视频管线（例如视频文件、自定义渲染纹理或屏幕捕获工具），而无需修改发布层。将组件添加到场景后， `ConvaiVisionPublisher` 会自动发现并推流。

### 接口契约

`IVisionFrameSource` 是通过 `ConvaiVisionPublisher` 进行视频推流以及通过 `VisionDebugPreview`.

```csharp
public interface IVisionFrameSource
{
    bool IsCapturing { get; }
    long FrameCount { get; }
    (int Width, int Height) FrameDimensions { get; }
    float TargetFrameRate { get; }
    string SourceId { get; }
    RenderTexture CurrentRenderTexture { get; }
    bool IsFrameReady { get; }
    event Action FrameReady;
    void StartCapture();
    void StopCapture();
}
```

你的实现必须是一个 `MonoBehaviour`. `ConvaiVisionPublisher` 使用 `GetComponent` 和 `GetComponentsInChildren`发现帧源，而这两个方法只适用于 Unity 组件。

### Y 轴翻转要求

`CurrentRenderTexture` 必须采用 **自上而下的方向** （相对于 Unity 默认的自下而上方向翻转 Y 轴）。LiveKit 和标准视频格式要求 Y=0 位于图像顶部。跳过此步骤会导致接收端的视频画面上下颠倒。

从源纹理写入输出 `Graphics.Blit` 时调用 `RenderTexture`:

```csharp
// sourceTexture：原始 Unity RenderTexture（自下而上）
// _outputRt：通过 CurrentRenderTexture 暴露的 RenderTexture（自上而下）
Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f));
```

参数 `scale.y = -1` 和 `offset.y = 1` 共同实现垂直轴翻转。将 `_outputRt` 赋给 `CurrentRenderTexture`。这是自定义 `IVisionFrameSource` 唯一需要进行的方向校正——发布期间运行的 LiveKit 纹理读取不会再进行任何翻转。

### 最简实现

以下代码框架实现了所有必需成员，并能正确处理 `FrameReady` 。请在 `CaptureFrame` 方法中填入实际的采集逻辑。

```csharp
using System;
using Convai.Runtime.Vision.Sources;
using UnityEngine;

public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource
{
    [SerializeField] private int _width = 1280;
    [SerializeField] private int _height = 720;
    [SerializeField] private float _targetFps = 15f;
    [SerializeField] private string _sourceId = "custom";

    private RenderTexture _outputRt;
    private long _frameCount;
    private float _captureInterval;
    private float _nextCaptureTime;

    // IVisionFrameSource

    public bool IsCapturing { get; private set; }
    public long FrameCount => _frameCount;
    public (int Width, int Height) FrameDimensions => IsCapturing ? (_width, _height) : (0, 0);
    public float TargetFrameRate => _targetFps;
    public string SourceId => _sourceId;
    public RenderTexture CurrentRenderTexture => _outputRt;
    public bool IsFrameReady => _frameCount > 0;
    public event Action FrameReady;

    public void StartCapture()
    {
        if (IsCapturing) return;

        _outputRt = new RenderTexture(_width, _height, 24, RenderTextureFormat.ARGB32)
        {
            name = $"CustomFrameSource_{_sourceId}"
        };
        _outputRt.Create();

        _frameCount = 0;
        _captureInterval = _targetFps > 0f ? 1f / _targetFps : 1f / 15f;
        _nextCaptureTime = Time.realtimeSinceStartup;
        IsCapturing = true;
    }

    public void StopCapture()
    {
        if (!IsCapturing) return;

        IsCapturing = false;

        if (_outputRt != null)
        {
            _outputRt.Release();
            Destroy(_outputRt);
            _outputRt = null;
        }
    }

    private void Update()
    {
        if (!IsCapturing) return;

        float now = Time.realtimeSinceStartup;
        if (now < _nextCaptureTime) return;

        _nextCaptureTime = now + _captureInterval;
        CaptureFrame();
    }

    private void OnDestroy() => StopCapture();

    private void CaptureFrame()
    {
        // 替换为你的实际源纹理
        RenderTexture sourceTexture = GetYourSourceTexture();
        if (sourceTexture == null) return;

        // 沿 Y 轴翻转到输出 RenderTexture
        Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f));

        _frameCount++;
        FrameReady?.Invoke();
    }

    private RenderTexture GetYourSourceTexture()
    {
        // 返回自定义管线中的 RenderTexture
        return null;
    }
}
```

`FrameReady` 必须在 **Unity 主线程上**. `ConvaiVisionPublisher` 和 `VisionDebugPreview` 都假定所有 `IVisionFrameSource` 回调都在主线程上执行。如果你的采集逻辑运行在后台线程，请使用一个在 `Update`中检查的标志，将事件触发操作转回主线程，如上面的框架代码所示。

### 公开生命周期状态

同时实现 `IVisionFrameSourceStatusProvider` 和 `IVisionFrameSource` ，以公开更丰富的生命周期状态，例如权限流程、延迟初始化或结构化错误信息。这样，发布器便可在无需轮询的情况下响应就绪状态的变化。

```csharp
public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource, IVisionFrameSourceStatusProvider
{
    // --- IVisionFrameSourceStatusProvider ---

    public VisionSourceState State { get; private set; } = VisionSourceState.Idle;
    public VisionSourceErrorKind ErrorKind { get; private set; } = VisionSourceErrorKind.None;
    public string StatusMessage { get; private set; } = string.Empty;
    public bool HasUsableFrame => FrameCount > 0;
    public event Action StatusChanged;

    private void SetState(VisionSourceState state, VisionSourceErrorKind error = VisionSourceErrorKind.None, string message = "")
    {
        State = state;
        ErrorKind = error;
        StatusMessage = message;
        StatusChanged?.Invoke();
    }

    public void StartCapture()
    {
        SetState(VisionSourceState.Starting);
        // ... initialise capture ...
        SetState(VisionSourceState.Ready);
    }

    public void StopCapture()
    {
        // ... release resources ...
        SetState(VisionSourceState.Stopped);
    }

    // ... rest of IVisionFrameSource implementation
}
```

### 自定义检查器

`IVisionFrameSource` 是自定义视频管线唯一受支持的扩展点。SDK 自带的帧源检查器属于内部实现细节，并非扩展接口。 `ConvaiVisionBaseEditor` 是内置 `CameraVisionFrameSource`, `和`WebcamVisionFrameSource `以及` QuestVisionFrameSource `检查器背后的共享基类，它是`internal `，因此请为你的组件编写独立的` CustomEditor `IVisionFrameSource` ，而不要继承 SDK 的编辑器基类。

### 自动发现

组件位于场景中后， `ConvaiVisionPublisher` 会按以下顺序自动发现它：

1. Inspector 中的 **Source** 字段（显式指定）。
2. `同一 GameObject 上的 GetComponent<CameraVisionFrameSource>()` （优先使用内置组件）。
3. `GetComponentsInChildren<MonoBehaviour>(true)` — 第一个 `IVisionFrameSource` （在同一 GameObject 或其子对象上找到）。

如果在步骤 3 中找到多个帧源，发布器会记录警告并选择第一个。请显式指定 **Source** 字段，以避免出现歧义。

### 后续步骤

{% content-ref url="/pages/885a0e6676256e7738207107695532701ef9b1ec" %}
[视觉调试预览](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/debug-preview.md)
{% endcontent-ref %}

{% content-ref url="/pages/9ee175b12da238182d707abd778abdc3a8706c80" %}
[Vision 脚本 API](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/scripting-api.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/convai-unity-sdk/features/vision/custom-frame-sources.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
