> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/custom-frame-sources.md). # Custom frame sources Implement `IVisionFrameSource` to publish any custom video pipeline — a video file, a custom render texture, or a screen capture utility — without modifying the publishing layer. Once your component is on the scene, `ConvaiVisionPublisher` discovers and streams it automatically. ### Interface contract `IVisionFrameSource` is the minimal contract required for video streaming via `ConvaiVisionPublisher` and live feed display via `VisionDebugPreview`. ```csharp public interface IVisionFrameSource { bool IsCapturing { get; } long FrameCount { get; } (int Width, int Height) FrameDimensions { get; } float TargetFrameRate { get; } string SourceId { get; } RenderTexture CurrentRenderTexture { get; } bool IsFrameReady { get; } event Action FrameReady; void StartCapture(); void StopCapture(); } ``` Your implementation must be a `MonoBehaviour`. `ConvaiVisionPublisher` discovers frame sources using `GetComponent` and `GetComponentsInChildren`, which only work on Unity components. ### Y-flip requirement `CurrentRenderTexture` must be in **top-down orientation** (Y-axis flipped from Unity's default bottom-up). LiveKit and standard video formats expect Y=0 at the top of the image. Skipping this step produces an upside-down feed at the receiving end. Apply the flip with a `Graphics.Blit` call when writing from your source texture into the output `RenderTexture`: ```csharp // sourceTexture: your raw Unity RenderTexture (bottom-up) // _outputRt: the RenderTexture you expose via CurrentRenderTexture (top-down) Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f)); ``` The `scale.y = -1` and `offset.y = 1` arguments together flip the vertical axis. Assign `_outputRt` to `CurrentRenderTexture`. ### Minimal implementation The following skeleton implements every required member and handles `FrameReady` correctly. Fill in the `CaptureFrame` method with your actual capture logic. ```csharp using System; using Convai.Runtime.Vision.Sources; using UnityEngine; public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource { [SerializeField] private int _width = 1280; [SerializeField] private int _height = 720; [SerializeField] private float _targetFps = 15f; [SerializeField] private string _sourceId = "custom"; private RenderTexture _outputRt; private long _frameCount; private float _captureInterval; private float _nextCaptureTime; // IVisionFrameSource public bool IsCapturing { get; private set; } public long FrameCount => _frameCount; public (int Width, int Height) FrameDimensions => IsCapturing ? (_width, _height) : (0, 0); public float TargetFrameRate => _targetFps; public string SourceId => _sourceId; public RenderTexture CurrentRenderTexture => _outputRt; public bool IsFrameReady => _frameCount > 0; public event Action FrameReady; public void StartCapture() { if (IsCapturing) return; _outputRt = new RenderTexture(_width, _height, 24, RenderTextureFormat.ARGB32) { name = $"CustomFrameSource_{_sourceId}" }; _outputRt.Create(); _frameCount = 0; _captureInterval = _targetFps > 0f ? 1f / _targetFps : 1f / 15f; _nextCaptureTime = Time.realtimeSinceStartup; IsCapturing = true; } public void StopCapture() { if (!IsCapturing) return; IsCapturing = false; if (_outputRt != null) { _outputRt.Release(); Destroy(_outputRt); _outputRt = null; } } private void Update() { if (!IsCapturing) return; float now = Time.realtimeSinceStartup; if (now < _nextCaptureTime) return; _nextCaptureTime = now + _captureInterval; CaptureFrame(); } private void OnDestroy() => StopCapture(); private void CaptureFrame() { // Replace with your actual source texture RenderTexture sourceTexture = GetYourSourceTexture(); if (sourceTexture == null) return; // Y-flip into the output RenderTexture Graphics.Blit(sourceTexture, _outputRt, new Vector2(1f, -1f), new Vector2(0f, 1f)); _frameCount++; FrameReady?.Invoke(); } private RenderTexture GetYourSourceTexture() { // Return the RenderTexture from your custom pipeline return null; } } ``` `FrameReady` must be raised on the **Unity main thread**. `ConvaiVisionPublisher` and `VisionDebugPreview` both assume all `IVisionFrameSource` callbacks execute on the main thread. If your capture logic runs on a background thread, marshal the event raise back using a flag checked in `Update`, as shown in the skeleton above. ### Expose lifecycle state Implement `IVisionFrameSourceStatusProvider` alongside `IVisionFrameSource` to expose richer lifecycle state — permission flow, delayed initialization, or structured error information. The publisher can then react to readiness changes without polling. ```csharp public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource, IVisionFrameSourceStatusProvider { // --- IVisionFrameSourceStatusProvider --- public VisionSourceState State { get; private set; } = VisionSourceState.Idle; public VisionSourceErrorKind ErrorKind { get; private set; } = VisionSourceErrorKind.None; public string StatusMessage { get; private set; } = string.Empty; public bool HasUsableFrame => FrameCount > 0; public event Action StatusChanged; private void SetState(VisionSourceState state, VisionSourceErrorKind error = VisionSourceErrorKind.None, string message = "") { State = state; ErrorKind = error; StatusMessage = message; StatusChanged?.Invoke(); } public void StartCapture() { SetState(VisionSourceState.Starting); // ... initialise capture ... SetState(VisionSourceState.Ready); } public void StopCapture() { // ... release resources ... SetState(VisionSourceState.Stopped); } // ... rest of IVisionFrameSource implementation } ``` ### Auto-discovery Once your component is on the scene, `ConvaiVisionPublisher` discovers it automatically in this order: 1. The **Source** field in the Inspector (explicit assignment). 2. `GetComponent()` on the same GameObject (built-in preference). 3. `GetComponentsInChildren(true)` — first `IVisionFrameSource` found on the same GameObject or children. If more than one frame source is found under step 3, the publisher logs a warning and selects the first. Assign the **Source** field explicitly to avoid ambiguity. ### Next steps {% content-ref url="/pages/eP36iBni1AamThCZdCLM" %} [Vision debug preview](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/debug-preview.md) {% endcontent-ref %} {% content-ref url="/pages/35fce4a7255d7ee06c08b0cc4b37b598f89f6068" %} [Vision scripting API](/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/scripting-api.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/vision/custom-frame-sources.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.