Advanced Topics

Scripting API reference, custom frame source implementation, domain events, WebGL platform behaviour, and cross-platform compatibility.

Advanced Vision Configuration and Custom Integration

This page covers the full scripting API of ConvaiVisionPublisher, the contract for implementing a custom frame source, the domain events emitted throughout the Vision lifecycle, platform-specific behaviour, and a cross-platform compatibility matrix. It is intended for developers who need programmatic control beyond what the Inspector provides.

ConvaiVisionPublisher Scripting API

Member

Signature

Description

IsPublishing

bool IsPublishing { get; }

true when a video track is actively published to the room.

FrameSource

IVisionFrameSource FrameSource { get; }

The currently active frame source. null on WebGL.

PublishPolicy

VisionPublishPolicy PublishPolicy { get; }

The current publish policy.

VideoTrackName

string VideoTrackName { get; }

The WebRTC video track name in use.

SetPublishPolicy

void SetPublishPolicy(VisionPublishPolicy policy)

Changes the publish policy. If publishing is active, the track is restarted with the new profile immediately.

EnablePublishing

void EnablePublishing(bool enabled)

Starts or stops publishing without changing the policy. Required when policy is Manual; available on all policies.

Example: conditional policy selection

using Convai.Modules.Vision;
using Convai.Runtime.Vision.Publishing;

ConvaiVisionPublisher publisher = GetComponent<ConvaiVisionPublisher>();

bool isHighBandwidthNetwork = /* your network quality check */;
VisionPublishPolicy chosen = isHighBandwidthNetwork
    ? VisionPublishPolicy.HighResponsiveness
    : VisionPublishPolicy.LowOverhead;

publisher.SetPublishPolicy(chosen);

Implementing a Custom Frame Source

If none of the built-in sources fits your use case — for example, you want to publish a render texture produced by a custom post-processing pipeline, or stream from a third-party SDK — implement IVisionFrameSource.

Interface Contract

namespace Convai.Runtime.Vision.Sources
{
    public interface IVisionFrameSource
    {
        bool IsCapturing { get; }
        long FrameCount { get; }
        (int Width, int Height) FrameDimensions { get; }
        float TargetFrameRate { get; }
        string SourceId { get; }

        // Must be a Y-flipped (top-down) RenderTexture
        RenderTexture CurrentRenderTexture { get; }

        bool IsFrameReady { get; }
        event Action FrameReady;

        void StartCapture();
        void StopCapture();
    }
}

Y-Flip Requirement

CurrentRenderTexture must be oriented top-down (Y-flipped relative to Unity's default bottom-up convention). Failing to flip the texture produces an upside-down video stream on the backend. Use Graphics.Blit with a flipped scale vector:

// Flip source into outputTexture (top-down orientation)
Graphics.Blit(sourceTexture, outputTexture, new Vector2(1f, -1f), new Vector2(0f, 1f));

Minimal Custom Implementation

using System;
using UnityEngine;
using Convai.Runtime.Vision.Sources;

public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource
{
    [SerializeField] RenderTexture _sourceTexture;

    RenderTexture _outputTexture;
    bool _isCapturing;
    long _frameCount;

    public bool IsCapturing => _isCapturing;
    public long FrameCount => _frameCount;
    public (int Width, int Height) FrameDimensions =>
        _outputTexture != null ? (_outputTexture.width, _outputTexture.height) : (0, 0);
    public float TargetFrameRate => 15f;
    public string SourceId => "custom";
    public RenderTexture CurrentRenderTexture => _outputTexture;
    public bool IsFrameReady => _outputTexture != null && _frameCount > 0;

    public event Action FrameReady;

    public void StartCapture()
    {
        _outputTexture = new RenderTexture(
            _sourceTexture.width, _sourceTexture.height, 24, RenderTextureFormat.ARGB32);
        _isCapturing = true;
    }

    public void StopCapture()
    {
        _isCapturing = false;
        if (_outputTexture != null) { _outputTexture.Release(); _outputTexture = null; }
    }

    void Update()
    {
        if (!_isCapturing || _sourceTexture == null || _outputTexture == null) return;

        // Y-flip the source texture into the output
        Graphics.Blit(_sourceTexture, _outputTexture, new Vector2(1f, -1f), new Vector2(0f, 1f));
        _frameCount++;
        FrameReady?.Invoke();
    }
}

Optional: IVisionFrameSourceStatusProvider

To expose detailed state information (used by VisionDebugPreview and the coordinator's health checks), also implement IVisionFrameSourceStatusProvider:

using Convai.Runtime.Vision.Sources;

public class MyCustomFrameSource : MonoBehaviour, IVisionFrameSource, IVisionFrameSourceStatusProvider
{
    public VisionSourceState State { get; private set; } = VisionSourceState.Idle;
    public VisionSourceErrorKind ErrorKind { get; private set; } = VisionSourceErrorKind.None;
    public string StatusMessage { get; private set; }
    public bool HasUsableFrame { get; private set; }
    public event Action StatusChanged;

    // ...
}

Assign your custom component to ConvaiVisionPublisher's Frame Source Component field. The publisher accepts any MonoBehaviour that implements IVisionFrameSource.

Domain Events

Vision publishes all significant state changes through the SDK's IEventHub. Subscribe to these events from any component that holds an IEventHub reference, or through the ConvaiManager's event infrastructure.

Event Reference

Event type

Namespace

Key fields

VisionCaptureStarted