For the complete documentation index, see llms.txt. This page is also available as Markdown.

Client-to-server messages

Reference for all messages a client sends to Convai's Live API server over the WebRTC data channel, including payloads, fields, and response details.

All client-to-server messages are sent as JSON over the WebRTC data channel established after a successful /connect call. For message format conventions, status codes returned by the server, and a full index of all message types in both directions, see the Message Glossary.

Message envelope

Every client-to-server message uses this envelope:

{
  "type": "<message-type>",
  "data": { ... }
}
Field
Type
Required
Description

type

string

Yes

The message type identifier.

data

object

No

Message payload. Structure varies by message type. Omit for messages with no payload.

The server acknowledges every client message with a server-response message. See Server response below.

Server response

The server sends a server-response message for every client-to-server message it receives, regardless of type.

{
  "type": "server-response",
  "event_type": "tts-toggle",
  "status": "success",
  "message": "TTS enabled",
  "extras": {
    "enabled": true
  }
}
Field
Type
Description

event_type

string

The client message type that triggered this response.

status

string

Processing status: "success", "error", "processing", or "pending".

message

string or null

Human-readable description of the result.

extras

object or null

Additional event-specific data. Structure varies by event type.

Status values:

Value
Meaning

"success"

Message processed successfully.

"error"

An error occurred. Check message for details.

"processing"

Message is being processed asynchronously.

"pending"

Message received but processing is delayed.


Interaction & events

trigger-message

Triggers a narrative event, sends contextual information to the bot, or initiates specific bot behaviors.

Field
Type
Required
Description

trigger_name

string

No

Name or identifier of the trigger.

trigger_message

string

No

Context or message content for the trigger.

Use cases:

  • Notify the bot of in-scene events, such as a player entering an area or completing a quest.

  • Send contextual triggers to drive narrative flow.

  • Initiate specific bot behaviors tied to named triggers.

Server response extras:

Field
Type
Description

trigger_name

string

The trigger name echoed from the request.

has_speak_tag

boolean

Whether the trigger message contained a speak tag.


user_text_message

Sends text as user input, simulating speech without audio.

Field
Type
Required
Description

text

string

Yes

The text to send as user input.

Server response extras:

Field
Type
Description

text

string

The text echoed from the request.


Context & state

update-template-keys

Updates template variables used in the bot's system prompt.

Field
Type
Required
Description

template_keys

object

Yes

Key-value pairs of template variable names and their updated string values.

Use cases:

  • Update dynamic prompt variables such as player name, stats, or game state.

  • Customize bot responses based on the current session state.


update-scene-metadata

Updates the descriptive scene context the bot knows about, including in-scene objects and their descriptions.

Field
Type
Required
Description

data.scene_metadata

object[]

Yes

Array of scene objects to describe.

data.scene_metadata[].name

string

Yes

Object identifier.

data.scene_metadata[].description

string

Yes

Human-readable description of the object.

Use cases:

  • Update interactable objects in the scene as the environment changes.

  • Adjust environment context for the bot without modifying action affordances.

update-scene-metadata updates descriptive context only. It does not modify action_config.objects. Use context-update to replace semantic action objects, or reconnect when changing v2 client tool declarations.


update-dynamic-info

Updates dynamic information injected into the bot's system prompt. Use this for basic single-field context updates. For mode-controlled updates with token budget tracking, use context-update instead.

Field
Type
Required
Description

dynamic_info.text

string

Yes

Dynamic context text to inject into the system prompt.


context-update

Updates the bot's runtime dynamic context with full control over mode, token budget, and LLM triggering. This is the preferred message for runtime context management.

Field
Type
Required
Default
Description

text

string

Yes, unless mode is "reset"

—

The context text to apply.

mode

string

No

"append"

How to apply the context.

run_llm

string

No

"auto"

Whether to trigger an LLM response after the update.

current_attention_object

string or object

No

—

Updates the active attention object for action-reference grounding.

action_config

object

No

—

Replaces the provided action affordance lists for the active session. See below.

remove_static

boolean

No

false

For "reset" mode only: when true, also clears the static context.

Updating action affordances mid-session

action_config accepts the semantic actions, objects, characters, and current_attention_object fields from the Connect API. Only the lists you provide are replaced—sending only objects leaves actions and characters untouched. Use this when the scene changes and the character's semantic affordances change with it.

action_config.tools is connection-scoped. A context-update that includes tools returns an error; reconnect to replace client tool declarations. Adding an object to scene_description or text does not make it a semantic action target. See Response contract and parsing.

Mode values

Value
Behavior

"append"

Adds text to existing runtime dynamic context.

"replace"

Replaces existing runtime dynamic context with text.

"reset"

Clears runtime dynamic context. Also clears static context if remove_static is true. text is not required.

run_llm values

Value
Behavior

"true"

Always trigger a bot response after the update.

"false"

Never trigger a bot response.

"auto"

Convai decides whether to trigger a response based on context.

Token budget

Dynamic context uses an estimated token budget:

Partition
Budget

static_text (session-level context supplied at connect time)

20,000 estimated tokens

Runtime text (from append and replace updates)

30,000 estimated tokens

Combined dynamic context

50,000 estimated tokens

When runtime text exceeds its 30,000-token budget, Convai trims the oldest runtime context updates and retains the newest updates that fit within the budget.

static_text is session and environment context supplied for this connection. It is preserved across reset calls unless remove_static is true.

Attention update rules

  • current_attention_object is validated against the connected session's action_config.objects.

  • Send either the object name string or the full object payload.

  • Send an empty string ("") to clear the current attention object.

  • Updating current_attention_object regenerates the system prompt, even when run_llm is "false".

Success response

On success, the server returns a server-response with the following extras:

Field
Type
Description

token_count

integer

Current total estimated tokens in dynamic context.

static_token_count

integer

Estimated tokens from static dynamic context.

runtime_token_count

integer

Estimated tokens from runtime dynamic context.

max_tokens

integer

Combined dynamic context budget (50,000 estimated tokens).

static_max_tokens

integer

Static dynamic context budget (20,000 estimated tokens).

runtime_max_tokens

integer

Runtime dynamic context budget (30,000 estimated tokens).

remaining_tokens

integer

Estimated tokens remaining before the combined limit is reached.

content

string

Full retained runtime dynamic context text.

Legacy word_count, static_word_count, runtime_word_count, max_words, and remaining_words fields may appear for older clients. The token fields are authoritative.

Error response

When a token limit is exceeded, the server returns a server-response with "status": "error":

  • Validation checks apply to the static, runtime, and combined estimated-token limits independently.

  • When combined dynamic context exceeds 40,000 estimated tokens, Convai logs a server-side warning.

  • Error messages include the estimated-token breakdown for both partitions.


Agentic action results

action-result

Returns one terminal result for a client-executed tool call. This message requires action protocol v2 and correlates to the id in a tool_call item or v2 action-response projection.

Field
Type
Required
Description

id

string

Yes

Correlation ID from the tool call. Length is 1–256 characters.

status

string

Yes

Terminal status: "completed", "error", or "cancelled".

output

JSON value

No

Result data. Use with "completed" when the model needs output from the operation.

error

JSON value

No

Failure or cancellation details. A completed result cannot include this field.

character_session_id

string

No

Session guard. When present, it must match the active character session.

Convai does not execute or authorize the requested operation. Validate the tool name and arguments, apply your application's permission and confirmation rules, execute on the client, and then return the terminal result.

The candidate implementation accepts a maximum serialized result size of 64 KiB. It allows up to 8 outstanding calls and up to 8 tool-result continuation rounds for one user turn. Convai waits 60 seconds by default for each result. A timed-out call becomes stale, and a late result is rejected.

An accepted result produces this acknowledgment:

Retrying the same terminal payload for the same id is accepted with extras.idempotent: true. Sending a different terminal payload for an already completed id returns conflicting_terminal_tool_result. Unknown, stale, cross-session, oversized, and participant-mismatched results return an error server-response with extras.error_code and, when available, extras.tool_call_id.

After accepting the result, Convai supplies it to the same model context so generation can continue. The acknowledgment is not an ordering barrier; continuation output can arrive before its server-response. Distinct calls may be outstanding in parallel. Do not infer that the SDK or Convai executed them sequentially.


Vision

Vision messages query or consume the session's vision ring buffer. Frames enter that buffer from the WebRTC video track (LiveKit) once vision_input_config is set on /connect. These RTVI control messages do not publish image bytes themselves.

vision-status

Queries the current vision buffer state without attaching frames or triggering the LLM. Use this after enabling vision, after camera-off, or whenever you need confirmation that frames are available before calling vision-trigger.

Field
Type
Required
Description

update_id

string

No

Client correlation id. Echoed in the ack and used for duplicate replay. If omitted, a top-level message id is used when present.

Success response

Field
Type
Description

vision_status_outcome

string

High-level buffer outcome.

active_source

string | null

Selected participant id, when a source is active.

active_source_label

string | null

Human-readable source label, when known.

last_frame_age_ms

integer | null

Age of the newest retained frame in milliseconds.

vision_buffer

object

Client-safe ring-buffer snapshot. Never includes image bytes.

update_id

string

Echoed when supplied on the request.

duplicate

boolean

true when this ack is a replay of a previous update_id.

vision_status_outcome / vision_buffer.status values

Value
Meaning

frames_available

Source is active and the buffer has retained frames.

buffer_empty

Source is active, but no frames are retained yet.

no_active_video

No active visual source is selected.

vision_not_enabled

Vision is not enabled for this session.

Use cases:

  • Confirm frames landed after the user enabled camera or screen share.

  • Inspect buffer depth / sampling windows before issuing a vision-trigger.

  • Detect camera-off / stale feed without attaching frames.


vision-trigger

Attaches buffered vision frames into LLM context and optionally triggers a bot turn. Frames are never auto-attached by this message alone when vision is disabled or the buffer is empty with no text.

Field
Type
Required
Default
Description

respond_mode

string

No

Session default for vision

"silent", "auto", or "must_respond". Invalid explicit values return an error ack.

text

string

No

Default vision prompt when LLM runs

Optional instruction attached with the frames. When present, text-only triggers can still proceed even if the buffer is empty.

frame_indices

integer[]

No

Default dual-horizon selection

Relative indices into the retained buffer. -1 is the newest frame. Duplicate indices collapse to one frame.

frame_ids

integer[]

No

—

Absolute frame presentation timestamps (attached_frame_pts / vision-status pts values). Takes precedence over frame_indices when both are set.

update_id

string

No

—

Client correlation id. Echoed in the ack and used for duplicate replay. If omitted, a top-level message id is used when present.

respond_mode values

Value
Behavior

"silent"

Attach frames only. Does not invoke the LLM.

"auto"

Attach frames and invoke the LLM only when the bot is idle and the user is not speaking. Otherwise the request is downgraded silently without attaching.

"must_respond"

Attach frames and invoke the LLM. If the bot is busy, Convai interrupts first. If the user is speaking, attach/reserve frames and respond after the user turn.

Connect-time respond_modes.vision seeds the default when respond_mode is omitted.

Success response

Field
Type
Description

requested_respond_mode

string

Resolved request mode, or "invalid".

actual_respond_mode

string

Mode actually applied after idle/busy gates.

requested_run_llm / actual_run_llm

string

"true", "false", or "auto".

llm_triggered

boolean

Whether an LLM turn was started.

downgraded

boolean

Whether the request was reduced (busy bot, empty buffer, etc.).

downgrade_reason

string | null

Why the request was downgraded, when applicable.

interrupted

boolean

true when must_respond interrupted an in-progress bot turn.

vision_trigger_outcome

string

Final outcome for the trigger.

vision_attach_outcome

string

Attach result when frames were selected.

vision_frames_attached

integer

Number of frames attached.

vision_image_tokens_est

integer

Estimated image tokens for the attach.

attached_frame_pts

integer[]

PTS values of attached frames.

frame_binding

string

How frames were chosen (default, frame_indices, frame_ids).

requested_frame_indices / requested_frame_ids

array | null

Echo of the request selection fields.

vision_buffer

object

Buffer snapshot after handling the trigger.

Common vision_trigger_outcome values

Value
Meaning

attached

Frames were attached.

deduped_stub

Frames matched the previous attach and were stubbed.

stale_skipped

Buffer was considered stale and no text was supplied.

frames_available

Buffer has frames, but the trigger did not attach (for example auto while bot is busy).

buffer_empty

Source active, no retained frames, and no text.

no_active_video

No active visual source, and no text.

vision_not_enabled

Vision is not enabled for the session.

invalid_respond_mode

Explicit respond_mode was not one of the allowed values.

frame_id_evicted / invalid_frame_ids / invalid_frame_indices / invalid_frame_indices_range / rate_limited

Frame binding failed.

update_id makes retries safe: a repeated vision-status or vision-trigger with the same ID replays the prior acknowledgment with "duplicate": true and does not attach or trigger again.

Use cases:

  • Ask the character to comment on the current camera or screen feed.

  • Silently prime vision context (respond_mode: "silent") ahead of a later user turn.

  • Re-attach specific recent frames via frame_indices or frame_ids.


Audio control

tts-toggle

Enables or disables the bot's text-to-speech audio output.

Field
Type
Required
Description

enabled

boolean

Yes

true to enable TTS output. false to disable it.

Server response extras:

Field
Type
Description

enabled

boolean

The enabled state echoed from the request.

Use cases:

  • Mute bot audio during cutscenes or narrated sequences.

  • Toggle audio output for accessibility requirements.


stt-toggle

Mutes or unmutes speech-to-text microphone input processing.

Field
Type
Required
Description

muted

boolean

Yes

true to stop listening. false to resume listening.

Server response extras:

Field
Type
Description

muted

boolean

The muted state echoed from the request.

Use cases:

  • Implement push-to-talk input by muting STT when the button is not held.

  • Disable voice input during moments where user speech should not be processed.


interrupt-bot

Immediately stops the bot's current speech.

No data payload is required.

Use cases:

  • Allow a user to interrupt the bot mid-speech.

  • Stop bot audio immediately when an in-scene event requires silence.


force-user-stopped-speaking

Signals to the server that the user has finished speaking. Use this in push-to-talk implementations to end the speech segment explicitly.

No data payload is required.

Use cases:

  • Signal end-of-speech when the push-to-talk button is released.

  • Provide a manual end-of-speech signal when VAD is not used.


Session management

reset-idle-timer

Resets the user idle timeout countdown. Send this to signal user activity and prevent idle disconnection.

No data payload is required. The server ignores any data content.

Use cases:

  • Reset the timer when the user performs a UI action such as a click or keypress.

  • Detect user activity outside of voice interaction (for example, mouse movement) and prevent idle timeout.

  • Keep a session alive during extended non-voice interactions.


usage-toggle

Enables or disables streaming of informational usage-update messages to this client.

Field
Type
Required
Description

enabled

boolean

Yes

true resumes usage-update streaming, false stops it.

This controls only whether the server pushes usage messages to your client. It does not affect usage tracking, aggregation, or billing. usage-update messages are available only when the session runs in debug mode with usage tracking enabled.

Use cases:

  • Stop the real-time cost stream when a debug usage panel is hidden.

  • Reduce data-channel traffic for clients that do not render live usage.


kill-pipeline

Terminates the session and closes the connection.

No data payload is required.

Use cases:

  • Cleanly end a session when the user exits the experience.

  • Release resources without waiting for an idle timeout.


Next steps

Message GlossaryConnect APIAudio Data (via data channel)

Last updated

Was this helpful?