> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/api-reference/core-api-reference/live-apis-beta/message-glossary.md).

# Message Glossary

Complete glossary of all message types available in Convai's Live APIs for real-time communication between clients and the server.

This glossary summarizes the message types used in Convai's Live APIs. Messages flow bidirectionally between your client application and Convai over WebRTC data channels.

***

## Message Directions

| Direction           | Description                                                            |
| ------------------- | ---------------------------------------------------------------------- |
| **Client → Server** | Messages you send to trigger actions, update state, or control the bot |
| **Server → Client** | Messages you receive for events, state changes, and real-time data     |

***

## Message Format

### Client → Server Messages

Messages sent from the client to the server use this structure:

```json
{
  "type": "<message-type>",
  "data": { ... }
}
```

* `type` *(string, required)*: The message type identifier
* `data` *(object, optional)*: Message payload (structure varies by type)

### Server → Client Messages

Most messages sent from the server to the client are wrapped in an RTVI envelope:

```json
{
  "label": "rtvi-ai",
  "type": "server-message",
  "data": {
    "type": "<message-type>",
    ...payload fields
  }
}
```

* `label` *(string)*: Always `"rtvi-ai"`
* `type` *(string)*: Always `"server-message"` for custom server messages
* `data` *(object)*: Contains the actual message with its own `type` and payload fields

There are **three envelope shapes** in total, and a client must handle all of them:

| Shape                  | Where the event type lives    | Used by                                                                 |
| ---------------------- | ----------------------------- | ----------------------------------------------------------------------- |
| Server-message wrapped | `data.type`                   | Most messages                                                           |
| Bot output stream      | top-level `type`              | `bot-llm-started`, `bot-llm-text`, `bot-llm-stopped`, `bot-tts-started` |
| Direct (legacy)        | top-level `type`, fields flat | `server-response`                                                       |

Resolve the effective event type like this:

```javascript
function eventType(message) {
  return message.type === "server-message" && message.data?.type
    ? message.data.type
    : message.type;
}
```

{% hint style="info" %}
In the detailed message documentation pages, examples show only the inner `data` payload for clarity.
{% endhint %}

### Field presence

Field presence is **not uniform** across message types. Some optional fields are emitted as `null`; others are omitted from the JSON entirely; nested optional fields such as `action-response.actions[].target` are dropped when unset.

Write clients defensively — use optional access rather than null checks. The full rules are in [Field presence rules](/api-docs/api-reference/core-api-reference/live-apis-beta/turn-lifecycle-and-message-ordering.md#field-presence-rules).

***

## Server Response Messages

For every client-to-server message, the server automatically sends a `server-response` message to acknowledge receipt and indicate the processing status. This is similar to HTTP response codes in REST APIs.

**Example Response:**

```json
{
  "type": "server-response",
  "event_type": "context-update",
  "status": "success",
  "message": "Context updated successfully (append mode)",
  "extras": {
    "token_count": 1523,
    "max_tokens": 50000,
    "remaining_tokens": 48477
  }
}
```

| Field        | Type   | Description                                                            |
| ------------ | ------ | ---------------------------------------------------------------------- |
| `event_type` | string | The original client message type that triggered this response          |
| `status`     | string | Processing status: `"success"`, `"error"`, `"processing"`, `"pending"` |
| `message`    | string | Human-readable description of the result (optional)                    |
| `extras`     | object | Additional event-specific data (optional)                              |

**Status Values:**

* `"success"` - Message processed successfully
* `"error"` - An error occurred (see `message` for details)
* `"processing"` - Message is being processed asynchronously
* `"pending"` - Message received but processing delayed

***

## Client → Server Messages

| Message Type                  | Purpose                                    | Details Page                                                                                                                                       |
| ----------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `trigger-message`             | Trigger narrative events or send context   | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#trigger-message)             |
| `user_text_message`           | Send text input as user                    | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#user_text_message)           |
| `update-template-keys`        | Update prompt template variables           | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#update-template-keys)        |
| `update-scene-metadata`       | Update scene objects                       | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#update-scene-metadata)       |
| `update-dynamic-info`         | Update dynamic context (basic)             | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#update-dynamic-info)         |
| `context-update`              | Update runtime context (with mode control) | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#context-update)              |
| `action-result`               | Return a correlated v2 client tool result  | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#action-result)               |
| `vision-status`               | Query vision buffer state                  | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#vision-status)               |
| `vision-trigger`              | Attach buffered frames / trigger vision    | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#vision-trigger)              |
| `tts-toggle`                  | Enable/disable bot audio                   | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#tts-toggle)                  |
| `stt-toggle`                  | Mute/unmute speech recognition             | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#stt-toggle)                  |
| `interrupt-bot`               | Stop bot speech immediately                | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#interrupt-bot)               |
| `force-user-stopped-speaking` | Signal end of user speech (push-to-talk)   | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#force-user-stopped-speaking) |
| `reset-idle-timer`            | Reset idle timeout monitoring              | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#reset-idle-timer)            |
| `usage-toggle`                | Enable/disable the client usage stream     | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#usage-toggle)                |
| `kill-pipeline`               | End the session                            | [client-to-server-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#kill-pipeline)               |
| `group-address`               | Address a group chat room for one turn     | [group-chat-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/group-chat-messages.md#group-address)                           |
| `group-address-part`          | One frame of a large `group-address`       | [group-chat-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/group-chat-messages.md#group-address-part)                      |

***

## Server → Client Messages

| Message Type                    | Purpose                                       | Format                 | Details Page                                                                                                                                                     |
| ------------------------------- | --------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bot-llm-started`               | Model generation began                        | Bot output stream      | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-llm-started-bot-llm-stopped)           |
| `bot-llm-text`                  | Selected legacy or raw text projection        | Bot output stream      | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-llm-text)                              |
| `bot-llm-stopped`               | Model generation finished                     | Bot output stream      | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-llm-started-bot-llm-stopped)           |
| `bot-tts-started`               | Speech synthesis began                        | Bot output stream      | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-tts-started)                           |
| `bot-started-speaking`          | Bot audio began                               | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-started-speaking-bot-stopped-speaking) |
| `bot-stopped-speaking`          | Bot audio ended                               | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-started-speaking-bot-stopped-speaking) |
| `server-response`               | Response for every client message             | Direct (legacy)        | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#server-response)                           |
| `interaction-created`           | Interaction ID created                        | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#interaction-created)                       |
| `usage-limit-reached`           | Quota exceeded notification                   | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#usage-limit-reached)                       |
| `bot-turn-completed`            | Bot finished speaking                         | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-turn-completed)                        |
| `bot-emotion`                   | Bot emotion for avatar                        | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#bot-emotion)                               |
| `behavior-tree-response`        | Behavior tree data                            | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#behavior-tree-response)                    |
| `moderation-response`           | Content moderation result                     | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#moderation-response)                       |
| `model-output`                  | Canonical typed v2 model output               | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#model-output)                              |
| `action-response`               | Actions/animations to trigger                 | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#action-response)                           |
| `final-user-transcription`      | User speech transcription                     | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#final-user-transcription)                  |
| `visemes`                       | Lip-sync data                                 | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#visemes)                                   |
| `neurosync-blendshapes`         | Facial animation data                         | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#neurosync-blendshapes)                     |
| `chunked-neurosync-blendshapes` | Batched facial animation                      | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#chunked-neurosync-blendshapes)             |
| `neurosync-blendshapes-cancel`  | Drop buffered ahead-delivered visuals         | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#neurosync-blendshapes-cancel)              |
| `blendshape-turn-stats`         | Turn statistics                               | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#blendshape-turn-stats)                     |
| `user-idle-warning`             | Idle timeout warning                          | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#user-idle-warning)                         |
| `llm-no-response`               | LLM chose not to respond                      | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#llm-no-response)                           |
| `vad-stt-started`               | STT started transcribing                      | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#vad-stt-started)                           |
| `vad-stt-stopped`               | STT stopped transcribing                      | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#vad-stt-stopped)                           |
| `vad-stt-debug`                 | VAD debug events (debug sessions only)        | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#vad-stt-debug)                             |
| `turn-trace`                    | Per-turn timing trace (debug sessions only)   | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#diagnostics)                               |
| `server-log`                    | Server log lines (debug sessions only)        | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#diagnostics)                               |
| `usage-update`                  | Per-turn usage and cost (debug sessions only) | Server-message wrapped | [server-to-client-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md#diagnostics)                               |
| `audio-data`                    | Audio chunks via data channel (custom mode)   | Server-message wrapped | See [Audio Data via Data Channel](/api-docs/api-reference/core-api-reference/live-apis-beta/audio-data-via-data-channel.md)                                      |
| `turn-complete`                 | Group chat turn finished                      | Server-message wrapped | [group-chat-messages.md](/api-docs/api-reference/core-api-reference/live-apis-beta/group-chat-messages.md#turn-complete)                                         |

**Format Key:**

* **Server-message wrapped**: Uses the full RTVI envelope format with `"type": "server-message"` and event data nested in `data.type` and subsequent fields
* **Direct (legacy)**: Uses `data` as a flat object with the event type and fields at the top level

{% hint style="info" %}
All client messages receive a `server-response` acknowledgment with success/error status and additional data.
{% endhint %}

***

## Common Response Extras by Event Type

When you receive a `server-response` message, the `extras` field may contain event-specific data:

| Event Type          | Extras Fields                                                  |
| ------------------- | -------------------------------------------------------------- |
| `context-update`    | `token_count`, `max_tokens`, `remaining_tokens`, `content`     |
| `tts-toggle`        | `enabled`                                                      |
| `stt-toggle`        | `muted`                                                        |
| `trigger-message`   | `trigger_name`, `has_speak_tag`                                |
| `user_text_message` | `text`                                                         |
| `action-result`     | `tool_call_id`, `idempotent`; errors also include `error_code` |

***

## Error Response Examples

### Invalid JSON

```json
{
  "type": "server-response",
  "event_type": "parse-error",
  "status": "error",
  "message": "Failed to parse message: invalid JSON or UTF-8 encoding"
}
```

### Missing Type Field

```json
{
  "type": "server-response",
  "event_type": "validation-error",
  "status": "error",
  "message": "Message missing required 'type' field"
}
```

### Unknown Message Type

```json
{
  "type": "server-response",
  "event_type": "unknown-message-type",
  "status": "error",
  "message": "Unknown message type: unknown-message-type",
  "extras": {
    "supported_types": ["trigger-message", "context-update", "tts-toggle", ...]
  }
}
```

***

## Message Categories

### Bot Response

The character's own output:

* `bot-llm-started` / `bot-llm-stopped` - Model generation boundaries
* `bot-llm-text` - The selected legacy or raw text projection
* `model-output` - Canonical typed output for clients that negotiate model output v2
* `bot-tts-started` - Speech synthesis began
* `bot-started-speaking` / `bot-stopped-speaking` - Audio boundaries
* `bot-turn-completed` - Server-side terminal state for the turn

### Context & State Management

Messages for managing conversation context and bot state:

* `context-update` - Unified context updates with mode control, including action affordances
* `update-dynamic-info` - Basic dynamic context updates
* `update-template-keys` - Update prompt template variables
* `update-scene-metadata` - Update scene object descriptions

### Agentic actions

Messages for correlated client-executed tools:

* `action-response` - Legacy semantic action or v2 tool-call compatibility projection
* `model-output` - Canonical semantic items when model output v2 is selected
* `action-result` - Terminal client result for a v2 `tool_call`

### Vision

Messages for querying and consuming the vision buffer:

* `vision-status` - Query whether frames are available and inspect buffer state
* `vision-trigger` - Attach buffered frames and optionally trigger a bot turn

### Audio Control

Messages for controlling audio input and output:

* `tts-toggle` - Enable/disable text-to-speech output
* `stt-toggle` - Mute/unmute speech-to-text input
* `interrupt-bot` - Interrupt current bot speech
* `force-user-stopped-speaking` - Signal end of user speech

### Interaction & Events

Messages for triggering events and sending user input:

* `trigger-message` - Trigger narrative events or contextual actions
* `user_text_message` - Send text as user input

### Session Management

Messages for managing the session lifecycle:

* `reset-idle-timer` - Reset idle timeout

### Animation & Visual Feedback

Messages containing animation and visual data:

* `bot-emotion` - Emotion data for avatar expressions
* `visemes` - Lip-sync blendshape data
* `neurosync-blendshapes` - Facial animation blendshapes (single frame)
* `chunked-neurosync-blendshapes` - Batched facial animation blendshapes
* `action-response` - Semantic actions or v2 client tool-call projections

### Transcription & Text

Messages containing text and transcription data:

* `final-user-transcription` - Final transcription of user speech

### System Events

Messages about system state and events:

* `server-response` - Acknowledgment of client messages
* `interaction-created` - Session interaction ID created
* `bot-turn-completed` - Bot turn finished
* `usage-limit-reached` - Usage quota exceeded
* `user-idle-warning` - User idle timeout warning
* `llm-no-response` - LLM chose not to respond

### Voice Activity Detection

Messages from the VAD-based STT gating system:

* `vad-stt-started` - STT service started transcribing
* `vad-stt-stopped` - STT service stopped transcribing
* `vad-stt-debug` - VAD debug events (debug mode only)

***

## Related Documentation

* [Connect API](/api-docs/api-reference/core-api-reference/live-apis-beta/connect-api.md) - Establish a live session
* [Turn lifecycle and message ordering](/api-docs/api-reference/core-api-reference/live-apis-beta/turn-lifecycle-and-message-ordering.md) - How a turn is delivered, and what ordering you can rely on
* [Response contract and parsing](/api-docs/api-reference/core-api-reference/live-apis-beta/response-contract-and-parsing.md) - How speech, actions, and emotion are separated, and what the server removes
* [Client to Server Messages](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md) - Detailed client message reference
* [Server to Client Messages](/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md) - Detailed server message reference
* [Audio Data via Data Channel](/api-docs/api-reference/core-api-reference/live-apis-beta/audio-data-via-data-channel.md) - Custom audio handling
* [Metrics](/api-docs/api-reference/core-api-reference/live-apis-beta/metrics.md) - Performance metrics and monitoring

***


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.convai.com/api-docs/api-reference/core-api-reference/live-apis-beta/message-glossary.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.