> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md).

# Server-to-client messages

Complete reference for all messages Convai sends to the client over a WebRTC data channel in Live APIs, including fields, types, and recommended actions.

The Convai Live API server sends these messages over the WebRTC data channel during an active session. Each message type signals a distinct event — an acknowledgment, a bot state change, animation data, or a session limit. See the [Message Glossary](/api-docs/api-reference/core-api-reference/live-apis-beta/message-glossary.md) for a summary of all message types and their envelope formats.

Most server messages use the RTVI envelope format shown under [`interaction-created`](#interaction-created). Subsequent examples show only the inner `data` payload. `server-response` uses a flat legacy shape, while the [bot output stream](#bot-output-stream) places its event type at the top level. See [Turn lifecycle and message ordering](/api-docs/api-reference/core-api-reference/live-apis-beta/turn-lifecycle-and-message-ordering.md#two-envelope-forms) for demultiplexing logic.

Field presence is not uniform. Some optional fields are emitted as `null`, while others are omitted. Read [Field presence rules](/api-docs/api-reference/core-api-reference/live-apis-beta/turn-lifecycle-and-message-ordering.md#field-presence-rules) and use optional access rather than assuming a key exists.

***

### Bot output stream

These messages carry the character's response text and its speech-state transitions. Unlike the rest of this page, they place the event type at the **top level** of the message rather than nesting it under `data`.

#### bot-llm-text

The bot text projection, streamed in chunks. Concatenate `data.text` in arrival order to rebuild the projection selected at `/connect`.

**Full message**

```json
{ "label": "rtvi-ai", "type": "bot-llm-text", "data": { "text": "Sure, on my way." } }
```

| Field  | Type   | Description                                          |
| ------ | ------ | ---------------------------------------------------- |
| `text` | string | An incremental chunk of the selected text projection |

With omitted capabilities or `bot_llm_text_mode: "legacy"`, this is the filtered text used by the conversational path. With `bot_llm_text_mode: "raw"`, it is provider-visible text before Convai's structured-output parsing and conversational filtering. Raw mode is diagnostic and may contain JSON, control syntax, refusal text, or other content that should not be executed or sent to speech synthesis. It is not guaranteed to carry non-text native tool-call deltas. See [Response contract and parsing](/api-docs/api-reference/core-api-reference/live-apis-beta/response-contract-and-parsing.md#bot-llm-text-modes).

**Recommended action:** Append to the in-progress bot message in your transcript UI.

***

#### bot-llm-started / bot-llm-stopped

Bracket the model generation phase for a turn. Both carry an empty `data` object.

```json
{ "label": "rtvi-ai", "type": "bot-llm-started", "data": {} }
{ "label": "rtvi-ai", "type": "bot-llm-stopped", "data": {} }
```

**Recommended action:** Show and hide a "thinking" indicator. Do not use `bot-llm-stopped` to gate action execution — see [Ordering guarantees](/api-docs/api-reference/core-api-reference/live-apis-beta/turn-lifecycle-and-message-ordering.md#ordering-guarantees).

***

#### bot-tts-started

Speech synthesis has begun for this turn. Carries an empty `data` object.

```json
{ "label": "rtvi-ai", "type": "bot-tts-started", "data": {} }
```

***

#### bot-started-speaking / bot-stopped-speaking

Mark the audio boundaries of the bot's turn. These use the `server-message` envelope, and additionally repeat `label` inside `data`.

**Full message**

```json
{
  "label": "rtvi-ai",
  "type": "server-message",
  "data": {
    "label": "rtvi-ai",
    "type": "bot-started-speaking",
    "response_id": "session-id:r4",
    "epoch": 1,
    "sequence": 3
  }
}
```

| Field               | Type    | Presence      | Description                        |
| ------------------- | ------- | ------------- | ---------------------------------- |
| `label`             | string  | Always        | Always `"rtvi-ai"`                 |
| `response_id`       | string  | Only when set | Identifier for this bot response   |
| `neurosync_turn_id` | integer | Only when set | NeuroSync turn identifier          |
| `epoch`             | integer | Only when set | NeuroSync connection/session epoch |
| `sequence`          | integer | Only when set | Per-turn message sequence number   |

**Recommended action:** Drive an `isSpeaking` indicator. Use `response_id` to correlate blendshape and cancel messages with the turn that produced them.

***

### Acknowledgment

#### server-response

`server-response` is sent for every client-to-server message to acknowledge receipt and report the processing outcome. It uses the **direct (legacy) format** — not the RTVI envelope — so the fields appear at the top level of the JSON object, not nested under `data`.

```json
{
  "type": "server-response",
  "event_type": "tts-toggle",
  "status": "success",
  "message": "TTS enabled",
  "extras": {
    "enabled": true
  }
}
```

| Field        | Type           | Description                                                               |
| ------------ | -------------- | ------------------------------------------------------------------------- |
| `type`       | string         | Always `"server-response"`                                                |
| `event_type` | string         | The client message type that triggered this response                      |
| `status`     | string         | Processing status: `"success"`, `"error"`, `"processing"`, or `"pending"` |
| `message`    | string \| null | Human-readable description of the result                                  |
| `extras`     | object \| null | Additional event-specific data                                            |

**Status values**

| Value          | Meaning                                      |
| -------------- | -------------------------------------------- |
| `"success"`    | Message processed successfully               |
| `"error"`      | An error occurred; see `message` for details |
| `"processing"` | Message is being processed asynchronously    |
| `"pending"`    | Message received but processing delayed      |

**`extras` fields by event type**

| `event_type`        | `extras` fields                                                                                                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `context-update`    | `token_count`, `static_token_count`, `runtime_token_count`, `max_tokens`, `static_max_tokens`, `runtime_max_tokens`, `remaining_tokens`, `content`, `update_id`, `revision`, `duplicate` |
| `tts-toggle`        | `enabled`                                                                                                                                                                                |
| `stt-toggle`        | `muted`                                                                                                                                                                                  |
| `usage-toggle`      | `enabled`                                                                                                                                                                                |
| `trigger-message`   | `trigger_name`, `has_speak_tag`                                                                                                                                                          |
| `user_text_message` | `text`                                                                                                                                                                                   |
| `action-result`     | `tool_call_id`, `idempotent`; errors also include `error_code`                                                                                                                           |

**About the `message` field**

`message` is a **human-readable diagnostic string intended for developers and logs**. It is not a stable identifier.

{% hint style="danger" %}
Do not branch application logic on the text of `message`. It is not versioned and its wording may change between releases. Branch on `status`, and on the fields in `extras`.
{% endhint %}

Representative values, to give a sense of what you will see:

| Situation                                                      | Example `message`                                                                                                |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Context applied normally                                       | `Context updated successfully (append mode, run_llm=auto)`                                                       |
| Context applied, response withheld because the user is talking | `Context updated silently (requested run_llm=true, but user is speaking - bot will respond after user finishes)` |
| Context applied, response withheld due to bot state            | `Context updated silently (requested run_llm=true, downgraded due to bot state: speaking)`                       |
| Context applied and the bot was interrupted to respond         | `Context updated with INTERRUPTION (run_llm=true interrupted bot speaking, triggering new response)`             |
| Toggles                                                        | `TTS enabled`, `STT muted`, `Usage-update streaming enabled`                                                     |
| Debounced duplicates                                           | `TTS toggle debounced (duplicate enable within window)`, `Bot interrupt debounced (duplicate within window)`     |
| Successful no-ops                                              | `Trigger message processed without response`, `Trigger processed but no context generated`                       |
| Failures                                                       | `Narrative design service not available`, `No text provided in dynamic info`, `Usage limit exceeded`             |
| Unhandled server error                                         | `Failed to process <event_type>: <error detail>`                                                                 |

When a handler succeeds with nothing to report, `message` is **omitted entirely** rather than sent as an empty string or `null`.

**Error example**

```json
{
  "type": "server-response",
  "event_type": "tts-toggle",
  "status": "error",
  "message": "TTS bypass filter not available"
}
```

**Validation error — invalid JSON**

```json
{
  "type": "server-response",
  "event_type": "parse-error",
  "status": "error",
  "message": "Failed to parse message: invalid JSON or UTF-8 encoding"
}
```

**Validation error — missing type field**

```json
{
  "type": "server-response",
  "event_type": "validation-error",
  "status": "error",
  "message": "Message missing required 'type' field"
}
```

**Validation error — unknown message type**

```json
{
  "type": "server-response",
  "event_type": "unknown-message-type",
  "status": "error",
  "message": "Unknown message type: unknown-message-type",
  "extras": {
    "supported_types": ["trigger-message", "context-update", "tts-toggle"]
  }
}
```

**Recommended action:** Check the `status` field on every `server-response`. Handle `"error"` responses by reading `message` and updating your UI. Use the `extras` fields to reflect current state — for example, updating a mute indicator when `event_type` is `stt-toggle`.

***

### Session & lifecycle

#### interaction-created

Sent early in the session lifecycle when an interaction ID is created. This is the first message that carries the full RTVI envelope.

**Full RTVI envelope**

```json
{
  "label": "rtvi-ai",
  "type": "server-message",
  "data": {
    "type": "interaction-created",
    "interaction_id": "int_abc123def456",
    "character_session_id": "cs_xyz789"
  }
}
```

**Inner `data` payload**

```json
{
  "type": "interaction-created",
  "interaction_id": "int_abc123def456",
  "character_session_id": "cs_xyz789"
}
```

| Field                  | Type   | Description                                    |
| ---------------------- | ------ | ---------------------------------------------- |
| `type`                 | string | Always `"interaction-created"`                 |
| `interaction_id`       | string | Unique identifier for this interaction session |
| `character_session_id` | string | Character session identifier                   |

**Recommended action:** Store `interaction_id` for analytics, logging, or session tracking.

***

#### usage-limit-reached

Sent when a usage quota is exceeded. Handle this message to display appropriate feedback and close the session.

```json
{
  "type": "usage-limit-reached",
  "quota_type": "minutes",
  "message": "You have exceeded your monthly quota"
}
```

| Field        | Type   | Description                                                      |
| ------------ | ------ | ---------------------------------------------------------------- |
| `type`       | string | Always `"usage-limit-reached"`                                   |
| `quota_type` | string | Type of quota exceeded, for example `"minutes"` or `"api_calls"` |
| `message`    | string | Human-readable message explaining the limit                      |

**Recommended action:** Display the `message` to the user and gracefully end the session.

***

#### bot-turn-completed

Sent when the bot reaches a terminal turn state: it finished speaking, was interrupted by the user, or aborted because required output could not be delivered.

```json
{
  "type": "bot-turn-completed",
  "was_interrupted": false
}
```

| Field             | Type    | Description                                                                           |
| ----------------- | ------- | ------------------------------------------------------------------------------------- |
| `type`            | string  | Always `"bot-turn-completed"`                                                         |
| `was_interrupted` | boolean | `true` if the user interrupted the bot; `false` if the turn completed normally        |
| `was_aborted`     | boolean | Optional. `true` if the turn ended because required bot output could not be delivered |
| `error_reason`    | string  | Optional. Machine-readable abort reason; currently `"audio_delivery_failed"`          |

`was_aborted` and `error_reason` are additive optional fields. Normal completions omit them.

**Recommended action:** Update UI state and re-enable user input controls when this message arrives.

***

#### user-idle-warning

Sent when the user has been idle for a configured period, warning that disconnection is approaching.

```json
{
  "type": "user-idle-warning",
  "remaining_seconds": 300,
  "message": "You've been idle. You will be disconnected in 5 minutes."
}
```

| Field               | Type           | Description                                          |
| ------------------- | -------------- | ---------------------------------------------------- |
| `type`              | string         | Always `"user-idle-warning"`                         |
| `remaining_seconds` | integer        | Seconds remaining before the session is disconnected |
| `message`           | string \| null | Optional human-readable warning message              |

**Recommended action:** Display the warning to the user and prompt for activity, or send a [`reset-idle-timer`](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#reset-idle-timer) message to reset the idle timer.

***

#### llm-no-response

Sent when the LLM explicitly decides not to respond to user input.

```json
{
  "type": "llm-no-response",
  "reason": "abstain"
}
```

| Field    | Type           | Description                                                                |
| -------- | -------------- | -------------------------------------------------------------------------- |
| `type`   | string         | Always `"llm-no-response"`                                                 |
| `reason` | string \| null | Reason for no response; `"abstain"` indicates the model chose not to speak |

**Recommended action:** Update your UI to indicate the bot chose not to respond, or handle the event silently depending on your UX requirements.

***

### Interaction & transcription

#### final-user-transcription

Sent with the finalized transcription of what the user said in the current turn.

```json
{
  "type": "final-user-transcription",
  "text": "Hello, how are you today?",
  "speaker_id": "user_123",
  "speaker_name": "Alice",
  "participant_id": "participant_456"
}
```

| Field            | Type           | Description                         |
| ---------------- | -------------- | ----------------------------------- |
| `type`           | string         | Always `"final-user-transcription"` |
| `text`           | string         | The transcribed text                |
| `speaker_id`     | string \| null | Identifier for the speaker          |
| `speaker_name`   | string \| null | Display name of the speaker         |
| `participant_id` | string \| null | Participant identifier              |

**Recommended action:** Display `text` in a chat UI or append it to the conversation history.

***

#### moderation-response

Sent when content moderation has processed user input.

```json
{
  "type": "moderation-response",
  "result": false,
  "user_input": "the flagged content",
  "reason": "Inappropriate language"
}
```

| Field        | Type           | Description                                             |
| ------------ | -------------- | ------------------------------------------------------- |
| `type`       | string         | Always `"moderation-response"`                          |
| `result`     | boolean        | `true` if content passed moderation; `false` if blocked |
| `user_input` | string         | The input text that was moderated                       |
| `reason`     | string \| null | Reason for blocking; `null` if content passed           |

**Recommended action:** If `result` is `false`, optionally display feedback to the user indicating the input was blocked.

***

#### behavior-tree-response

Sent with behavior tree data for character AI behavior.

```json
{
  "type": "behavior-tree-response",
  "bt_code": "...",
  "bt_constants": "...",
  "narrative_section_id": "section_1"
}
```

| Field                  | Type   | Description                          |
| ---------------------- | ------ | ------------------------------------ |
| `type`                 | string | Always `"behavior-tree-response"`    |
| `bt_code`              | string | Behavior tree code                   |
| `bt_constants`         | string | Constants for the behavior tree      |
| `narrative_section_id` | string | Current narrative section identifier |

***

### Canonical model output

#### model-output

Sent when the client negotiates `capabilities.model_output_version: 2`. This is the typed authority for renderable and executable model output. The legacy `action-response` may also be emitted as a compatibility projection; process one authority, not both.

```json
{
  "type": "model-output",
  "version": 2,
  "output_id": "out_abc123",
  "logical_turn_id": "turn_42",
  "format": "convai-combined-json",
  "raw": "{\"response\":\"I will open it.\",\"actions\":[\"Wave\"]}",
  "items": [
    {
      "type": "message",
      "role": "assistant",
      "channel": "final",
      "content": "I will open it."
    },
    {
      "type": "semantic_action",
      "id": "act_abc123",
      "name": "Wave",
      "target": null
    }
  ],
  "final": true
}
```

| Field             | Type      | Description                                                                                                              |
| ----------------- | --------- | ------------------------------------------------------------------------------------------------------------------------ |
| `type`            | string    | Always `"model-output"`.                                                                                                 |
| `version`         | integer   | Always `2`.                                                                                                              |
| `output_id`       | string    | Envelope identifier. Deduplicate repeated delivery by this field.                                                        |
| `logical_turn_id` | string    | Optional correlation ID shared by output envelopes from one logical turn. A present value is at most `128` UTF-8 bytes.  |
| `format`          | string    | `"text"`, `"convai-combined-json"`, `"semantic-actions-json"`, or `"client-tool-calls-json"`.                            |
| `raw`             | string    | Exact provider or runtime output retained for diagnostics. Never execute or render this field as trusted content.        |
| `items`           | object\[] | Convai-validated semantic items.                                                                                         |
| `final`           | boolean   | Always `true` for this completed envelope. It does not mean that no later envelope can share the same `logical_turn_id`. |

| Item `type`       | Fields                                     | Meaning                                                                                                                                                                   |
| ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `message`         | `role`, `channel`, `content`               | Assistant text for `"final"` or `"commentary"`.                                                                                                                           |
| `semantic_action` | `id`, `name`, `target`                     | Parsed semantic action. `target` may be `null`.                                                                                                                           |
| `tool_call`       | `id`, `name`, `target`, `arguments`        | Correlated client tool call. Return [`action-result`](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#action-result) for its `id`. |
| `emotion`         | `name`, `scale`                            | Turn emotion with scale `1`, `2`, or `3`.                                                                                                                                 |
| `extension`       | `schema`, `version`, `payload`, `fallback` | Schema-versioned extension item. This preview does not define a display or quick-response extension schema.                                                               |

Current producers emit final-channel messages, semantic actions, client tool calls, and emotions. Commentary-channel messages and `extension` items are represented by the candidate protocol and parsers, but the current runtime does not produce them.

Multiple envelopes can share one `logical_turn_id`, such as a text envelope followed by a semantic action or client tool call. Deduplicate only by `output_id`. Convai does not execute or authorize `tool_call` items; validate and execute them in your application before returning a result.

***

### Actions

#### action-response

Sent with an ordered compatibility projection of semantic actions or client tool calls. See the [Connect API](/api-docs/api-reference/core-api-reference/live-apis-beta/connect-api.md) for `action_config` and capability selection.

```json
{
  "type": "action-response",
  "actions": [
    { "name": "Move To", "target": "cube" },
    { "name": "Wave" }
  ]
}
```

| Field              | Type      | Description                              |
| ------------------ | --------- | ---------------------------------------- |
| `type`             | string    | Always `"action-response"`               |
| `actions`          | object\[] | Ordered array of actions to trigger      |
| `actions[].name`   | string    | Action or animation identifier           |
| `actions[].target` | string    | Optional target object or character name |

Legacy semantic actions use `{ name, target? }`. Convai validates the semantic action name and any non-empty target against the session affordances before emission.

Action protocol v2 also projects client tool calls in this shape:

```json
{
  "type": "action-response",
  "actions": [
    {
      "kind": "tool_call",
      "id": "call_abc123",
      "name": "open_training_record",
      "arguments": {
        "record_id": "record-42"
      }
    }
  ]
}
```

| Field       | Type   | Description                                                                              |
| ----------- | ------ | ---------------------------------------------------------------------------------------- |
| `kind`      | string | Always `"tool_call"` for a v2 client tool call.                                          |
| `id`        | string | Correlation ID for the required terminal result.                                         |
| `name`      | string | Declared client tool name.                                                               |
| `target`    | string | Optional compatibility field. Do not treat it as authorization for the tool's arguments. |
| `arguments` | object | JSON object validated against the tool's declared input schema.                          |

Array order is preserved, but Convai does not execute the operations or promise sequential client execution. Apply your own authorization, scheduling, cancellation, and retry policy. If you negotiated model output v2, consume `model-output.items` and ignore the duplicate `action-response` projection.

***

### Animation & lip sync

#### bot-emotion

Sent when the bot expresses an emotion. Use this to trigger avatar animations or update visual feedback elements.

```json
{
  "type": "bot-emotion",
  "emotion": "happy",
  "scale": 2
}
```

| Field     | Type    | Description                                                             |
| --------- | ------- | ----------------------------------------------------------------------- |
| `type`    | string  | Always `"bot-emotion"`                                                  |
| `emotion` | string  | Emotion name, for example `"happy"`, `"sad"`, `"excited"`, or `"angry"` |
| `scale`   | integer | Intensity level: `1` = subtle, `2` = moderate, `3` = intense            |

**Recommended action:** Trigger the corresponding avatar expression or animation for the received `emotion` and `scale`.

***

#### visemes

Sent frequently during bot speech with lip-sync data for avatar mouth animation. Values represent blend weights for each mouth shape.

```json
{
  "type": "visemes",
  "visemes": {
    "sil": 0.0,
    "pp": 0.8,
    "ff": 0.0,
    "th": 0.0,
    "dd": 0.0,
    "kk": 0.0,
    "ch": 0.0,
    "ss": 0.0,
    "nn": 0.0,
    "rr": 0.0,
    "aa": 0.2,
    "e": 0.0,
    "ih": 0.0,
    "oh": 0.0,
    "ou": 0.0
  }
}
```

| Field     | Type   | Description                                       |
| --------- | ------ | ------------------------------------------------- |
| `type`    | string | Always `"visemes"`                                |
| `visemes` | object | Map of viseme keys to blend weights (`0.0`–`1.0`) |

**Viseme keys**

| Key   | Phonemes  |
| ----- | --------- |
| `sil` | Silence   |
| `pp`  | P, B, M   |
| `ff`  | F, V      |
| `th`  | TH        |
| `dd`  | T, D      |
| `kk`  | K, G      |
| `ch`  | CH, J, SH |
| `ss`  | S, Z      |
| `nn`  | N, L      |
| `rr`  | R         |
| `aa`  | A         |
| `e`   | E         |
| `ih`  | I         |
| `oh`  | O         |
| `ou`  | U, W      |

**Recommended action:** Apply the viseme weights to avatar lip-sync blend shapes each time this message arrives.

***

#### neurosync-blendshapes

Sent with a single frame of facial animation blendshape data (251 values per frame).

```json
{
  "type": "neurosync-blendshapes",
  "blendshapes": [0.0, 0.1, 0.05, 0.0]
}
```

| Field         | Type     | Description                                                   |
| ------------- | -------- | ------------------------------------------------------------- |
| `type`        | string   | Always `"neurosync-blendshapes"`                              |
| `blendshapes` | float\[] | Array of 251 blendshape values, each in the range `0.0`–`1.0` |

**Recommended action:** Apply the blendshape values to the avatar facial rig for the current frame.

***

#### chunked-neurosync-blendshapes

Sent with multiple frames of blendshape data batched into a single message. Use this message type instead of `neurosync-blendshapes` when the server sends batched data for efficiency.

```json
{
  "type": "chunked-neurosync-blendshapes",
  "blendshapes": [
    [0.0, 0.1, 0.05],
    [0.02, 0.12, 0.04],
    [0.01, 0.09, 0.06]
  ]
}
```

| Field         | Type        | Description                                                                         |
| ------------- | ----------- | ----------------------------------------------------------------------------------- |
| `type`        | string      | Always `"chunked-neurosync-blendshapes"`                                            |
| `blendshapes` | float\[]\[] | Array of blendshape frames; each frame contains 251 values in the range `0.0`–`1.0` |

**Recommended action:** Queue the frames and apply them sequentially to produce smooth facial animation.

***

#### neurosync-blendshapes-cancel

Sent when an ahead-delivered NeuroSync turn is truncated — by an interruption, a forced turn end, or a superseded output session. Clean completion does **not** emit this message. Only sent to clients that opt in to ahead-delivered chunks.

```json
{
  "type": "neurosync-blendshapes-cancel",
  "response_id": "session-id:r4",
  "neurosync_turn_id": 4,
  "epoch": 1,
  "sequence": 18,
  "valid_through_frame_index": 179,
  "reason": "interruption"
}
```

| Field                       | Type            | Description                                                                                         |
| --------------------------- | --------------- | --------------------------------------------------------------------------------------------------- |
| `response_id`               | string          | Lifecycle response identifier to cancel                                                             |
| `neurosync_turn_id`         | integer         | NeuroSync turn identifier to cancel                                                                 |
| `epoch`                     | integer         | NeuroSync connection/session epoch                                                                  |
| `sequence`                  | integer         | Per-turn message sequence                                                                           |
| `valid_through_frame_index` | integer \| null | Inclusive last frame index backed by released audio. When omitted or `null`, hard-discard the owner |
| `reason`                    | string          | Cancellation reason, for example `"interruption"`                                                   |

**Recommended action:** Retire buffered blendshapes for the owner identified in the message.

* If `valid_through_frame_index` is present, keep frames up to and including that index and drop everything after it.
* If it is omitted or `null`, hard-discard the owner's buffered frames immediately. This happens on active interruptions and completed-owner supersessions, where keeping a visual tail would leave lips moving after the audio has stopped.
* Do **not** apply the cancel to the currently active owner unless it matches the owner in the message.

***

#### blendshape-turn-stats

Sent at the end of a bot turn with statistics about the blendshape generation for that turn. Use this for debugging or analytics.

```json
{
  "type": "blendshape-turn-stats",
  "stats": {
    "total_blendshapes": 150,
    "total_audio_bytes": 48000,
    "total_turn_duration_ms": 3000.0,
    "total_audio_duration_ms": 2800.0,
    "fps": 50.0,
    "was_interrupted": false
  }
}
```

| Field                           | Type    | Description                                          |
| ------------------------------- | ------- | ---------------------------------------------------- |
| `type`                          | string  | Always `"blendshape-turn-stats"`                     |
| `stats.total_blendshapes`       | integer | Total blendshape frames generated in the turn        |
| `stats.total_audio_bytes`       | integer | Total audio data size in bytes                       |
| `stats.total_turn_duration_ms`  | float   | Total turn duration in milliseconds                  |
| `stats.total_audio_duration_ms` | float   | Total audio duration in milliseconds                 |
| `stats.fps`                     | float   | Blendshape frames per second                         |
| `stats.was_interrupted`         | boolean | `true` if the turn was interrupted before completion |

***

### Audio

#### audio-data

Sent when audio is routed through the data channel instead of (or in addition to) the standard WebRTC audio track. This message is only received when `audio_routing` is set to `"data_only"` or `"both"` in the `audio_config` of the `/connect` request.

```json
{
  "type": "audio-data",
  "sample_rate": 48000,
  "channels": 1,
  "audio": "AAEAAg==...",
  "includes_wav_header": false
}
```

| Field                 | Type    | Description                                                                    |
| --------------------- | ------- | ------------------------------------------------------------------------------ |
| `type`                | string  | Always `"audio-data"`                                                          |
| `sample_rate`         | integer | Audio sample rate in Hz, for example `16000`, `24000`, or `48000`              |
| `channels`            | integer | Number of audio channels: `1` = mono, `2` = stereo                             |
| `audio`               | string  | Base64-encoded audio data (raw PCM or WAV with header)                         |
| `includes_wav_header` | boolean | `true` if the audio payload includes a 44-byte WAV header; `false` for raw PCM |

For complete decoding steps, playback implementation, and configuration options see [Audio Data via Data Channel](/api-docs/api-reference/core-api-reference/live-apis-beta/audio-data-via-data-channel.md).

***

### Voice activity detection

These messages come from the VAD-based speech-to-text gating system, which activates the STT service only when speech is detected. Use them to drive listening indicators.

#### vad-stt-started

The STT service has been unmuted and is processing audio after detecting confirmed speech.

```json
{
  "type": "vad-stt-started",
  "timestamp": "2026-08-10T10:30:45.123Z",
  "pre_roll_ms": 1500
}
```

| Field         | Type    | Description                                                                                               |
| ------------- | ------- | --------------------------------------------------------------------------------------------------------- |
| `timestamp`   | string  | ISO 8601 timestamp when STT was unmuted                                                                   |
| `pre_roll_ms` | integer | Amount of audio captured *before* speech detection that was prepended to the stream, so nothing is missed |

**Recommended action:** Show a "listening" or "transcribing" indicator.

***

#### vad-stt-stopped

The STT service has been muted after the hangover period expired.

```json
{
  "type": "vad-stt-stopped",
  "timestamp": "2026-08-10T10:30:48.456Z",
  "reason": "hangover_elapsed",
  "audio_duration_ms": 3200
}
```

| Field               | Type            | Description                                                 |
| ------------------- | --------------- | ----------------------------------------------------------- |
| `timestamp`         | string          | ISO 8601 timestamp when STT was muted                       |
| `reason`            | string          | Why transcription stopped, for example `"hangover_elapsed"` |
| `audio_duration_ms` | integer \| null | Total audio processed in this segment                       |

**Recommended action:** Remove the listening indicator.

***

#### vad-stt-debug

Detailed VAD state-change, speech-detection, and silence-detection events. Only emitted when the session is connected with `debug: true` and debug events are enabled. The payload shape varies by event and is intended for diagnostics rather than application logic.

***

### Diagnostics

These messages are only emitted when the session is connected with `debug: true`.

| Message        | Purpose                                                                                                                                                                                  |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `turn-trace`   | Per-turn timing and stage-transition trace                                                                                                                                               |
| `server-log`   | Server-side log lines surfaced to the client                                                                                                                                             |
| `usage-update` | Per-turn usage and cost information. Streaming can be toggled with [`usage-toggle`](/api-docs/api-reference/core-api-reference/live-apis-beta/client-to-server-messages.md#usage-toggle) |
| `metrics`      | Pipeline performance metrics. See [Metrics](/api-docs/api-reference/core-api-reference/live-apis-beta/metrics.md)                                                                        |

{% hint style="warning" %}
Diagnostic message payloads are not part of the stable API surface and may change without notice. Do not build application logic on them.
{% endhint %}

***

### Related pages

{% content-ref url="/pages/sSGFU83mQqghMh4MsRmp" %}
[Message Glossary](/api-docs/api-reference/core-api-reference/live-apis-beta/message-glossary.md)
{% endcontent-ref %}

{% content-ref url="/pages/KRZRHxk7U8Q3yHwi9YFj" %}
[Connect API](/api-docs/api-reference/core-api-reference/live-apis-beta/connect-api.md)
{% endcontent-ref %}

{% content-ref url="/pages/Zc962pHlSz8gkDgP6twF" %}
[Audio Data (via data channel)](/api-docs/api-reference/core-api-reference/live-apis-beta/audio-data-via-data-channel.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.convai.com/api-docs/api-reference/core-api-reference/live-apis-beta/server-to-client-messages.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.