> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/how-the-emotion-system-works.md).

# How the emotion system works

Understand how Emotion resolves the emotion signal in Convai's response, smooths it, and composes the character's face and mood.

`ConvaiEmotionController` turns the emotion signal Convai sends with its response into a smoothed, composited facial expression, and separately tracks the character's longer-lived resting mood. This page explains where that input comes from, how the controller resolves and smooths it, and how the result reaches the character's face.

***

### Where emotion's input comes from

Unlike Gaze, Body Animation, and Body Language, which decide everything from state already local to the scene — the dialogue state, the rig, nearby targets — Emotion's decisions are only partly local. Two things arrive from outside the character on every turn:

* **The transient emotion itself.** Convai emits it as part of its response: an emotion label and an intensity, chosen by whichever detection provider `EmotionDetectionMode` on `ConvaiEmotionController` requests. **Responsive** (`EmotionDetectionMode.Nrclex`, the default) reads the reply as it streams, so the face can change more than once within one reply. **Accurate** (`EmotionDetectionMode.Llm`) reads the finished reply once, arriving later but weighing meaning rather than wording — the better choice for a character speaking any language other than English. **Off** requests no signal at all; a character with no `ConvaiEmotionController` is treated as `Off` by default. This setting lives on the character in Unity — it decides which provider Convai runs, not a Convai console setting.
* **Mood commands.** When Convai's response includes a set-mood or reaction instruction, it reaches this character through `MoodCommandHandlerAdapter` — infrastructure Convai adds alongside `ConvaiEmotionController` automatically, never a component you add or configure yourself. The adapter calls the same `SetMood` and `SetEmotionOverride` methods your own gameplay code can call — see [Moods](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/moods.md).

Everything downstream of these two entry points — resolution, smoothing, blending, micro-expression life, and output — runs locally, the same as every other embodiment module.

***

### How the controller resolves and smooths a signal

An incoming label is resolved through the character's `EmotionTaxonomyAsset` first — canonical labels such as `joy` plus whatever server aliases the vocabulary defines. A label the taxonomy cannot resolve logs one warning and falls back to neutral.

The resolved label and intensity then feed an internal score accumulator that owns two separate, independently-read channels:

| Channel           | Public surface                                         | What it represents                                                                                                                                          |
| ----------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Transient emotion | `CurrentResolvedEmotion`, `CurrentNormalizedIntensity` | The character's reaction to the most recent line. Rises and decays with each incoming signal, and settles back toward the mood below once a reaction fades. |
| Mood              | `CurrentMoodLabel`, `CurrentMoodScore`                 | What the character rests at between reactions — set by its personality's baseline or by `SetMood`, and persistent until something changes it.               |

An active reaction never overwrites the mood, and the mood never appears as the active transient emotion — a character can visibly react with `surprise` while resting on a `joy` mood underneath. See [Moods](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/moods.md) for controlling the second channel from your own code.

Two additional settings shape the transient channel before it reaches the face: an optional short overshoot on arrival (micro-burst) that gives an expression a punchier entry, and optional blending, which lets a primary emotion show alongside related taxonomy complements (`joy` and `trust`, for example) instead of one emotion replacing another outright. Both are authored on `ConvaiEmotionProfile` — see [Emotion profile](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/emotion-profile.md).

***

### How expression reaches the face

Expression recipes name *what should move* in semantic terms — `MouthSmileLeft`, `BrowOuterUpRight`, and the rest — rather than naming blendshapes on one particular mesh. At runtime those semantics resolve against whichever blendshapes the character's own face actually has, through a curated lookup covering ARKit, Reallusion CC3, Reallusion CC4 Extended, and MetaHuman. One profile therefore drives any supported rig with no per-character authoring; a rig matching none of those conventions needs a `CustomRigConventionMap`.

Emotion does not write blendshapes directly. It submits its composed expression, and — when enabled — a continuous micro-expression life layer (idle drift plus a speech-emphasis accent), to the character's shared facial compositor, the same single writer LipSync and every other facial contributor submit to. See [Facial composition](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/facial-composition.md) for how the compositor resolves overlapping claims on the same region, such as the mouth during speech.

Emotion can also drive arbitrary shader float properties — blush, tear glisten, sweat sheen — from composed scores through an optional material property binding, entirely independent of the blendshape path. See [Emotion output bindings](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/output-bindings.md).

***

### What "speaking" means for expression

Prosody coupling and the character's speaking-driven micro-expression accent only apply while the character is actually performing a speaking turn, and that signal comes from [Conversation Flow](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/conversation-flow.md) when the character has it: expression follows the same `Speaking` dialogue state Body Animation's talk layer reads, not a raw speech-started/speech-stopped event. Conversation Flow arbitrates the end of a turn against local evidence a raw event does not see, so keying expression off it keeps the face's speaking bias releasing at the same moment the body's does — see [How a speaking turn ends](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/conversation-flow.md#how-a-speaking-turn-ends).

With no Conversation Flow module present on the character, Emotion falls back to the raw speech-started/speech-stopped event.

***

### Conversation Flow is added automatically when needed

Unlike Gaze and Body Language, none of Emotion's dialogue-driven behavior is on by default — the listening lift, the thinking look, and the two one-shot reaction accents (on entering `Reacting` and `Interrupted`) all ship at zero strength on `ConvaiEmotionProfile`. Raising any of **Listening Reaction Strength**, **Thinking Reaction Strength**, **Reacting Accent Strength**, or **Interrupted Flinch Strength** above `0` asks Convai to add a `ConvaiConversationFlowController` to the character at runtime if none exists yet, the same way Body Animation's **Auto Create Conversation Flow** setting does. The Console logs once, naming the character, so a component you did not add is never a silent surprise. See [Conversation flow](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/conversation-flow.md#convai-adds-it-automatically-when-needed).

***

### EmotionDimensions: the cross-module signal

Alongside its categorical label, every resolved emotion also carries continuous `EmotionDimensions` — `Valence`, `Arousal`, `Agency`, and `Approach`, each clamped to `[-1, 1]`. Categorical labels stay authoritative for authored facial recipes; the dimensions give Gaze, Body Language, and Body Animation one coherent modulation signal to read, instead of each module needing to know Emotion's taxonomy.

***

### Key concepts

| Concept                     | What it is                                                                                                                                                                                    |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ConvaiEmotionController`   | The `MonoBehaviour` that owns the entire pipeline for one character. Add one per character.                                                                                                   |
| `ConvaiEmotionProfile`      | The `ScriptableObject` asset holding every tunable parameter: smoothing, micro-burst, blending, mood, and micro-expression life.                                                              |
| `EmotionTaxonomyAsset`      | The `ScriptableObject` defining the emotion vocabulary — canonical labels, server aliases, and complements. The built-in default is Plutchik's nine emotions including neutral.               |
| `MoodCommandHandlerAdapter` | Hidden infrastructure Convai adds alongside `ConvaiEmotionController` so a mood or reaction instruction in Convai's response can reach this character. Never authored directly.               |
| `EmotionReading`            | An immutable snapshot of the current state: dominant label and score, all scores, mouth influence, and the mood label and score. Available every frame via `ConvaiEmotionController.Current`. |
| `ConvaiCharacterEventRelay` | An Inspector-friendly component that exposes emotion and mood change callbacks as Unity Events — no code required.                                                                            |

***

### Component placement

| Component                   | Where to place it                                                                        | Notes                                                                                                                                         |
| --------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `ConvaiEmotionController`   | On the character's root `GameObject`, alongside the character's other embodiment modules | One per character                                                                                                                             |
| `ConvaiEmotionProfile`      | Anywhere in your `Assets/` folder as a `ScriptableObject` asset                          | Shared across multiple characters if needed; Convai copies a package-shipped profile for you the first time you edit it on a shared character |
| `EmotionTaxonomyAsset`      | Anywhere in your `Assets/` folder                                                        | Optional — omit to use the built-in Plutchik set                                                                                              |
| `MoodCommandHandlerAdapter` | Added automatically alongside `ConvaiEmotionController`                                  | Never add or remove this yourself                                                                                                             |
| `ConvaiCharacterEventRelay` | On any `GameObject` in the scene                                                         | Auto-resolves `ConvaiCharacter` on the same `GameObject`; drag a different character if needed                                                |

***

### Next steps

{% content-ref url="/pages/PyOgkztYoFXIQ0l3uTfN" %}
[Emotion quick start](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/quick-start.md)
{% endcontent-ref %}

{% content-ref url="/pages/s7IkM1rkC8ni25ijDX3u" %}
[Moods](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/moods.md)
{% endcontent-ref %}

{% content-ref url="/pages/4kPK7vkbjtuDa9qfc75O" %}
[Emotion profile](/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/emotion-profile.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/embodiment/emotion/how-the-emotion-system-works.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.