> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md).

# 在原生 TypeScript 中添加口型同步

在原生 TypeScript 中，将 Convai Web SDK 随角色语音流式传输的面部动画帧播放到 3D 模型上，无需 UI 框架。

使用 Convai 随角色语音一同流式传输的混合形状帧，驱动 three.js 角色的变形目标。本页适用于使用不依赖框架的 TypeScript 自行渲染角色的场景。完成后，角色的嘴部会与语音同步动作，并在语音停止或被打断时恢复到中性状态。

### 前提条件

* 已连接的 `ConvaiClient`。请参阅 [使用原生 TypeScript 入门](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript.md).
* 一个 three.js 网格，其变形目标的名称与所请求格式的通道名称一致。请参阅 [混合形状通道顺序参考](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md).

### 启用口型同步

设置 `enableLipsync: true` 并在 `blendshapeConfig.format`: `中选择帧格式` 'mha' `（251 个 MetaHuman 通道，默认格式）或` 'arkit'

```typescript
import { ConvaiClient } from '@convai/web-sdk/vanilla';

const client = new ConvaiClient({
  apiKey: 'YOUR_API_KEY',
  characterId: 'YOUR_CHARACTER_ID',
  enableLipsync: true,
  blendshapeConfig: { format: 'arkit' },
});

await client.connect();
```

如果不设置 `enableLipsync: true`，Convai 不会发送任何帧，队列也会一直为空。帧会分块到达，SDK 会将其缓存在 `client.blendshapeQueue`.

### 绑定变形目标

模型加载后，只需构建一次从帧索引到变形目标槽位的查找表。帧的顺序为： `ARKIT_ORDER_61` （适用于 `'arkit'` ）或 `METAHUMAN_ORDER_251` （适用于 `（适用于`.

```typescript
import * as THREE from 'three';
import { ARKIT_ORDER_61 } from '@convai/web-sdk/lipsync-helpers';

function bindMorphs(mesh: THREE.Mesh, channelNames: readonly string[]): Int32Array {
  const dictionary = mesh.morphTargetDictionary ?? {};
  const slots = new Int32Array(channelNames.length).fill(-1);
  channelNames.forEach((name, i) => {
    // ARKit 通道使用 PascalCase 命名；许多导出的变形目标则使用 camelCase 命名。
    const camel = name.charAt(0).toLowerCase() + name.slice(1);
    const slot = dictionary[name] ?? dictionary[camel];
    if (slot !== undefined) slots[i] = slot;
  });
  return slots;
}

const slots = bindMorphs(faceMesh, ARKIT_ORDER_61);

function applyFrame(frame: Float32Array) {
  const influences = faceMesh.morphTargetInfluences!;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = frame[i];
  }
}

function clearFace() {
  const influences = faceMesh.morphTargetInfluences!;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = 0;
  }
}
```

最后九个 `'arkit'` 这些值是头部和眼睛的旋转角度，而不是变形目标。找不到对应的槽位，因此会被跳过。

### 播放帧

运行一个 `requestAnimationFrame` 循环，在角色说话时推进时钟，播放该时间对应的帧，并消费已经播放过的帧。使用 `getFrameWithAlpha`读取帧；该函数会在一轮开始时的前 10 帧内渐显面部，并在最后 10 帧内渐隐。

```typescript
const queue = client.blendshapeQueue;

let playing = false;
let lastTime = 0;
let elapsedMs = 0;
let lastFrameIndex = -1;

function tick(now: number) {
  // 遇到中断或响应被取消时：将面部恢复为中性状态。
  if (queue.consumeNormalizationSignal()) {
    clearFace();
    playing = false;
  }

  const speaking = queue.isBotSpeaking();
  const draining = queue.hasReceivedEndSignal() && queue.hasFrames();

  if (speaking || draining) {
    if (!playing) {
      playing = true;
      lastTime = now;
      elapsedMs = 0;
      lastFrameIndex = -1;
    }
    elapsedMs += now - lastTime;
    lastTime = now;

    const frameDurationMs = 1000 / queue.getPlaybackFps();
    const targetIndex = Math.floor(elapsedMs / frameDurationMs);
    const advance = Math.min(targetIndex - lastFrameIndex, queue.length);

    if (advance > 0) {
      const frame = queue.getFrameWithAlpha(advance - 1);
      if (frame) applyFrame(frame);
      queue.consumeFrames(advance);
      lastFrameIndex = targetIndex;
    }
  }

  // 本轮结束且所有帧均已播放：为下一轮重置。
  if (queue.isConversationEnded()) {
    clearFace();
    queue.reset();
    playing = false;
  }

  requestAnimationFrame(tick);
}

requestAnimationFrame(tick);
```

循环不会将 `getFrameAtTime()` 与 `consumeFrames()`. `getFrameAtTime()` 搭配使用：索引从队列开头计算，因此在推进播放后再消费帧会导致播放推进两次。两种方法选其一，不要同时使用。

客户端本身会在你发送消息、触发触发器或用户开始说话时，在队列中开始新一轮；而当你调用 `sendInterruptMessage()`时，则会清空队列。你的循环只需读取并消费帧。

### 使用映射器重新映射通道

若要在帧进入队列时将其转换为你自己的绑定布局，请设置一个映射器。映射器接收每个原始帧，并返回一个 `Float32Array`。将其作为 `blendshapeConfig.customMapper`，或者在运行中的队列上设置映射器：

```typescript
client.blendshapeQueue.setMapper((input) => {
  const out = new Float32Array(52); // 你的面部绑定通道数
  out[0] = input[17]; // 将 ARKit 的 JawOpen 映射到面部绑定的索引 0
  return out;
});

client.blendshapeQueue.clearMapper(); // 恢复原始帧
```

设置映射器后，帧将不再采用 `ARKIT_ORDER_61` 或 `METAHUMAN_ORDER_251`的顺序，因此应根据映射器的输出布局来绑定各个索引。

### 验证口型同步

发送一条消息并听取回复。角色说话时嘴部会动，并在说完最后一个词后恢复到中性状态。在回复过程中调用 `sendInterruptMessage()` ：面部会立即恢复，而不是停留在最后一个口型。调试时，如需查看队列快照，请记录 `queue.getDebugInfo()`.

### 后续步骤

{% content-ref url="/pages/29f9aac19fcc2f8cd08338eeb8c9340da2c133cf" %}
[使用 blendshape 驱动面部动画](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md)
{% endcontent-ref %}

{% content-ref url="/pages/c52240c5f5c86a00159cb30d9d04c5b6d5ea3c4a" %}
[应用 MetaHuman 限制驱动器](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md)
{% endcontent-ref %}

{% content-ref url="/pages/f1fc7386520b3ebeeafa719a8d9efb554e1d7024" %}
[ConvaiClient 参考](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/convaiclient-core-api.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
