> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md).

# 在原生 TypeScript 中添加口型同步

在原生 TypeScript 中，将 Convai Web SDK 随角色语音流式传输的面部动画帧播放到 3D 模型上，无需 UI 框架。

使用 Convai 与角色语音一起流式传输的混合形状帧来驱动 three.js 角色的变形目标。当您在无框架的 TypeScript 中自行渲染角色时，请使用此页面。最终，角色的嘴巴会随着语音同步移动，并在停止或被中断时恢复到中性状态。

### 先决条件

* 已连接的 `ConvaiClient`时发送。参见 [从原生 TypeScript 开始](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript.md).
* 一个 three.js 网格体，其变形目标的命名与您请求的格式通道一致。参见 [Blendshape 通道顺序参考](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md).

### 启用唇形同步

设置 `enableLipsync: true` 并在其中选择帧格式 `blendshapeConfig.format`: `'mha'` （251 个 MetaHuman 通道，默认值）或 `'arkit'` （61 个值）。

```typescript
import { ConvaiClient } from '@convai/web-sdk/vanilla';

const client = new ConvaiClient({
  apiKey: 'YOUR_API_KEY',
  characterId: 'YOUR_CHARACTER_ID',
  enableLipsync: true,
  blendshapeConfig: { format: 'arkit' },
});

await client.connect();
```

如果没有 `enableLipsync: true`，Convai 不会发送任何帧，队列会保持为空。帧会分块到达，SDK 会将它们缓冲在 `client.blendshapeQueue`.

### 绑定变形目标

模型加载后，只需构建一次从帧索引到变形目标槽位的查找表。帧顺序为 `ARKIT_ORDER_61` 适用于 `'arkit'` 并且 `METAHUMAN_ORDER_251` 适用于 `'mha'`.

```typescript
import * as THREE from 'three';
import { ARKIT_ORDER_61 } from '@convai/web-sdk/lipsync-helpers';

function bindMorphs(mesh: THREE.Mesh, channelNames: readonly string[]): Int32Array {
  const dictionary = mesh.morphTargetDictionary ?? {};
  const slots = new Int32Array(channelNames.length).fill(-1);
  channelNames.forEach((name, i) => {
    // ARKit 通道使用 PascalCase；许多导出会将变形目标命名为 camelCase。
    const camel = name.charAt(0).toLowerCase() + name.slice(1);
    const slot = dictionary[name] ?? dictionary[camel];
    if (slot !== undefined) slots[i] = slot;
  });
  return slots;
}

const slots = bindMorphs(faceMesh, ARKIT_ORDER_61);

function applyFrame(frame: Float32Array) {
  const influences = faceMesh.morphTargetInfluences!;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = frame[i];
  }
}

function clearFace() {
  const influences = faceMesh.morphTargetInfluences!;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = 0;
  }
}
```

最后九个 `'arkit'` 值是头部和眼睛的旋转，不是变形目标。它们找不到槽位，因此会被跳过。

### 播放这些帧

运行一个 `requestAnimationFrame` 在角色说话时推进时钟、播放该时间点的帧并消耗已经过的帧的循环。使用 `getFrameWithAlpha`，它会在一次轮次的前 10 帧淡入面部，并在最后 10 帧淡出。

```typescript
const queue = client.blendshapeQueue;

let playing = false;
let lastTime = 0;
let elapsedMs = 0;
let lastFrameIndex = -1;

function tick(now: number) {
  // 中断或取消的回复：将面部恢复为中性。
  if (queue.consumeNormalizationSignal()) {
    clearFace();
    playing = false;
  }

  const speaking = queue.isBotSpeaking();
  const draining = queue.hasReceivedEndSignal() && queue.hasFrames();

  if (speaking || draining) {
    if (!playing) {
      playing = true;
      lastTime = now;
      elapsedMs = 0;
      lastFrameIndex = -1;
    }
    elapsedMs += now - lastTime;
    lastTime = now;

    const frameDurationMs = 1000 / queue.getPlaybackFps();
    const targetIndex = Math.floor(elapsedMs / frameDurationMs);
    const advance = Math.min(targetIndex - lastFrameIndex, queue.length);

    if (advance > 0) {
      const frame = queue.getFrameWithAlpha(advance - 1);
      if (frame) applyFrame(frame);
      queue.consumeFrames(advance);
      lastFrameIndex = targetIndex;
    }
  }

  // 轮次结束且每一帧都已播放：为下一轮重置。
  if (queue.isConversationEnded()) {
    clearFace();
    queue.reset();
    playing = false;
  }

  requestAnimationFrame(tick);
}

requestAnimationFrame(tick);
```

循环永远不要同时使用 `getFrameAtTime()` 与 `consumeFrames()`. `getFrameAtTime()` 从队列前端索引，因此在它推进播放后再消费帧等于推进了两次。请二选一，不要同时使用。

当您发送消息、触发触发器或用户开始说话时，客户端本身会在队列中启动每一轮，并在您调用 `sendInterruptMessage()`时将其清空。您的循环只负责读取和消费。

### 使用映射器重新映射通道

要在帧进入队列时将其转换为您骨骼绑定自身的布局，请设置一个映射器。映射器会接收每个原始帧并返回一个 `Float32Array`。将其作为 `blendshapeConfig.customMapper`传入，或将其设置到一个活动队列上：

```typescript
client.blendshapeQueue.setMapper((input) => {
  const out = new Float32Array(52); // 你的骨骼绑定的通道数量
  out[0] = input[17]; // 将 ARKit JawOpen 映射到你的骨骼绑定的槽位 0
  return out;
});

client.blendshapeQueue.clearMapper(); // 恢复为原始帧
```

设置映射器后，帧不再遵循 `ARKIT_ORDER_61` 或 `METAHUMAN_ORDER_251`，因此请改为将槽位绑定到映射器的输出布局。

### 验证唇形同步

发送一条消息并监听回复。角色说话时嘴巴会动，并在最后一个词之后恢复到中性。调用 `sendInterruptMessage()` 在回复中途：面部会立即恢复平静，而不会卡在最后一个形状上。调试时若要获取队列快照，请记录 `queue.getDebugInfo()`.

### 后续步骤

{% content-ref url="/pages/29f9aac19fcc2f8cd08338eeb8c9340da2c133cf" %}
[使用 blendshape 驱动面部动画](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md)
{% endcontent-ref %}

{% content-ref url="/pages/c52240c5f5c86a00159cb30d9d04c5b6d5ea3c4a" %}
[应用 MetaHuman 限制驱动器](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md)
{% endcontent-ref %}

{% content-ref url="/pages/f1fc7386520b3ebeeafa719a8d9efb554e1d7024" %}
[ConvaiClient 参考](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/convaiclient-core-api.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.