> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react/real-time-lipsync.md).

# 在 React 中添加口型同步

从 Convai 随角色语音流式传输的 blendshape 帧驱动 React Three Fiber 场景中 3D 角色的面部。

在一个角色的面部制作动画 `@react-three/fiber` 来自 Convai Web SDK 会与角色声音一起缓冲的 blendshape 流中的场景。使用此页面时，你已经渲染了一个带有变形目标的角色模型，并希望它的嘴巴跟随其语音。

### 前提条件

* 使用 Convai Web SDK 的 React 应用。参见 [开始使用 React](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react.md).
* `three` 和 `@react-three/fiber` 已安装，并且有一个 glTF 角色，其网格具有变形目标。
* 与流通道同名的变形目标，或者到这些目标的映射。通道名称列在 [Blendshape 通道顺序参考](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md).

### 启用 blendshape 流

设置 `enableLipsync: true` 并使用以下设置选择帧格式： `blendshapeConfig.format`. `"mha"` 每帧发送 251 个 MetaHuman 通道，并且这是默认值； `"arkit"` 发送 61 个 ARKit 通道。

```tsx
const client = useConvaiClient({
  apiKey: 'YOUR_API_KEY',
  characterId: 'YOUR_CHARACTER_ID',
  enableLipsync: true,
  blendshapeConfig: { format: 'arkit' },
});
```

帧会进入 `client.blendshapeQueue`。除非设置了 `enableLipsync` 为 `true`.

### 绑定变形目标

为每个网格解析一次：每个帧通道要写入哪个变形目标。 `ARKIT_ORDER_61` 和 `METAHUMAN_ORDER_251` 按帧顺序列出通道名称。

{% code title="src/lipsync.ts" %}

```ts
import type { Mesh } from 'three';
import { ARKIT_ORDER_61 } from '@convai/web-sdk/lipsync-helpers';

export function bindMorphs(mesh: Mesh, channels: readonly string[] = ARKIT_ORDER_61) {
  const slots = new Int32Array(channels.length).fill(-1);
  channels.forEach((name, i) => {
    const slot = mesh.morphTargetDictionary?.[name];
    if (slot !== undefined) slots[i] = slot;
  });
  return slots;
}

export function applyFrame(mesh: Mesh, slots: Int32Array, frame: Float32Array) {
  const influences = mesh.morphTargetInfluences;
  if (!influences) return;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = frame[i] ?? 0;
  }
}
```

{% endcode %}

如果你的变形目标使用其他名称，请将映射器作为 `blendshapeConfig.customMapper`传入，或者从你自己的名称列表构建插槽表。参见 [使用 blendshape 驱动面部动画](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md).

### 在渲染循环中播放帧

在角色说话时，按播放速率从队列中消费帧，并在它停止后清空剩余帧。 `getPlaybackFps()` 返回当前回合的帧率，且 `getFrameWithAlpha()` 会淡化前 10 帧和后 10 帧，这样嘴巴就不会突然张开或冻结。

{% code title="src/Face.tsx" %}

```tsx
import { useEffect, useRef } from 'react';
import { useFrame } from '@react-three/fiber';
import type { Mesh } from 'three';
import type { BlendshapeQueue } from '@convai/web-sdk/core';
import { applyFrame, bindMorphs } from './lipsync';

export function Face({ mesh, queue }: { mesh: Mesh; queue: BlendshapeQueue }) {
  const slots = useRef<Int32Array | null>(null);
  const elapsed = useRef(0);

  useEffect(() => {
    slots.current = bindMorphs(mesh);
  }, [mesh]);

  useFrame((_, delta) => {
    if (!slots.current) return;

    // 发生中断：立即让脸部恢复为中性。
    if (queue.consumeNormalizationSignal()) {
      applyFrame(mesh, slots.current, new Float32Array(slots.current.length));
    }

    const playing =
      queue.isBotSpeaking() || (queue.hasReceivedEndSignal() && queue.hasFrames());

    if (!playing) {
      elapsed.current = 0;
      if (queue.isConversationEnded()) queue.reset();
      return;
    }

    elapsed.current += delta;
    const frameTime = 1 / queue.getPlaybackFps();
    let frame: Float32Array | null = null;

    while (elapsed.current >= frameTime && queue.hasFrames()) {
      frame = queue.getFrameWithAlpha(0);
      queue.consumeFrames(1);
      elapsed.current -= frameTime;
    }

    if (frame) applyFrame(mesh, slots.current, frame);
  });

  return null;
}
```

{% endcode %}

渲染 `<Face mesh={headMesh} queue={client.blendshapeQueue} />` 在你的 `<Canvas>`中，其中 `headMesh` 是承载面部变形目标的网格。

{% hint style="warning" %}
对于 `format: 'mha'` 的 glTF 角色，在写入每帧之前，为每帧应用 MetaHuman 限制驱动，否则每次下颌张开时嘴唇都会分离。参见 [应用 MetaHuman 限制驱动](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md).
{% endhint %}

### 验证口型同步

连接，并通过 `ConvaiWidget` 或 `AudioRenderer`播放角色的音频，并发送一条消息。当角色回答时， `client.blendshapeQueue.getDebugInfo()` 报告一个不断增长的 `framesConsumed`.

{% hint style="success" %}
嘴巴会随着角色的声音开合，并在回复结束后恢复静止。
{% endhint %}

### 故障排查

#### 脸部没有移动

**症状：** 角色在说话，但 `getDebugInfo().frameCount` 保持为 `0`.

**原因：** `enableLipsync` 不在 `true` 客户端连接时使用的配置中。

**修复：** 添加 `enableLipsync: true`，然后断开并重新连接，以便发送新的配置。

**验证：** `frameCount` 在角色说话时上升。

#### 只有部分变形在动

**症状：** 帧已被消费，但脸部的某些部分仍然不动。

**原因：** 网格上的变形目标名称与所选格式的通道名称不匹配。

**修复：** 比较 `mesh.morphTargetDictionary` 与 [Blendshape 通道顺序参考](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md) 并映射这些差异。

**验证：** 由 `bindMorphs` 返回的插槽表没有 `-1` 你期望的通道条目。

### 下一步

{% content-ref url="/pages/29f9aac19fcc2f8cd08338eeb8c9340da2c133cf" %}
[使用 blendshape 驱动面部动画](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md)
{% endcontent-ref %}

{% content-ref url="/pages/c52240c5f5c86a00159cb30d9d04c5b6d5ea3c4a" %}
[应用 MetaHuman 限制驱动器](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md)
{% endcontent-ref %}

{% content-ref url="/pages/4478f4a3cec9e37c9839b186fe7a71671f76fa3e" %}
[Blendshape 声道顺序参考](/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react/real-time-lipsync.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.