> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react/real-time-lipsync.md).

# 在 React 中添加口型同步

从 Convai 随角色语音流式传输的 blendshape 帧驱动 React Three Fiber 场景中 3D 角色的面部。

在 `@react-three/fiber` 场景中为角色面部制作动画。Convai Web SDK 会将 blendshape 流与角色语音一起缓冲。若你已使用形变目标渲染角色模型，并希望角色的嘴部动作与语音同步，请参阅本页。

### 前提条件

* 使用 Convai Web SDK 的 React 应用。请参阅 [React 快速入门](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react.md).
* `three` 和 `@react-three/fiber` 均已安装，并且拥有一个网格包含形变目标的 glTF 角色。
* 形变目标的名称需与流中的通道名称一致，或者需要提供名称映射。通道名称列于 [Blendshape 通道顺序参考](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md).

### 启用 blendshape 流

将 `enableLipsync: true` 设为启用，并通过 `blendshapeConfig.format`. `"mha"` 每帧发送 251 个 MetaHuman 通道，也是默认格式； `"arkit"` 每帧发送 61 个 ARKit 通道。

```tsx
const client = useConvaiClient({
  apiKey: 'YOUR_API_KEY',
  characterId: 'YOUR_CHARACTER_ID',
  enableLipsync: true,
  blendshapeConfig: { format: 'arkit' },
});
```

帧会传入 `client.blendshapeQueue`。除非 `enableLipsync` 为 `true`.

### 绑定形变目标

针对每个网格，只需解析一次每个帧通道对应的形变目标。 `ARKIT_ORDER_61` 和 `METAHUMAN_ORDER_251` 按帧中的顺序列出通道名称。

{% code title="src/lipsync.ts" %}

```ts
import type { Mesh } from 'three';
import { ARKIT_ORDER_61 } from '@convai/web-sdk/lipsync-helpers';

export function bindMorphs(mesh: Mesh, channels: readonly string[] = ARKIT_ORDER_61) {
  const slots = new Int32Array(channels.length).fill(-1);
  channels.forEach((name, i) => {
    const slot = mesh.morphTargetDictionary?.[name];
    if (slot !== undefined) slots[i] = slot;
  });
  return slots;
}

export function applyFrame(mesh: Mesh, slots: Int32Array, frame: Float32Array) {
  const influences = mesh.morphTargetInfluences;
  if (!influences) return;
  for (let i = 0; i < slots.length; i++) {
    if (slots[i] >= 0) influences[slots[i]] = frame[i] ?? 0;
  }
}
```

{% endcode %}

如果你的变形目标使用其他名称，请将映射器作为 `blendshapeConfig.customMapper`传入，或者根据自己的名称列表构建槽位表。请参阅 [使用混合变形驱动面部动画](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md).

### 在渲染循环中播放帧

角色说话时，按照播放速率从队列中读取帧；角色停止说话后，继续处理完剩余帧。 `getPlaybackFps()` 返回当前轮次的帧速率，而 `getFrameWithAlpha()` 会淡化前后各 10 帧，避免嘴巴突然张开或定格。

{% code title="src/Face.tsx" %}

```tsx
import { useEffect, useRef } from 'react';
import { useFrame } from '@react-three/fiber';
import type { Mesh } from 'three';
import type { BlendshapeQueue } from '@convai/web-sdk/core';
import { applyFrame, bindMorphs } from './lipsync';

export function Face({ mesh, queue }: { mesh: Mesh; queue: BlendshapeQueue }) {
  const slots = useRef<Int32Array | null>(null);
  const elapsed = useRef(0);

  useEffect(() => {
    slots.current = bindMorphs(mesh);
  }, [mesh]);

  useFrame((_, delta) => {
    if (!slots.current) return;

    // 遇到中断时：立即将面部恢复到中性状态。
    if (queue.consumeNormalizationSignal()) {
      applyFrame(mesh, slots.current, new Float32Array(slots.current.length));
    }

    const playing =
      queue.isBotSpeaking() || (queue.hasReceivedEndSignal() && queue.hasFrames());

    if (!playing) {
      elapsed.current = 0;
      if (queue.isConversationEnded()) queue.reset();
      return;
    }

    elapsed.current += delta;
    const frameTime = 1 / queue.getPlaybackFps();
    let frame: Float32Array | null = null;

    while (elapsed.current >= frameTime && queue.hasFrames()) {
      frame = queue.getFrameWithAlpha(0);
      queue.consumeFrames(1);
      elapsed.current -= frameTime;
    }

    if (frame) applyFrame(mesh, slots.current, frame);
  });

  return null;
}
```

{% endcode %}

渲染 `<Face mesh={headMesh} queue={client.blendshapeQueue} />` 放在 `<Canvas>`中，其中 `headMesh` 是包含面部变形目标的网格。

{% hint style="warning" %}
对于 glTF 角色使用 `format: 'mha'` 时，每帧写入前都要应用 MetaHuman 限制驱动器，否则每次张开下颌时嘴唇都会分离。请参阅 [应用 MetaHuman 限制驱动器](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md).
{% endhint %}

### 验证唇形同步

连接并通过 `ConvaiWidget` 或 `AudioRenderer`播放角色音频，然后发送一条消息。角色回答时， `client.blendshapeQueue.getDebugInfo()` 会报告一个不断增长的 `framesConsumed`.

{% hint style="success" %}
角色说话时，嘴巴会随声音张合，并在回复结束后恢复静止。
{% endhint %}

### 故障排查

#### 面部没有动作

**症状：** 角色会说话，但 `getDebugInfo().frameCount` 始终为 `0`.

**原因：** `客户端连接时使用的配置中，` enableLipsync `不是` true。

**修复方法：** 添加 `enableLipsync: true`，然后断开连接并重新连接，以发送新配置。

**验证：** `frameCount` 会在角色说话时递增。

#### 只有部分变形目标会移动

**症状：** 帧已被消耗，但部分面部区域保持静止。

**原因：** 网格上的变形目标名称与所选格式的通道名称不匹配。

**修复方法：** 将 `mesh.morphTargetDictionary` 与 [Blendshape 通道顺序参考](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md) 进行比较，并映射不同之处。

**验证：** bindMorphs `返回的槽位表` 中没有 `-1` 预期通道对应的条目。

### 后续步骤

{% content-ref url="/pages/29f9aac19fcc2f8cd08338eeb8c9340da2c133cf" %}
[使用 blendshape 驱动面部动画](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape.md)
{% endcontent-ref %}

{% content-ref url="/pages/c52240c5f5c86a00159cb30d9d04c5b6d5ea3c4a" %}
[应用 MetaHuman 限制驱动器](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/lipsync-and-blendshape/metahuman-limit-drivers-required-for-mha.md)
{% endcontent-ref %}

{% content-ref url="/pages/4478f4a3cec9e37c9839b186fe7a71671f76fa3e" %}
[Blendshape 声道顺序参考](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/mappings-reference.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/react/real-time-lipsync.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
