> For the complete documentation index, see [llms.txt](https://docs.convai.com/api-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md).

# 实时口型同步

## 启用唇形同步

创建一个带有唇形同步配置的客户端：

```typescript
import { ConvaiClient } from '@convai/web-sdk/vanilla';

const client = new ConvaiClient({
  apiKey: 'your-api-key',
  characterId: 'your-character-id',
  enableLipsync: true,          // 启用 blendshape 流式传输
  blendshapeFormat: 'mha',      // 'arkit' 或 'mha'（默认：'mha'）
});

// 连接后开始接收 blendshape
await client.connect();
```

## 配置选项

### ConvaiConfig

```typescript
interface ConvaiConfig {
  // ... 其他选项
  
  /**
   * 启用唇形同步/面部动画 blendshape（默认：false）。
   * 启用后，会以 60fps 流式传输实时 blendshape 数据。
   */
  enableLipsync?: boolean;
  
  /**
   * 要从服务器接收的 blendshape 格式（默认：'mha'）。
   * 'arkit' - 61 个元素（52 个 blendshape + 9 个旋转值）
   * 'mha' - 251 个元素（MetaHuman 格式）
   */
  blendshapeFormat?: 'arkit' | 'mha';
}
```

### 所有选项示例

```typescript
const client = new ConvaiClient({
  apiKey: 'your-api-key',
  characterId: 'your-character-id',
  enableVideo: true,
  enableLipsync: true,
  blendshapeFormat: 'arkit',
  startWithVideoOn: false,
});
```

## 创建唇形同步播放器

实现一个播放器类来处理动画循环：

```typescript
class LipsyncPlayer {
  private client: ConvaiClient;
  private isPlaying: boolean = false;
  private animationFrameId: number | null = null;
  private startTime: number = 0;
  
  constructor(
    client: ConvaiClient, 
    private onFrame: (frame: Float32Array) => void
  ) {
    this.client = client;
    
    // 跟踪机器人开始说话的时间以同步时序
    client.on('speakingChange', (isSpeaking) => {
      if (isSpeaking) {
        this.startTime = performance.now();
      }
    });
  }

  start(): void {
    if (this.isPlaying) return;
    this.isPlaying = true;
    this.animate();
  }

  stop(): void {
    if (!this.isPlaying) return;
    this.isPlaying = false;
    if (this.animationFrameId !== null) {
      cancelAnimationFrame(this.animationFrameId);
      this.animationFrameId = null;
    }
  }

  private animate = (): void => {
    if (!this.isPlaying) return;

    const queue = this.client.blendshapeQueue;
    
    if (queue.hasFrames() && queue.isConversationActive()) {
      // 计算自机器人开始说话以来经过的时间
      const elapsedTime = (performance.now() - this.startTime) / 1000;
      
      // 根据经过时间获取帧（与音频同步）
      const result = queue.getFrameAtTime(elapsedTime);
      
      if (result) {
        this.onFrame(result.frame);
      }
    }

    this.animationFrameId = requestAnimationFrame(this.animate);
  };
}

// 用法
const lipsyncPlayer = new LipsyncPlayer(client, (blendshapes) => {
  applyBlendshapesToCharacter(blendshapes, character.morphTargetInfluences);
});

lipsyncPlayer.start();

// 辅助函数：将 blendshape 映射到你的角色的 morph target
function applyBlendshapesToCharacter(frame: Float32Array, influences: number[]) {
  // 简单直接映射（前 N 个 blendshape 到前 N 个 morph target）
  const maxIndex = Math.min(frame.length, influences.length);
  for (let i = 0; i < maxIndex; i++) {
    influences[i] = frame[i];
  }
  
  // 或者，如果你的角色 morph 排列顺序不同，可使用自定义映射：
  // influences[10] = frame[17]; // 将 jawOpen（ARKit 索引 17）映射到你的下巴 morph（索引 10）
  // influences[15] = frame[18]; // 将 mouthClose（ARKit 索引 18）映射到你的嘴部 morph（索引 15）
}
```

BlendQueue 函数：

```tsx
// 基于时间的访问（实际使用中最重要！）
queue.getFrameAtTime(elapsedSeconds) // 返回 { frame, frameIndex }

// 状态检查
queue.isConversationActive() // 机器人在说话吗？
queue.isConversationEnded()  // 统计信息到达了吗？
queue.isAllFramesConsumed()  // 播放完成了吗？

// 统计信息
queue.getTurnStats()         // TurnStats 对象
queue.getTimeLeftMs()        // 剩余时间（毫秒）
queue.getFramesConsumed()    // 已播放多少帧
queue.getDebugInfo()         // 完整状态快照

// 中断
queue.interrupt()     // 在 sendInterruptMessage() 时自动调用
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.convai.com/api-docs/zh/cha-jian-yu-ji-cheng/web-plugins/convai-web-sdk/vanilla-typescript/real-time-lipsync.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.