For the complete documentation index, see llms.txt. This page is also available as Markdown.

Use your own 3D model

Render a Convai character with a MetaHuman GLB model that you host, including Full and Lite files and your own animation clips.

Render a Convai character with a 3D model you host yourself, instead of the body Convai serves for the character. Use this page when you have your own MetaHuman exported to GLB. At the end, your model loads in place of the default body and talks with lip sync, gaze, and blinking through the character's normal Convai connection.

Prerequisites

What the model needs

Convai Embody drives a body through the names of its bones and blendshapes. The rig it does everything on is convai-mha-1, the Unreal Engine MetaHuman rig; a file with the MetaHuman names exactly as Unreal Engine exports them gets lip sync, blinking, eye movement, gaze and breathing. Other faces and skeletons load with what they have (see Models from other tools below).

Part
Names
What uses it

Head and spine bones

head, neck_01, neck_02, spine_02, spine_04

Gaze and breathing. A missing bone turns off the motion that drives it.

Body bones

The rest of the MetaHuman skeleton, from pelvis and spine_01 through spine_05 to the limbs

Animation clips, which target bones by name.

Face blendshapes

MetaHuman expression morph targets, CTRL_expressions_*

Lip sync, blinking, and eye movement.

Face materials (optional)

Materials named Eyes*, Teeth*, and Eyeocclusion*

Eye and teeth shading. Turn it off with faceMaterials: false.

Draco-compressed files load as they are; the decoder is fetched from dracoDecoderPath.

Models from elsewhere

Models from other tools load and are checked when they do. Convai Embody reads what the model has and reports what it can and cannot do, with the fix for each problem:

  • ARKit faces (Ready Player Me, Apple face capture, most VRM and Character Creator exports) and renamed MetaHuman faces are mapped automatically, so lip sync, blinking and eye movement work. The map is in character.report.face.morphMap; pass it as morphMap to pin it or correct an entry.

  • A model with no blendshapes cannot move its face. The report says so; the body still animates.

  • Animations from Mixamo, the Unreal mannequin and Rigify are retargeted onto the body. Animations inside the model file are used when you declare none. A clip that is missing from its file, will not load, or moves none of the body's bones is reported by name.

  • Gaze and breathing need the MetaHuman head, neck and spine bone names; on other skeletons they are off, and the report says which bones are missing.

Check a model before wiring a conversation to it:

Describe your model with a resolver

Convai Embody learns where a body's files are from a resolver: an object with one resolve method that returns the body's manifest. The default resolver asks Convai. Supply your own to point at your files.

Field
Description

characterId

The character ID you were asked to resolve. Return it unchanged.

avatarId

Any stable identifier for this model. It appears as character.bundle.avatarId.

representations

The model for each rig. List one, with rig: 'convai-mha-1'.

representations[].model

The model files. One entry is enough; see Ship a Full and a Lite file for two.

representations[].clips

Animation clips, name to { url }. Each name must equal an animation's name inside that file. Optional; see Add your own animations.

Render the character with your model

Pass the resolver with the character ID and credential you already use. The model comes from your resolver, and the conversation still comes from the character's Convai connection, so the character keeps its voice, knowledge, and memory.

In React, define the resolver outside the component or wrap it in useMemo. A new resolver object loads the model again, so a resolver written inline in the component reloads the model on every render. Change it only when the model should change.

Keep Convai's animations with your model

To use your model with the animations Convai serves for the character, ask the default resolver for the character's manifest and replace only its model files.

The clips are made for the MetaHuman skeleton, so they fit any model on the same rig.

Ship a Full and a Lite file

Many phones allow 256 vertex-shader uniforms in all, and three.js spends one on each blendshape, so a head with all 251 MetaHuman blendshapes fails to draw on them. A body can ship two files: Full, with every blendshape, and Lite, with a reduced set. List both, with lod 0 for Full and 1 for Lite, and the most blendshapes on any one mesh as morphTargets.

Convai Embody loads Lite on phones and tablets, and on any GPU too small for Full's blendshapes, and Full everywhere else. Keep Lite at 192 blendshapes or fewer, which leaves room for the rest of the shader on a 256-uniform phone. character.model reports which file loaded, and modelVariant: 'full' or 'lite' forces one, for example to check Lite on a desktop.

Preview a model without a conversation

To render a model with no Convai connection, for example in an asset review page, pass connect: false. With a resolver and connect: false, no credential is needed.

The model plays its clips, breathes, blinks, and follows the camera, but it has no voice and no lip sync. character.client is null.

Verify the model

Run the app and send the character a message.

Troubleshooting

Symptom
Cause and fix

no supported representation: character offers [<rigs>], embody supports [convai-mha-1]

The resolver's representation names another rig. Set rig: 'convai-mha-1' for a MetaHuman model.

The console warns rig "convai-mha-1": missing required bone(s) — …

The model lacks the head or spine bones listed in the warning, so gaze or breathing is off for them. Export the full MetaHuman skeleton with its Unreal Engine bone names.

The body loads but the face stays still while the character talks

The model's blendshapes do not have MetaHuman CTRL_expressions_* names. Export the face's expression morph targets from the MetaHuman.

The head is not drawn on some phones

The model has more blendshapes than the phone's GPU allows. Ship a Lite file, as in Ship a Full and a Lite file.

The model is the wrong size or below the floor

The model is scaled to targetHeight (1.7 meters) when it loads, and its soles are placed at groundY. Set either option to change them.

The file never loads, or the Draco decoder fails

Read the browser's network panel. Files on another host need CORS, and a page whose content security policy blocks gstatic.com needs dracoDecoderPath pointed at a self-hosted decoder.

Next steps

Add your own animations to a charactercreateCharacter reference

Last updated

Was this helpful?