How scene metadata works

Understand how scene object metadata reaches Convai at connect time, through live updates, and via tracked property changes.

Scene metadata is how you give a Convai character knowledge of the objects in your level. Add a UConvaiObjectComponent to any Actor — a door, a machine, a piece of equipment — and every character in the level will know that object exists, what it is called, and what it does. When a player asks "What's in this room?" or "Is the north gate open?", the character uses this information to answer.

How tagged objects reach Convai

When the session starts, the plugin collects all tagged objects and sends their names and descriptions to Convai in one batch. Objects you add or remove while the game is running are synced through a separate update message.

Internally, every UConvaiObjectComponent registers itself with the UConvaiSubsystem global pool at BeginPlay. When a chatbot calls StartSession(), UConvaiChatbotComponent gathers registered object components from the subsystem, builds the action_config payload, and sends it to Convai as part of the /connect handshake. Objects added after the session is live travel through a separate update-scene-metadata message instead.

Objects that were present at session start live in a frozen snapshot and cannot be updated mid-session without restarting the session. New object entries added after session start are sent through update-scene-metadata on the next debounce flush, unless the mutation call uses bFlushImmediately = true.

Context channels at a glance

The plugin gives Convai characters three channels of world context, each with a different purpose:

	Scene metadata	Tracked properties	SetObjectInAttention
What it describes	Static Actors and props in the scene	Live state of an Actor UPROPERTY or function	The specific object the character is focused on right now
When it's sent	Once, at session start (frozen snapshot) or through runtime environment updates for new entries	At session start, then on detected value changes	On an explicit call or gaze pipeline trigger
Typical use	"A locked door at the main entrance"	"The door is now unlocked"	"The character notices the fire extinguisher"

Use all three together for the most context-rich experience.

The Convai object component

UConvaiObjectComponent is an ActorComponent you attach to any Actor you want Convai characters to know about. One component on the Actor is enough; every chatbot in the level sees it automatically without per-chatbot configuration.

At BeginPlay the component joins the UConvaiSubsystem pool. Every chatbot that starts a session after that point can include the Actor in its environment.

Object identity

Each UConvaiObjectComponent carries an ObjectEntry (FConvaiObjectEntry) that describes the Actor to Convai:

• Name — the label chatbots use when referring to this object. It must be unique per level. If two components share a name, UConvaiSubsystem renames the duplicate automatically and writes a warning to the Output Log.
• Description — a plain-language sentence or paragraph Convai receives at session start.

Convai receives the object's name in the name field of its action_config.objects entry. Keeping names short and descriptive (for example "FrontDoor" rather than "BP_Door_01") makes references easier to resolve.

Tracked properties

Beyond static identity, you can attach live-state watchers to supported UPROPERTY values or supported pure, parameterless functions using TrackedProperties (TArray<FConvaiTrackedProperty>).

Each FConvaiTrackedProperty holds:

• PropertyPath — a dotted path to the UPROPERTY or function, resolved at runtime against the owning Actor (for example "bIsLocked", "Stats.HP", "GetCurrentRoomName").
• Description — what the value means in plain language.
• StateValueDescriptions — optional per-value annotations, most useful for enums and bools.
• ShouldRespond (EC_RunLLMOption) — what happens when the value changes at runtime.

At session start the current value of every tracked property is seeded into the chatbot's context with EC_RunLLMOption::Never, so the character is informed without speaking. On each detected change, the plugin sends a dynamic context state update with the configured ShouldRespond: Auto lets Convai decide whether to react, Always requests a response, and Never updates silently.

The key the chatbot sees for a tracked property is "<ObjectName>.<PropertyPath>" — for example "FrontDoor.bIsLocked".

Where distance and direction come from

Scene metadata registers an object's identity — its name and description. It does not compute where that object sits relative to a chatbot; that job belongs to the spatial awareness subsystem, which runs across every registered UConvaiObjectComponent, other character, and the player, and delivers a distance-and-direction sentence to each chatbot every poll.

If you have earlier notes or Blueprints describing a per-object bAutoGenerateProximityState flag or a synthesized "<ObjectName>.ProximityToYou" state key, that mechanism was retired in 4.0.0-beta.22. Spatial awareness replaced it project-wide — see How spatial awareness works for the current model, and Spatial awareness settings reference for every setting.

The Convai Object Component still ships a debug helper for this pipeline: bDebugDrawProximityPaths draws the navigation path from each subscribed chatbot to the object in the editor viewport — green when reachable, red when not, cyan when the chatbot is already there. It is not gated on any other setting and works whenever spatial awareness needs a nav path to the object. Disable it before shipping.

Movement state

A UConvaiObjectComponent can also detect whether the Actor it is attached to is moving. MovementAwareness (FConvaiObjectMovementSettings) samples the object's translation and rotation each poll and folds a moving/stopped verdict into the spatial-awareness sentence — a moving crate reads as moving toward or away from the chatbot without any further configuration.

Turning on bExposeMovementState (Blueprint display name Add Movement State) additionally publishes a durable "<ObjectName>.Movement" state that a chatbot can be asked about directly, and makes a confirmed stop explicit in the description. Two response settings decide whether a transition should also prompt the character to react:

• StartedMovingResponse (When Movement Starts) — how the chatbot responds when the object begins moving.
• StoppedMovingResponse (When Movement Stops) — how the chatbot responds when the object comes to a confirmed stop.

Both use EC_RunLLMOption: Never (the default) updates the state silently, Auto lets Convai decide whether to react, and Always requests a response on every transition. See Scene metadata component reference for the full field table.

What gets sent at session start vs during gameplay

The session handshake delivers a frozen snapshot of the object list (action_config.objects). The bEnableActions bool on EnvironmentData controls the connect-time action configuration: when it is false, the action_config block is not sent at session start, and attention resolution through SetObjectInAttention has no effect.

Once a session is live, object and character mutation methods on UConvaiChatbotComponent update the local environment mirror and schedule a sync. New scene entries and removals are sent through update-scene-metadata; existing entries that were already present in the connect-time snapshot keep their original action_config data until reconnect.

One important constraint applies: if an object was already included in action_config at session start, its description in the frozen snapshot cannot be changed mid-session. Only objects that are new to the session travel through the live update-scene-metadata lane. To propagate a description change to an existing object, call StopSession then StartSession to reconnect with the updated description.

Batching updates

When you call multiple add or remove methods in quick succession, the plugin coalesces them into a single network message rather than sending one message per call. This is the default behavior. Passing bFlushImmediately = true skips the batch window and sends immediately. Use bFlushImmediately sparingly — high-frequency calls produce many small WebRTC messages and increase network overhead.

Next steps

Scene metadata quick startScene metadata component referenceSpatial awarenessManaging the environment at runtime