LogoLogo
PlaygroundVideosBlogPricingForum
  • Welcome
  • Convai Playground
    • Playground Walkthrough
    • Get Started
    • Character Creator Tool
      • Create Character
      • Language and Speech
      • Knowledge Bank
      • Personality and Style
      • State of Mind
      • Memory
      • Actions
      • Narrative Design
      • Character Versioning
    • F.A.Q
  • Plugins & Integrations
    • Introduction
    • Unity Plugin
      • Pre-Requisites
      • Compatibility
      • Downloads
        • Limitations of WebGL Plugin
      • Setting Up Unity Plugin
      • Troubleshooting Guide
        • Disable Assembly Validation
        • Missing Newtonsoft Json
        • Microphone Permission Issues
        • Default Animations Incompatibility
        • Animations have Facial Blendshapes
        • Jaw Bone in Avatar is not Free
        • macOS Permission Issues
      • Creating a Convai Powered Scene from Template
      • Importing Ready Player Me (RPM) Characters
      • Importing Custom Characters
      • Adding Actions to your Character
      • Adding Lip-Sync to your Character
      • Adding Narrative Design to your Character
        • Narrative Design Keys
      • Adding NPC to NPC Conversation
      • Adding Scene Reference and Point-At Crosshairs
      • Utilities
        • Character Emotion
        • Player Data Container
        • Long Term Memory
        • Language Support
        • Managing sessionID Locally
        • Transcript UI System
        • Pre-built UI Prefabs
        • Input Management
        • Notification System
        • Settings Panel
        • Dynamic Information Context
      • Building For Supported Platforms
        • Building for iOS/iPadOS
        • Building for WebGL
        • Convai XR
          • Building for VR
            • VR Automatic Installation
            • VR Manual Installation
          • Building for MR
            • MR Automatic Installation
            • MR Manual Installation
          • Building for AR
          • Interacting with XR UI Elements
        • Building for macOS Universal apps
      • Changelogs
      • Tutorials
        • Narrative Design
        • NPC2NPC
    • Unreal Engine
      • Supported Platforms
      • Installation
      • Guides
        • Dynamic Environment Info
        • Change AI Character Movement Speed
        • Integration with Pixel Streaming
        • Adjust Interaction Radius
        • Creating MetaHuman Characters
          • Adding MetaHuman
          • Adding LipSync to MetaHuman (From plugin version 3.0.0 or later )
          • Change the Parent Class for MetaHuman
          • Change the parent class for Player.
          • Adding LipSync to MetaHuman (Deprecated)
        • Creating ReadyPlayerMe Characters
          • Download Plugins
          • Adding ReadyPlayerMe Character
          • Change the parent class for Player.
        • Creating Reallusion Characters
          • Creating a Reallusion Character
          • Importing Reallusion character and setting up the Convai plugin
          • Binding objects to Reallusion Character
        • Event-Aware Convai Characters
        • Operations Based on the Presence of Certain Words
        • Narrative Design
          • Narrative Design Trigger
          • Narrative Design Keys
        • Actions Guide
          • Stage 1: Default Actions
            • Moves To
            • Follows
          • Stage 2: Custom Actions
            • Simple actions
            • Adding Descriptions to Actions
          • Stage 3: Custom Actions with Single Parameter
          • Main Character and Attention Object
        • Customization
          • Push to Talk
          • Alter Character Response Audio Rate
        • Speech To Text Transcription
        • Enable Multiplayer Support
        • 3D Chat Widget
        • Long Term Memory
        • Saving and Loading Session
      • Blueprints Reference
        • Convai Player
        • Convai Chatbot
        • Convai Environment
        • Convai Object Entry
        • Convai Result Action
        • Convai Extra Params
        • Speech To Text
        • Text To Speech
        • Utility Functions
      • Troubleshoot Guide
        • Missing Unreal Engine Tool Set in Microsoft Visual Studio Toolchain
        • Convai Module Not Found
        • MetaHuman Plugin Conflict
        • Failure to Load Character IDs
      • Microphone Settings
        • Set Audio Gain
        • Set Microphone
        • Microphone test
        • List Microphone Devices
      • Mac Microphone Permission: Required for UE 5.0 and 5.3
      • Guides V2 (Under Development)
        • Getting Started
          • Installation
          • Simple Talking Cube
          • Adding Premade Chat and Settings UI
    • Web Plugin
      • PlayCanvas Plugin
        • Adding External Script
        • First Person View
        • Adding characters to scene
        • Character Animations
        • Convai Integration
        • Chat Overlay
      • Convai Web SDK
        • Getting Started
        • Facial Expressions
        • Actions
      • JavaScript Chat UI SDK
        • Getting Started
        • ChatBubble Props
      • Narrative Design Guide
        • Narrative Design Triggers
        • Narrative Design Keys
      • GLB Characters for Convai
      • GLB/FBX animations for Convai
    • Modding Framework
      • Modding Cyberpunk 2077
    • Other Integrations
      • Roblox
        • Sample Game 1
        • Sample Game 2
        • Code Example
          • Character Conversation API
      • Discord
        • Create a Discord Bot
        • Hosting Discord Bot from Personal Server
        • Hosting Discord Bot from Replit
      • Omniverse Extension
      • Third-Party API Integrations
        • ElevenLabs API Integration
    • Convai Pixel Streaming Embed
      • Whitelisting & Publishing an Experience
      • React Typescript
      • ReactJS (JavaScript)
      • Vanilla JavaScript (ES Modules)
      • CDN (UMD Script)
      • On Premise Deployment
      • API Reference
  • Reference
    • Core API Reference
      • Character Base API
      • Interaction API
      • Core AI Setting API
      • Backstory API
      • Chat History API
      • Knowledge Bank API
      • Narrative Design API
      • Action API
      • Language List API
      • Voice List API
      • Character List API
      • Evaluation API
Powered by GitBook
On this page
  • Convai's Evaluation API endpoint
  • Headers
  • Request Body
  • Example Payload
  • Response
  • Code Snippet

Was this helpful?

  1. Reference
  2. Core API Reference

Evaluation API

API to evaluate a conversation transcript based on user defined attributes (e.g. clarity of responses, courtesy, listening skills, and more).

This API is accessible only with the Scale and Enterprise plans.

Convai's Evaluation API endpoint

POST https://api.convai.com/character/evaluate_conversation

This API allows you to evaluate a conversation transcript based on user defined attributes e.g. clarity of responses, courtesy, listening skills, and more. The evaluation will be returned in a structured JSON format.

Headers

Name
Type
Description

CONVAI-API-KEY*

String

The unique api-key provided for every account.

Request Body

Name
Type
Description

session_id

String (required)

The ID of the session you want to evaluate.

character_id

String (required)

The ID of the character performing the evaluation.

prompt

String (required)

A predefined or custom prompt containing the transcript and specific instructions for evaluation.

variables

JSON Object (optional)

A set of key-value pairs providing additional data required for evaluation (e.g., customer name, item details). This depends on the prompt being passed.

Example Payload

{
  "session_id": "<SESSION ID>",
  "character_id": "<CHARACTER ID>",
  "prompt": "<EVALUATION PROMPT>",
  "variables": {
    "customer_name": "Suzie Denver",
    "customer_age": "54 years",
    "item_details": [
      {
        "item": "Burger",
        "price": "$4",
        "quantity": 3
      },
      {
        "item": "Fries",
        "price": "$2",
        "quantity": 2
      }
    ]
  }
}

Example prompt

You are an human analyst designated to evaluate employee-customer interactions in a given scenario. Based on the conversation transcript provided below, analyze the interaction and provide an evaluation according to the specified attributes. Each attribute should be rated on a scale of Excellent, Good, Fair, or Needs Improvement. Provide specific examples from the transcript to support your rating. Return the evaluation results in a structured JSON format for easy parsing.
Transcript:
[[conversation_history]]


Here are some data that might be needed to evaluate the conversations:
Actual Order: [[item_details]]
Customer Name: [[customer_name]]
Customer Age: [[customer_age]]


Please evaluate the above conversation based on the attributes listed and provide a Rating and Feedback for each of them. If an attribute is not applicable or cannot be assessed, return null in the json output. Return the result in the following format:


{
  "evaluation": {
    "correctness_of_responses": {
      "rating": "",
      "feedback": "[Did the employee place the actual order correctly based on the customer's request?]"
    },
    "clarity_of_responses": {
      "rating": "]",
      "feedback": "[Did the employee's responses appear clear and easy to understand based on the text provided?]"
    },
    "conciseness_and_relevance": {
      "rating": "",
      "feedback": "[Were the employee’s responses concise and focused on the relevant details, without unnecessary information?]"
    },
    "courtesy_and_respect": {
      "rating": "",
      "feedback": "[Did the employee demonstrate politeness and respect toward the customer in their language and responses?]"
    },
    "listening_skills": {
      "rating": "",
      "feedback": "[Did the employee respond appropriately, indicating they were actively listening and addressing customer concerns?]"
    },
    "product_knowledge": {
      "rating": "",
      "feedback": "[Did the employee demonstrate accurate knowledge of the menu, promotions, or policies?]"
    },
    "problem_solving_and_issue_resolution": {
      "rating": "",
      "feedback": "[Evaluate how well the employee addressed and resolved issues]"
    },
    "response_time": {
      "rating": "",
      "feedback": "[Analyze the flow of the conversation and whether there were any delays in response]"
    },
    "order_accuracy": {
      "rating": "",
      "feedback": "[Evaluate the accuracy of order confirmation, if applicable]"
    },
    "follow_up_and_conversation_closure": {
      "rating": "",
      "feedback": "[Review how the conversation was closed and whether appropriate follow-up occurred]"
    },
    "overall_summary": {
      "overall_rating": "",
      "overall_feedback": "[Provide a brief summary of the employee's performance across all attributes]"
    }
  }
}

If you focus on the prompt, there are certain text within [[ ]] . These are expected-variables. Now, [[conversation_history]] is a compulsory expected-variable, that has to be present in the prompt. The rest of them depends on your requirements, to be passed to the prompt as needed.

So the variables key, in the body of the request, should be of length expected-variables - 1, i.e, there should be values for all the other keys mentioned in the [[ ]] brackets, except for conversation_history which is fetched from the session_id provided. The variables list can be empty if you are passing no other expected-variables in the prompt.

Not passing the conversation_history variable in the prompt will lead to 500 error response from the server.

Response

On success, the API returns a structured evaluation of the conversation, covering multiple attributes.

{
  "evaluation": {
    "clarity_of_responses": {
      "rating": "Excellent",
      "feedback": "The employee provided clear and concise responses throughout the conversation."
    },
    "conciseness_and_relevance": {
      "rating": "Good",
      "feedback": "Responses were relevant but could be more concise."
    },
    "courtesy_and_respect": {
      "rating": "Excellent",
      "feedback": "The employee was polite and respectful."
    },
    "listening_skills": {
      "rating": "Good",
      "feedback": "The employee responded appropriately, though some concerns were addressed after a delay."
    },
    "product_knowledge": {
      "rating": "Excellent",
      "feedback": "The employee demonstrated accurate knowledge of the menu."
    },
    "problem_solving_and_issue_resolution": {
      "rating": "Needs Improvement",
      "feedback": "Issue resolution took longer than expected."
    },
    "response_time": {
      "rating": "Fair",
      "feedback": "There were noticeable delays between responses."
    },
    "order_accuracy": {
      "rating": "Excellent",
      "feedback": "All items were confirmed accurately."
    },
    "follow_up_and_conversation_closure": {
      "rating": "Good",
      "feedback": "The employee followed up appropriately, though the closure could have been smoother."
    },
    "overall_summary": {
      "overall_rating": "Good",
      "overall_feedback": "The employee's performance was generally good, but there is room for improvement in responsiveness."
    }
  }
}

{
    "ERROR": "Invalid API key provided."
}
{
    "ERROR": "Invalid variable list. Please recheck the variables being sent",
    "Reference ID": "<REFERENCE-ID>"
}

Code Snippet

import requests
import json

EVALUATION_URL = "https://api.convai.com/character/evaluate_conversation"

# Headers
headers = {
  'Content-Type': 'application/json',
  'CONVAI-API-KEY': '<YOUR API KEY>'
}

# Payload
payload = {
    "session_id": "<SESSION ID>",
    "character_id": "<CHARACTER ID>",
    "prompt": "<YOUR CUSTOM PROMPT>",
    "variables": {
        "customer_name": "Suzie Denver",
        "customer_age": "54 years",
        "item_details": json.dumps([
            {"item": "Burger", "price": "$4", "quantity": 3},
            {"item": "Fries", "price": "$2", "quantity": 2}
        ])
    }
}

# Make the request
response = requests.post(EVALUATION_URL, headers=headers, json=payload)

# Handle the response
evaluation_response = response.json()

try:
    evaluation = evaluation_response["evaluation"]
    print("Evaluation: ", evaluation)
except KeyError:
    print("Error: ", evaluation_response)
PreviousCharacter List API

Last updated 1 month ago

Was this helpful?