This guide will show you how to make Mods that add Convai's features to Cyberpunk 2077
video link
This guide will show you how to make Mods that add Convai's features to any pre-existing or already released game, regardless of which engine it is using as long as it has a modding framework capable of creating logic based mods and is able to call scripts or executables or at least write to text files. We will use Cyberpunk 2077 and Cyber Engine Tweaks it's modding framework as an example. The mod we'll create will let players talk to the characters through their microphone or keyboard and the character can even select and perform actions. Lipsync is a little tricky and depends on how the game implements it. If you're working on a mod for an Unreal Engine game, check out our guide for the Hogwarts Legacy mod on Nexus Mods, where we explain lip sync in more detail. We'll be use the following tools: Convai SDK(C++) : This repository provides a set of libraries to access the Convai Character GetResponse API via gRPC streaming. Cyber Engine Tweaks : Cyber Engine Tweaks is a framework giving modders a way to script mods using Lua with access to all the internal scripting features, it also comes with a UI to configure the different mods you are using or using the console directly.
Before you begin, you'll need to set up the Convai SDK. If you haven't done so already, please follow the steps outlined here. Cyber Engine Tweaks can found at nexusmods.
To grasp the Mod creation process, let's walk through the steps I took for the Cyberpunk 2077 Mod. You can find the code I used in the Source Code folder of the mod. We'll begin with "main.cc," which holds the logic for "main.exe." Essentially, "main.cc" captures either the player's audio from microphone or text input from keyboard depending on the value in user_method_of_input.txt which is set by the lua mod, main.cc looks in the "get_response_config_example.txt" file for details like character ID, API key, available actions, and more which is again set by the lua mod. It then sends this information to Convai. After that, it plays the audio of the resulting dialogue of the character the player is interacting with and saves both the dialogue and action to a text file. This text file is later used to display subtitles and choose the appropriate in-game action. Additionally you can even get lipsync viseme data from the convai sdk for accurate lipsync. We use Cyber Engine Tweaks to create a Lua mod, which waits for the player to press F6 or F7 or whichever key the player selected as the hotkey for the mod. When it is pressed, the Lua mod finds the character the player is looking at, extracts information like the character's name, and uses it to find the character ID from a dictionary defined in the Lua file. This information is then written to the "get_response_config_example.txt" file. Next, we would use os.execute to run "main.exe" but since Cyber Engine Tweaks does not support os.execute so we need to look at a text file called control.txt and waits for it to have "start" and when it does run the main logic, the player will need to double click main.exe before launching the game. You can take a look at main.cc in source code folder to get a better understanding of what's going on. The execution completes after playing the dialogue. We parse the dialogue text and action value to display as subtitles. Using string matching we check which action was selected and execute the correct if block for that action. Reviewing the code in "init.lua", "main.cc" will provide a clearer understanding.
A special shout-out goes to the modding community at Cyberpunk 2077. Without their invaluable assistance, tools, guides, and documentation, none of this would have been achievable. Much appreciation for their indispensable contributions!
Convai introduces a Modding Framework, enabling easy integration of AI-driven NPCs into games across engines like Unreal, Unity, and more.
video link
This guide will show you how to make Mods that add Convai's features to any pre-existing or already released Unreal Engine game, from version 4.12 to 5.2. We'll be use the following tools:
Convai SDK(C++): This repository provides a set of libraries to access the Convai Character GetResponse API via gRPC streaming.
UE4SS Lua scripting system platform: C++ Modding API, SDK generator, blueprint mod loader, live property editor and other dumping utilities for UE4/5 games.
Let's get started by setting up the Convai SDK. First, go to the repository's readme and follow the instructions for installing Bazel and building the project. If you encounter a "roots.pem" error, go to the file "convai-sdk-cpp\bazel-convai-sdk-cpp\external\com_github_grpc_grpc\src\core\lib\security\security_connector\ssl_utils.cc" and find line 60. Change the path to only "roots.pem". Then, copy the "roots.pem" file from "convai-sdk-cpp/bazel-convai-sdk-cpp/external/com_github_grpc_grpc/etc/" and paste it next to the "main.exe" file in the "convai-sdk-cpp\bazel-bin" directory or whichever directory you're using to execute "main.exe" from. When packaging your mod, ensure that you include the "roots.pem" file with "main.exe". If you've installed the mod, UE4SS should already be installed. To make blueprint Mods, you will need to install the unreal engine version that was used to make that game. For example Hogwarts Legacy used UE4.27 so we need to use that version for making the Blueprint Mod. Locate the exe file of the game and right click on it, select properties and go to details tab and you will find the version.
To grasp the Mod creation process, let's walk through the steps I took for the Hogwarts Legacy Mod. You can find the code I used in the repository. We'll begin with "main.cc," which holds the logic for "main.exe." Essentially, "main.cc" captures the player's audio, looks in the "get_response_config_example.txt" file for details like character ID, API key, available actions, and more. It then sends this information to Convai. After that, it plays the audio of the resulting dialogue of the character the player is interacting with and saves both the dialogue and action to a text file. This text file is later used to display subtitles and choose the appropriate in-game action. Additionally you can even get lipsync viseme data from the convai sdk for accurate lipsync. Once we have the "main.exe," we execute it only when the player interacts with a character. We use UE4SS to create a Lua mod, which waits for the player to press F7. When F7 is pressed, the Lua mod finds the nearest character of the class "NPC_Character", extracts information like the character's name, and uses it to find the character ID from a dictionary defined in the Lua file. This information is then written to the "get_response_config_example.txt" file. Next, we use os.execute to run "main.exe." The execution completes after playing the dialogue. We parse the dialogue text and action value to display as subtitles. For the action value, we save it to a global variable. In the blueprint Mod, we create an empty function named "actionfunction" in the ModActor with one output pin. In the Lua mod, we create a custom event with the same name, setting the output value to the action value. When the blueprint mod calls the empty function, it instead triggers the custom event defined in the Lua mod, providing the action value. The blueprint mod will then use this to trigger the right action. To add Lipsync, Create a function in your blueprint mod that prints the curve names, this will give a list of blendshapes you can move. Check which blendshapes are related to lips. Use the setmorphtarget node to create a function in blueprint mod to move a blendshape by a given value. Once you have a list of blendshape names and a function that takes a blendshape name and value and moves it by that value, the next step is to make a function in your lua mod that essentially uses a couple of equations to figure out how much to move a given blendshape by and then calls the function you made above for each blendshapes that is needed for Lipsync. To create lua mods with UE4SS, Go to this folder Phoenix\Binaries\Win64\Mods and then make a folder with your mod name and then open Phoenix\Binaries\Win64\Mods\mods.txt and enable your mod. Make a Scripts\main.lua file which will house the logic of your lua mod. To create a blueprint mod with UE4SS, open a blank project with no content using unreal engine which is the same version as the game you are modding. The name of the blank project should be the same as the one in your game folder, for example Hogwarts Legacy has a folder called phoenix so I named the blank level phoenix. In project settings/packaging enable "Use Pak File" and "Generate Chunks" and any other setting that is required for the game you are modding. For example Hogwarts Legacy uses Io Store so I had to enable that setting as well. Then in editor preference/experimental enable "Allow ChunkID Assignments". Next create a blank level with any name, followed by creating a Mods folder and then a folder with your mod's name and within it create a actor with name ModActor. Drag this ModActor into the level. Open the ModActor and make a couple of blueprint functions which you can later call from your lua mod. Right click on the Level you made and go to asset actions/assign to chunk and select a number, repeat the same for ModActor and any other asset with the same number. Go to file/package project/build configuration and select shipping and then click on Windows to package your mod. In the folder you chose for packaging the mod go to WindowsNoEditor/{project_name}/Content/Paks and rename the three files with the number you selected above to match the name of the folder you made in Mods folder in unreal engine. To call functions from the blueprint mod in our Lua mod, we use "FindFirstOf" to locate the ModActor we created above and then call the required function.Reviewing the code in "main.lua", "main.cc", "convailevel.umap" and "ModActor.uasset" will provide a clearer understanding.
A special shout-out goes to the modding community at UE4SS and Hogwarts Legacy. Without their invaluable assistance, tools, guides, and documentation, none of this would have been achievable. Much appreciation for their indispensable contributions!