Unreal Engine 5 NVIDIA Llama 2 OpenAI Whisper
Players solve real coding challenges delivered as narrative emails from AI-driven NPCs, converse naturally via voice/text chat, and navigate a dynamic tech workplace.
| System | Tech Stack | What It Does |
|---|---|---|
| Task Pipeline | LeetCode API β Llama 2 (7B/13B) | Pulls coding problems & rewrites them as in-game emails from CEO/manager NPCs |
| NPC Brains | UE5 Behavior Trees + Blackboard | Randomized daily schedules (Work/Eat/Rest/Meetings) with priority interrupts based on time/energy/task load |
| Voice Chat | Whisper.cpp β F5-TTS | Player speaks β transcribed β sent to LLM β NPC replies via realistic TTS with 3D spatial audio |
| Face Animation | NVIDIA Audio2Face β LiveLink | TTS audio drives MetaHuman blendshapes in real-time for cinematic lip-sync & expressions |
| World Sim | UE5 NavMesh + Simulated Clock | NPCs autonomously navigate office spaces based on dynamic behavior trees |
π Text or Voice Input
β
π£οΈ Whisper.cpp transcribes speech β text
β
π§ Game Engine bundles context:
{player_msg + npc_state + task_progress + relationship}
β
π€ Local LLM generates narrative NPC response
β
π F5-TTS converts text β MP3 audio stream
β
π NVIDIA Audio2Face generates blendshape animation from audio
β
π€ UE5 + MetaHuman plays 3D audio + live facial animation via LiveLink
| Component | Minimum | Recommended |
|---|---|---|
| GPU | GTX 1660 | RTX 3060+ (for real-time Audio2Face) |
| RAM | 16 GB | 32 GB |
| Storage | 20 GB free | 50 GB SSD |
| OS | Win 10 / Ubuntu 22.04+ | Win 11 / Ubuntu 24.04 |
| Engine | Unreal Engine 5.6+ | Unreal Engine 5.6+ |
| Priority | Feature | Description |
|---|---|---|
| β | NPC-to-NPC Interaction | NPCs converse with each other, share task updates, and react to player progress organically |
| β | Working Collaboration Workflows | Players and NPCs jointly debug code, review PRs, and solve multi-stage challenges in shared virtual workspaces |
- Unreal Engine. Epic Games, Inc. [Online] https://www.unrealengine.com
- MetaHuman. Epic Games, Inc. [Online] https://www.unrealengine.com/metahuman
- Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
- Radford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning.
- NVIDIA Audio2Face. NVIDIA Corporation. [Online] https://developer.nvidia.com/audio2face
- LeetCode. LeetCode. [Online] https://leetcode.com
Β© 2026 Nicholas Wilson Kurniawan