Build interactive worlds that keep evolving with multimodal LLMs.
Understand a world, simulate how it changes, and interact with what happens next.
Quick Start · Documentation · Architecture · Roadmap · Contributing
简体中文 · ⭐ Star the repository to follow its progress.
Note
Early development — APIs and stored data may change.
Demo: Current prototype demo.
github-demo1.mp4
Infinite World combines multimodal understanding, world simulation, and interaction in one continuous loop:
- Understand. Read text, images, video, audio, and external events.
- Simulate. Turn those inputs into world state, rules, and possible next actions.
- Interact. Let people, agents, and events influence what happens next.
- Continue. Record scenes, choices, state changes, and branches as the world runs.
Faster model inference is making interactive, continuously changing worlds more practical.
Infinite World connects multimodal understanding, world state, and interaction so people and agents can shape what happens next.
Most multimodal generation still produces isolated pieces of content. Infinite World connects understanding, simulation, and interaction in a world that keeps running.
A continuously running world can support new forms of games and interactive content, with people and agents shaping the same evolving environment. The same system may also help embodied agents understand context, simulate possible outcomes, and decide what to do next.
Infinite World is open source so developers, creators, and researchers can build it together: add models, inputs, interactions, and outputs; create new worlds; and share what they learn.
| Direction | Progress |
|---|---|
| Text interaction | Use text input and choices to influence the world state and upcoming scenes. In development. |
| Voice interaction | Use browser voice input for the same world interaction. In development. |
| Continuous worlds | Preserve world state and context across scenes. Continuous generation is being improved. |
| Local preview | Configure worlds, run them locally, and inspect history and metrics. Prototype available. |
| Live | Interactive live streaming is in development. |
| More interactions | Chat, audio, mouse, and keyboard input are planned. |
- Node.js 20+
- pnpm 10+
- FFmpeg (used for voice transcription and live output)
- whisper.cpp CLI (
whisper-cli)
Clone the repository and install the workspace dependencies:
git clone https://github.com/BuzzPlay/infinite-world.git
cd infinite-world
pnpm installInstall whisper.cpp and FFmpeg for your platform. For example, on macOS:
brew install whisper-cpp ffmpeg
On Linux, install or build the whisper.cpp CLI and FFmpeg with your distribution's package
manager. On Windows, download or build the Windows CLI and add its directory to PATH.
The API build prepares the ggml-base Whisper model automatically and stores it outside the
repository. The first build or first voice request needs an Internet connection. Set
INFINITE_WORLD_WHISPER_BIN is optional when whisper-cli is already on PATH. Set it only when
the executable has a custom name or location. INFINITE_WORLD_WHISPER_MODEL is also optional; set
it only when using a custom model path.
Start the API and Web application in separate terminals:
pnpm dev:api
pnpm dev:web
Open http://localhost:5173.
Prompt, image, and interaction
|
v
World understanding
|
v
Persistent state and context ----> Scene generation ----> Preview / output
^ |
| v
+---------------- History and feedback
The runtime separates seven concepts:
| Concept | Meaning |
|---|---|
| World | A persistent setting with its own rules, context, and state. |
| Scene | A generated moment produced from the current world state. |
| Interaction | A choice, event, or input that may change the world. |
| Branch | A possible continuation created by an option or event. |
| History | The recorded states, scenes, choices, and paths of a world. |
| Output | A way to observe or share a run, such as a preview or RTMP stream. |
| Preview | A local surface for testing and shaping a world before sharing it. |
See the architecture documentation for runtime boundaries, lifecycle, and API contracts.
These clips show the interaction directions. Current implementation status is listed below.
| Direction | Concept video | Status |
|---|---|---|
| Text and image | 1.mp4 |
In development |
| Sound | 2.mp4 |
Planned |
| Mouse and keyboard | 3.mp4 |
Planned |
| Live output | 4.mp4 |
In development |
| Multimodal interaction | 5.mp4 |
Planned |
- How can generated scenes retain identity, state, and causal continuity over time?
- How should text, media, people, and external events update a shared world state?
- Which boundaries let models, renderers, inputs, and outputs remain replaceable?
- How can persistent worlds be evaluated while keeping generated records under user control?
- What new game and interaction formats become possible when a generated world keeps running?
- Create interactive worlds shaped by text-based choices. (In development)
- Run worlds continuously with local preview and live output. (In development)
- Connect more live platforms.
- Add more interaction inputs.
- Explore new interaction formats and game forms.
See the Product Roadmap for detailed stages and acceptance criteria.
Ways to contribute:
- reproducible world examples and generation presets;
- provider, input, preview, and output adapters;
- continuity evaluation and failure cases;
- documentation, platform testing, and interaction experiments.
Read CONTRIBUTING.md before opening a pull request. Changes that affect project boundaries should also follow the Agent Guide.
Questions and experiments are welcome in GitHub Discussions. Bugs and scoped proposals can be opened as GitHub Issues.
Infinite World is licensed under the Apache License 2.0.