A 3D AI companion library with voice interaction, memory, tools, and customizable avatars. Build interactive AI agents with real-time voice conversations, persistent memory, and beautiful 3D visualizations.
- π 3D Avatars: Multiple renderer types including BlobXyz, Crystal Ball, Orbital Shards, Stars Genesis, and Black Hole
- π€ Voice Pipeline: Real-time voice interaction with STT, LLM, and TTS via LiveKit
- π§ Memory: Long-term memory support with Zep integration for context-aware conversations
- π οΈ Tools: MCP (Model Context Protocol) integration for external capabilities
- π¨ Soul: Customizable personality templates with emotional traits
- β‘ Skills: Native behaviors and capabilities
- π Dynamic Updates: Update configuration on-the-fly without reconnecting
- π¦ TypeScript: Fully typed with comprehensive type definitions
npm install kwami # or pnpm add kwami # or yarn add kwami
Kwami requires three (Three.js) as a peer dependency:
npm install three
import { Kwami } from 'kwami' // Get a canvas element const canvas = document.getElementById('canvas') as HTMLCanvasElement // Create a Kwami instance const kwami = new Kwami(canvas, { soul: { name: 'Luna', personality: 'friendly and creative', }, agent: { voice: { llm: { model: 'gpt-4o' }, tts: { voice: 'nova' }, }, }, avatar: { renderer: 'blob-xyz', }, }) // Connect and start conversation await kwami.connect('user-123', { onUserTranscript: (text) => console.log('User:', text), onAgentResponse: (text) => console.log('Agent:', text), onStateChange: (state) => console.log('State:', state), }) // Send a message kwami.sendMessage('Hello!') // Cleanup when done await kwami.disconnect()
The visual representation of your AI companion. Multiple renderer types are available:
- BlobXyz: Animated 3D blob with customizable skins (donut, poles, vintage)
- Crystal Ball: Mystical crystal visualization
- Orbital Shards: Dynamic shard formations orbiting a core
- Stars Genesis: Starfield background with animated elements
- Black Hole: Minimalist black hole visualization
Handles the voice pipeline and AI processing:
- Voice Pipeline: STT (Speech-to-Text), LLM (Language Model), TTS (Text-to-Speech)
- LiveKit Integration: Real-time voice communication
- Dynamic Configuration: Update voice settings without reconnecting
Defines the AI's personality and behavior:
- Customizable traits and emotional characteristics
- Pre-built templates (friendly, professional, creative, etc.)
- System prompts and conversation style
- Emotional state tracking
Long-term memory for context-aware conversations:
- Zep integration for persistent memory
- Message history and context retrieval
- Semantic search capabilities
External capabilities via MCP (Model Context Protocol):
- Register custom tools
- Execute external functions
- Dynamic tool registration/unregistration
Native behaviors and capabilities:
- Built-in skill system
- Custom skill definitions
- Context-aware execution
const kwami = new Kwami(canvas, { // Avatar configuration avatar: { renderer: 'blob-xyz', // or 'crystal-ball', 'orbital-shards', etc. scene: { background: 'stars', camera: { position: [0, 0, 5] }, }, }, // Agent configuration agent: { voice: { llm: { provider: 'openai', model: 'gpt-4o', }, tts: { provider: 'openai', voice: 'nova', }, stt: { provider: 'deepgram', }, }, }, // Soul configuration soul: { name: 'Luna', personality: 'friendly and creative', traits: ['curious', 'helpful'], }, // Memory configuration memory: { adapter: 'zep', // Zep configuration... }, // Tools configuration tools: { mcp: { // MCP server configuration... }, }, })
Update configuration on-the-fly:
// Update voice settings kwami.updateVoice({ tts: { voice: 'alloy' }, }) // Update soul kwami.updateSoul({ emotionalTone: 'enthusiastic', }) // Register a new tool kwami.registerTool({ name: 'getWeather', description: 'Get current weather', // ... })
- Node.js 18+
- pnpm (recommended) or npm/yarn
pnpm install
# Build the library pnpm build # Watch mode (TypeScript) pnpm dev # Type checking pnpm typecheck # Run tests pnpm test # Run tests once pnpm test:run # Lint code pnpm lint # Fix linting issues pnpm lint:fix # Format code pnpm format # Check formatting pnpm format:check # Clean build artifacts pnpm clean
kwami/
βββ src/
β βββ agent/ # Voice pipeline and LiveKit integration
β βββ avatar/ # 3D avatar renderers
β βββ memory/ # Memory adapters (Zep)
β βββ soul/ # Personality system
β βββ skills/ # Native behaviors
β βββ tools/ # Tool registry (MCP)
β βββ types/ # TypeScript definitions
β βββ utils/ # Utilities
β βββ Kwami.ts # Main class
β βββ index.ts # Public API exports
βββ scripts/ # Build scripts
βββ dist/ # Build output
βββ package.json
The library exports a comprehensive set of types and utilities. See the TypeScript definitions for full API documentation:
Kwami- Main classAvatar,Scene,StarField- Avatar componentsAgent,LiveKitAdapter,VoiceSession- Agent componentsSoul- Personality managementMemory- Memory operationsToolRegistry- Tool managementSkillManager- Skill execution
Apache-2.0