Synapse

A Unity library for building natural AI interactions in games

Overview

Synapse is a comprehensive Unity package that enables developers to create intelligent, voice-interactive NPCs and AI-powered experiences. It provides a unified, provider-agnostic API for speech recognition, text-to-speech, and large language models, making it easy to build immersive conversational AI without being locked into any single service provider.

Key Features

Speech Recognition (ASR)

• ElevenLabs Real-time Streaming - WebSocket-based continuous transcription
• OpenAI Whisper API - Cloud-based speech recognition
• Local Whisper - On-device transcription using whisper.unity
• Two-Phase ASR - Hybrid approach using local VAD to trigger online transcription, reducing API costs
• Voice Activity Detection - Built-in speech detection with configurable thresholds
• Shared Listener System - Multiple consumers can share a single microphone input

Text-to-Speech (TTS)

• ElevenLabs Streaming TTS - High-quality, real-time voice synthesis
• Multi-Voice Support - Connection pooling for concurrent NPC speech
• Viseme Lip Sync - Synchronized phoneme-to-viseme mapping for animations
• Streaming Audio Playback - Real-time PCM audio with low latency
• Voice Provider Abstraction - Runtime voice listing and selection

Large Language Models (LLM)

• OpenAI - GPT-4, GPT-3.5, and compatible models
• Anthropic - Claude models with native API support
• Groq - Ultra-fast LPU inference
• Cerebras - High-performance Llama inference
• Together AI - Open-source model hosting
• Ollama - Local/self-hosted LLM support
• Streaming Responses - Real-time token streaming
• Tool/Function Calling - Structured interactions with game systems
• Conversation Management - Session tracking, templates, and persistence

NPC Framework

• NpcBrain - Central orchestrator combining ASR, LLM, and TTS
• Character Configuration - Personality, background, and speaking style
• Memory System - Fuzzy memory with LLM-powered summarization
• Perception System - Visual and spatial awareness context
• Action System - JSONL response parsing for world manipulation
• Emote System - Body language and expression control
• Activation Mechanisms - Push-to-talk, gaze detection, multi-model triggers
• Built-in Handlers - Quest, inventory, animation, and expression handlers

Architecture Highlights

• Provider-Agnostic Design - Swap service providers without code changes
• ScriptableObject Configuration - Unity-native asset-based setup
• Service Registry - Central management with dependency injection support
• Streaming-First - Real-time processing for responsive interactions
• Fully Testable - Interface abstractions with NSubstitute mocking support
• 950+ Unit Tests - Comprehensive test coverage

Supported Platforms

• All Unity platforms with network access
• Unity 2019.4 LTS or later
• Includes local Whisper support for offline speech recognition

Dependencies

Package	Purpose
UniTask	Async/await support for Unity
NativeWebSocket	Cross-platform WebSocket (including WebGL)
Newtonsoft.Json	JSON serialization
whisper.unity	Local speech recognition (optional)

Quick Start

1. Install via Unity Package Manager
2. Create a service config asset (ASR, TTS, or LLM)
3. Add a ServiceHost component to your scene
4. Configure API credentials and settings
5. Subscribe to events or use UnityEvents in the inspector

// Example: Basic ASR setup
[SerializeField] private ASRServiceHost _asrHost;

void Start()
{
    _asrHost.OnFinalTranscription.AddListener(text =>
        Debug.Log($"Player said: {text}"));
}

public void StartListening() => _asrHost.StartListening();

Use Cases

• Conversational NPCs - Voice-interactive characters with memory and personality
• Voice Commands - Speech-driven game controls and interactions
• AI Companions - Intelligent sidekicks with contextual awareness
• Accessibility - Voice input/output for inclusive game design
• Procedural Dialogue - Dynamic, AI-generated conversations
• Virtual Assistants - In-game help systems with natural language understanding