sandai-core

Embed a conversational 3D AI avatar in any website; on-device, privacy-first, and framework-agnostic.

See what it's able to do in a custom implementation from DIE KAVALLERIE 🐴

Live playground: sandai.org

sandai-core is the JavaScript / TypeScript SDK for dropping a fully interactive 3D AI avatar into any web page or web app. The avatar is a real-time AI avatar. A browser-based digital human, mascot, animal, you name it, that can speak, listen, think, and emote. It renders inside a heavily optimized runtime in an <iframe>, so a single embed gives you a complete conversational AI avatar: on-device text-to-speech, speech recognition, an on-device LLM, emotion inference, and lip-sync — with no backend to run and nothing to install.

By default, everything runs locally in the browser via WebAssembly and WebGL/WebGPU. No API keys, no per-message costs, and no user audio ever leaves the device. You can:

• use the local defaults and run on-device models for LLM, ASR + VAD, TTS, emotion inference, and much more
• slot in your own LLM, TTS, ASR, etc.
• mix and match according to your use-case

sandai-core is the framework-agnostic entry point of the stack. It builds on ai-vrm-chat → ai-character → @davidcks/r3f-vrm (Three.js / React Three Fiber). If you're in React, use sandai-react, which re-exports everything here plus hooks and components.

Features

• One-line embed: drop in an <iframe> and drive it with a typed SandaiClient. No 3D engine, no game engine, no local GPU server, no API gateway.
• 3D VRM avatars: load any .vrm model and get a fully animated talking avatar with auto-blink, gaze, idle motion, camera focus, and everything else that turns a static 3D Model unto a talking, vibrant AI-Enabled Avatar. You can bring your own character, use a bundled one, or pick from over 100.000 Models at sandai.org.
• On-device LLM: a local LLM (Gemma, Qwen, and others via transformers.js / ONNX and MediaPipe LiteRT-LM) generates replies in-browser. Or plug in your own LLM and just stream text in.
• Text-to-speech, 100+ voices, 30+ languages: built-in multilingual TTS powered by Piper. Or supply your own audio with plug and play sentence splitting already taken care for you.
• Real-time lip-sync: accurate lip-sync driven from phonemes or directly from the audio waveform, with coarticulation so the mouth never looks robotic.
• Speech recognition (ASR) + VAD: the avatar can listen and reply on its own: on-device Whisper / Moonshine speech-to-text with Silero voice-activity detection.
• Emotion inference & expression: text is mapped to 28 GoEmotions-style emotions that automatically drive facial expressions and graded body animations.
• Webcam puppeteering / VTubing mocap (experimental): mirror your own face and pose onto the avatar live via MediaPipe + IK.
• Typed RPC control: a fully typed postMessage bridge lets the parent page call any method on the character by dot-path, with full argument and return-type autocomplete.
• WebGL & WebGPU rendering, with an optional experimental GPU path-tracer for cinematic, ray-traced scenes.
• Privacy-first, no backend: conversation, voice, and inference happen on the user's device. Great for GDPR-sensitive products, offline-capable apps, and keeping inference costs at zero.
• Framework-agnostic: plain TypeScript core that works with React, Vue, Svelte, Angular, or no framework at all.

Use cases

AI agents and website assistants, customer support and onboarding, AI companions and VTubers, interactive characters for games and education, AI mascorts and brand mascots and virtual presenters, and any product that benefits from a face you can talk to.

Table of Contents

• sandai-core
  • Features
  • Use cases
  • Table of Contents
  • Installation
  • Quick Start
  • SandaiClient
    • Constructor
    • init()
    • destroy()
  • Managers
    • LoadManager
    • AuthManager
    • InteractionManager
    • RPCManager
  • Configuring the iframe URL
  • Voices
  • The postMessage protocol
  • Types
  • Development
  • License

Installation

npm install sandai-core
Copy

Quick Start

Put an iframe on your page pointing at sandai.org, then construct a SandaiClient against its id:

<iframe
  id="sandai-iframe"
  src="https://sandai.org/chat?vrmUrl=https://cdn.example/ada.vrm&voiceName=ruri"
  allow="microphone https://sandai.org; cross-origin-isolated"
  style="width: 100%; height: 600px; border: 0;"
></iframe>
Copy

import { SandaiClient } from "sandai-core";

const client = new SandaiClient("sandai-iframe", userId, apiKey); //userId and apiKey are optional, though you might see a watermark if you don't provide them. 
// Use skipAuth: true in the debug settings for development.
// 
// Consider subscibing at https://sandai.org/subscribe
// It's $5.99 (USD) and you can pay with visa/mastercard and solana
// 
// You can also reach us at info@sandai.org or +49 160 1439797
// if you have any questions.

// Wait for the iframe to load and authenticate.
const { loadState, authState } = await client.init();

// Once loaded, drive the character:
const { speechEndPromise } = await client.interactionManager.say("Hello!");
await speechEndPromise; // resolves when the character finishes speaking

// Or have its on-device LLM reply:
const reply = await client.interactionManager.respond("What's your name?");

// Clean up listeners when you're done:
client.destroy();
Copy

Audio can't play until the user has interacted with the iframe (browser autoplay policy). LoadManager tracks this for you. See hasInteractedPromise below. If the character won't speak, that's usually why.

SandaiClient

The entry point. Wraps a target iframe and exposes the four managers as public properties: interactionManager, loadManager, authManager, rpcManager.

Constructor

new SandaiClient(
  iframeId: string,
  userId?: string,
  apiKey?: string,
  debugOptions?: { skipAuth?: boolean; skipLoadCheck?: boolean },
)
Copy

• iframeId: the DOM id of the target <iframe>. Throws if the element doesn't exist or isn't an iframe.
• userId / apiKey: credentials passed to AuthManager. Both default to "anon" (Free tier) unless you skip Auth.
• debugOptions.skipAuth: skip the auth handshake. Convenient locally, but against the Sandai TOS for live apps.
• debugOptions.skipLoadCheck: skip load/readiness tracking. Also disables the interaction confirmation gate, which in turn makes it so you don't know if audio playback is enabled. Sometimes some things go wrong during the load step, sometimes somewhere else. Usually it's related to the model. This lets you debug if your model is the culprit.

init()

await client.init(): Promise<{ loadState, authState }>
Copy

Runs the full startup sequence. It first calls destroy() to clear any prior listeners, then:

1. Load: waits for the iframe to report ready (via LoadManager).
2. Auth: sends credentials and waits for the access tier (via AuthManager).

It also transparently handles the CORS fallback: if loading fails with loadError, init() fetches the vrmUrl (read from the iframe's own query string) from the parent context, hands the bytes to the character over RPC (arrayBufferProvider.urlify → updateVrmUrl), and retries the load. This is a-ok and intended default behaviour, but introduces a small bit of latency, maybe 5-50ms depending on the client device.

destroy()

Removes all registered message listeners (its own plus the load and auth managers'). Call it on teardown, or just call init() again — it destroys first.

Managers

All four are constructed for you by SandaiClient and reachable as properties.

LoadManager

Tracks the iframe's load lifecycle and whether the user has interacted with it yet.

await client.loadManager.init(): Promise<{
  ready: boolean;
  state: "loading" | "ready" | "loadError" | "timeout";
  hasInteracted: boolean;
  hasInteractedPromise: Promise<boolean>;
}>
Copy

• Resolves { ready: true, state: "ready" } once the iframe reports readiness.
• Resolves { ready: false, state: "loadError" } if the character failed to load (e.g. a bad vrmUrl). SandaiClient.init() uses this to trigger the CORS fallback.
• Resolves { state: "timeout" } after 20s of silence — rare, usually means something is wrong with the embed.
• hasInteractedPromise resolves once the user interacts with the iframe. Audio playback requires this, so await it before expecting say/respond to be audible. It rejects on loadError/timeout (the interaction state can't be determined then).

public loadState reflects the latest state. The iframe re-broadcasts its load state every 5 seconds, so a late-attached client still catches up.

AuthManager

Sends credentials to the iframe and receives the user's access tier.

client.authManager.tier; // string, e.g. "Free" | "Plus" | "Pro" (default "Free")
await client.authManager.init(): Promise<"initialized" | "timeout">
Copy

• Sends { type: "auth", userId, apiKey } and waits for the iframe's { type: "auth", tier } reply, storing it on tier.
• Retries with backoff up to 6 attempts (timeout grows 1000 * attempt ms); falls back to the Free tier on exhaustion.
• "anon" credentials resolve to the Free tier. The tier is enforced on the iframe/server side (via Supabase); sandai-core just relays it.

InteractionManager

The high-level speech API.

// Speak text. Emotion + lip-sync are inferred by the character.
say(message: string, options?): Promise<{ speechEndPromise: Promise<void> }>

// Ask the character's on-device LLM for a reply, then speak it. Resolves with the text.
respond(message: string, context?: string, options?): Promise<string>

// Interrupt the current utterance.
stop(): Promise<void>
Copy

say resolves as soon as speech starts; await the returned speechEndPromise to know when it ends. Shared options:

Option	Type	Notes
voiceName	string	A Piper voice key (see Voices).
useAudioBasedLipSync	boolean	Drive lip-sync from the audio instead of the text phonemes.
emotionInferrenceType	"sentiment" | "distilbert"	distilbert is more accurate but downloads a model; sentiment needs no download.
emotionIntensityMultiplier	number	Scales the inferred facial-expression intensity.

say additionally accepts audioFile (string | ArrayBuffer | Blob | File) to play your own audio instead of the built-in TTS. String URLs are fetched, Blobs/Files are read, and the resulting ArrayBuffer is transferred (not copied) to the iframe. respond additionally accepts modelProvider ({ model, dtype, pipeline: "text" | "auto" | "mediapipe" }) to choose the local LLM for that call.

RPCManager

A typed escape hatch into the character. Anything the in-iframe AICharacterManager (and the managers below it) exposes is callable by dot-path, fully typed with argument and return types.

// Call a method on the loaded character. Types come from ai-vrm-chat/index.rpc.
await client.rpcManager.call("setEmotion", "joy", 0.8);
await client.rpcManager.call("vrmManager.focusManager.focus");

// Call a method that's valid even when the character FAILED to load
// (types come from ai-vrm-chat/index-reduced.rpc — the Misc surface).
const blobUrl = await client.rpcManager.postLoadErrorCall(
  "arrayBufferProvider.urlify",
  arrayBuffer,
);
Copy

call<P>(path, ...args) is typed against AllRpcMethods; postLoadErrorCall<P>(path, ...args) against the reduced (Misc) surface for error states. Transferable arguments (ArrayBuffer, typed arrays, OffscreenCanvas, ReadableStream, etc.) are detected and transferred automatically. AllRpcMethods is re-exported from this package for convenience:

import type { AllRpcMethods } from "sandai-core";
Copy

Configuring the iframe URL

The character configures itself from the iframe's query string (vrmUrl, voiceName, environment, showControls, raytrace, ...).

Param	Type	Description
vrmUrl (or vrmurl)	string	URL of the .vrm to load.
voiceName	string	A VoiceNames key from ai-character (defaults to Yui if not set or unknown).
environment	string	Environment preset name (default transparent) (iframe backgrounds are also transparent and will show the parent page behind them). See Environment.
showControls	bool	Show the chat overlay + the emotion sliders. Nice showcase and testbed for you to see your character in action.
cameraType	default | orthographic	Camera projection.
lowPerformanceMode	bool	Lighter render/update path (no shadows, simpler lighting).
webGPURenderer	bool	Use the WebGPU renderer (incompatible with raytracing).
raytrace	bool	Enable the GPU path-tracer (experimental).
raytraceOpt	JSON	Path-tracer overrides (URI-encoded JSON). See Rendering.
initialFocus	JSON	Initial camera focus (URI-encoded AICFocusProps from ai-character).
decoupleOverlappingExpressionsExperimental	bool	Forwarded to ai-character (split conflicting blendshapes, not perfect, fix models in blender when deploying in production).
animationSmoothing	float	Global motion smoothing 0..1.
bodyAnimationSmoothing / headAnimationSmoothing / faceAnimationSmoothing / eyesAnimationSmoothing / mouthAnimationSmoothing / armsAnimationSmoothing / legsAnimationSmoothing / handsAnimationSmoothing	float	Per-region smoothing 0..1.
animationSmoothingMode	dampen | ease | dampen_and_ease	Smoothing strategy.

Example:

https://your-host/?vrmUrl=https://cdn.example/ada.vrm&voiceName=ruri&environment=tokyo&showControls=true
Copy

See also the ai-vrm-chat README, which is built into sandai.org. UrlBuilder constructs that URL with typed params:

import { UrlBuilder } from "sandai-core";

interface SandaiParams {
  vrmUrl: string;
  voiceName: string;
  environment: string;
  showControls: boolean;
}

const src = new UrlBuilder<SandaiParams>("https://sandai.org/chat")
  .setParam("vrmUrl", "https://cdn.example/ada.vrm")
  .setParam("voiceName", "ruri")
  .setParam("environment", "tokyo")
  .build();

iframe.src = src;
Copy

setParam(key, value) skips null/undefined, and JSON-encodes (URI-encoded) object values to match what ai-vrm-chat expects for params like initialFocus and raytraceOpt. build() returns the final URL.

Voices

sandai-core re-exports the same voices map as ai-character (~100 Piper voices across 30+ languages), each with name, modelUrl, modelConfigUrl, optional speakerId, gender, language, and languageName. Use the keys as voiceName in say/respond or the iframe URL.

import { voices } from "sandai-core";

const english = Object.values(voices).filter((v) => v.language.startsWith("en"));
Copy

The postMessage protocol

Under the managers, everything is window.postMessage. The client validates event.origin against the iframe's origin. If you ever need to talk to the iframe without this SDK, this is the contract.

Parent → iframe:

Message	Sent by	Reply
{ type: "auth", userId, apiKey }	AuthManager	{ type: "auth", tier, lifecycle }
{ type: "sendSpeech", id, message, options }	InteractionManager say	{ type: "speechStarted", id }, then { type: "speechStopped", id }
{ type: "respond", id, message, context, options }	InteractionManager respond	{ type: "characterResponded", id, message }
{ type: "stopSpeech", id }	InteractionManager stop	{ type: "speechStopped", id }
{ type: "iframe-rpc-request", id, path, args }	RPCManager	{ type: "iframe-rpc-response", id, result }

Iframe → parent (unprompted):

Message	Meaning
{ type: "loadState", state, lifecycle, hasInteracted, message }	Load/readiness updates. Re-broadcast every ~5s; state is loading / ready / loadError.

Types

Exported from the package root (src/types.ts and InteractionManager.ts):

• AuthClientMessage, AuthServerMessage
• LoadStateServerMessage
• InteractionManagerServerMessage = RespondServerMessage | SpeakServerMessage | StopServerMessage
• SendSpeechMessage, RespondMessage, StopSpeechMessage, SharedMessageOptions
• AllRpcMethods (re-exported from ai-vrm-chat/index.rpc)

Development

bun install
bun run dev     # ladle stories
bun run build   # generates typedoc API docs, then rollup (CJS + ESM + d.ts)
bun run doc     # regenerate the typedoc docs only
Copy

sandai-core's RPC types are derived from ai-vrm-chat's generated index.rpc surface, so they track the character's public API. If the character's methods change, rebuild ai-vrm-chat's RPC types first.

License

MIT. Contributions welcome — open an issue or PR. For support, reach out to davidckss@proton.me.