sandai-react - v2.1.7

sandai-react

The React way to embed a conversational 3D AI avatar; on-device, privacy-first, one component.

See what it's able to do in a custom implementation from DIE KAVALLERIE 🐴

Live playground: sandai.org

sandai-react is the React SDK for dropping a fully interactive 3D AI avatar into any React or Next.js app. Mount a single <AI3DCharacter /> component and you get a real-time AI avatar. A browser-based digital human, mascot, animal, you name it, that can speak, listen, think, and emote. Under the hood it renders the same heavily optimized runtime as sandai-core, so you get a complete conversational AI avatar: on-device text-to-speech, speech recognition, an on-device LLM, emotion inference, and lip-sync, with no backend to run and nothing to install.

By default, everything runs locally in the browser via WebAssembly and WebGL/WebGPU. No API keys, no per-message costs, and no user audio ever leaves the device.

sandai-react re-exports all of sandai-core and wraps it in idiomatic React: a declarative component that builds the embed URL from props and hands you a typed SandaiClient in onLoad. It builds on the same stack: sandai-core → ai-vrm-chat → ai-character → @davidcks/r3f-vrm (Three.js / React Three Fiber).

Features

  • One component, zero boilerplate: render <AI3DCharacter />, get a typed SandaiClient in onLoad. No iframe wiring, no postMessage plumbing, no 3D engine, no game engine, no local GPU server, no API gateway.
  • 3D VRM avatars: load any .vrm model and get a fully animated talking avatar with auto-blink, gaze, idle motion, and camera focus. Bring your own character, use a bundled one, or pick from over 100,000 models at sandai.org.
  • On-device LLM: a local LLM (Gemma, Qwen, and others via transformers.js / ONNX and MediaPipe LiteRT-LM) generates replies in-browser. Or plug in your own and just stream text in.
  • Text-to-speech, 100+ voices, 30+ languages: built-in multilingual TTS powered by Piper, selectable per character with the typed voiceName prop.
  • Real-time lip-sync: accurate lip-sync driven from phonemes or directly from the audio waveform, with coarticulation so the mouth never looks robotic.
  • Speech recognition (ASR) + VAD: the avatar can listen and reply on its own via on-device Whisper / Moonshine speech-to-text with Silero voice-activity detection.
  • Emotion inference & expression: text is mapped to 28 GoEmotions-style emotions that automatically drive facial expressions and graded body animations.
  • Fully typed: props, voices (VoiceNames), environments (Environments), and the entire SandaiClient RPC surface ship with TypeScript types and autocomplete.
  • WebGL & WebGPU rendering, with an optional experimental GPU path-tracer (raytrace) for cinematic, ray-traced scenes.
  • Built-in dev tooling: AI3DCharacterDocs renders interactive, runnable documentation for every method the character exposes — a live playground inside your own app.
  • Privacy-first, no backend: conversation, voice, and inference happen on the user's device. Great for GDPR-sensitive products, offline-capable apps, and keeping inference costs at zero.

Use cases

AI agents and website assistants, customer support and onboarding, AI companions and VTubers, interactive characters for games and education, AI mascots and brand mascots and virtual presenters, and any React product that benefits from a face you can talk to.

Table of Contents

Installation

npm install sandai-react

Peer dependencies (React 19):

npm install react react-dom

Quick Start

Mount the component, size its container, and grab the client in onLoad:

import { AI3DCharacter, SandaiClient } from "sandai-react";
import { useState } from "react";

export default function App() {
  const [client, setClient] = useState<SandaiClient>();

  return (
    <div style={{ width: "100%", height: "600px" }}>
      <AI3DCharacter
        url="https://sandai.org/chat"
        vrmUrl="https://cdn.example/ada.vrm"
        voiceName="ruri"
        environment="tokyo"
        // userId & apiKey are optional. Without them you may see a watermark.
        // Grab yours in the dashboard: https://sandai.org/dashboard
        onLoad={async (c) => {
          setClient(c);
          // Speak some text (emotion + lip-sync inferred automatically):
          const { speechEndPromise } = await c.interactionManager.say("Hi there!");
          await speechEndPromise;
          // Or have the on-device LLM reply:
          await c.interactionManager.respond("What's your name?");
        }}
      />
    </div>
  );
}

The component renders an iframe at width: 100%; height: 100%, so give its parent an explicit size.

Audio can't play until the user has interacted with the page (browser autoplay policy). If the character won't speak, that's almost always why — the underlying LoadManager tracks interaction state for you.

Components

AI3DCharacter

The one component you mount. It builds the Sandai embed URL from its props, renders the avatar in an iframe, constructs a SandaiClient, and hands it to you via onLoad once it's ready.

import { AI3DCharacter } from "sandai-react";

Props

Prop Type Description
url string Required. Base URL of the Sandai chat interface (e.g. https://sandai.org/chat).
onLoad (client: SandaiClient) => void Required. Fired once the SandaiClient is initialized and ready to drive.
vrmUrl string URL of the .vrm model to load as the character.
voiceName VoiceNames TTS voice key (100+ multilingual Piper voices).
environment Environments Environment preset (e.g. transparent, tokyo, sky, candyshop, ...).
showControls boolean Show the debug overlay: chat input, emotion sliders, and built-in voices. Handy for previewing how the 5 VRM blendshapes combine into Sandai's 27 emotions.
lowPerformanceMode boolean Lighter render path (drops bloom/depth-of-field) for older or low-end devices.
cameraType "default" | "orthographic" Camera projection. default is perspective.
initialFocus FocusProps Initial camera focus (focusIntensity, cameraOffset, lookAtOffset, trackCharacterLookAt, focusMode).
raytrace boolean Enable the experimental GPU path-tracer.
headAnimationSmoothing number Dampen head motion, 0..1 (0 = raw, 1 = frozen).
bodyAnimationSmoothing number Dampen body motion, 0..1.
armsAnimationSmoothing number Dampen arm motion, 0..1.
legsAnimationSmoothing number Dampen leg motion, 0..1.
handsAnimationSmoothing number Dampen hand motion, 0..1.
faceAnimationSmoothing number Dampen face/expression motion, 0..1. Try 0.1–0.9 if the mouth looks choppy.
decoupleOverlappingExpressionsExperimental boolean Best-guess split of conflicting blendshapes (e.g. a model that opens its mouth when happy).
apiKey string Your API key (from the dashboard). Optional; omit and you may see a watermark.
userId string Your user ID (from the dashboard).
debugMode boolean Skips auth + load checks. Convenient locally; turn off in production (it disables the interaction gate used to enable audio).
style React.CSSProperties Inline styles for the underlying iframe.
className string Class name for the underlying iframe.

Example

import { AI3DCharacter } from "sandai-react";

<AI3DCharacter
  url="https://sandai.org/chat"
  vrmUrl="https://example.com/model.vrm"
  voiceName="ruri"
  environment="tokyo"
  showControls
  faceAnimationSmoothing={0.4}
  onLoad={(client) => console.log("Sandai client ready", client)}
/>

AI3DCharacterDocs

Renders interactive, runnable documentation for every method the loaded SandaiClient exposes — a live API playground you can drop straight into your own app. Pass it the client from AI3DCharacter's onLoad.

Props

Prop Type Description
client SandaiClient The instance from AI3DCharacter's onLoad callback.

Example

import { AI3DCharacter, AI3DCharacterDocs } from "sandai-react";
import { useState } from "react";

const App = () => {
  const [client, setClient] = useState(null);

  return (
    <>
      <AI3DCharacter url="https://sandai.org/chat" onLoad={setClient} showControls />
      {client && <AI3DCharacterDocs client={client} />}
    </>
  );
};

FunctionTester

A reusable form that dynamically generates input fields from a schema and calls a function on submit. It powers AI3DCharacterDocs, and you can use it standalone to build quick test harnesses around any function.

Props

Prop Type Description
func Function The function to invoke; receives the form values as arguments.
schema SchemaType[] Field definitions for the function parameters.

Example

import { FunctionTester } from "sandai-react";

const myFunction = (name, age) => console.log(`Name: ${name}, Age: ${age}`);

const schema = [
  { id: "name", type: "text", label: "Name" },
  { id: "age", type: "number", label: "Age" },
];

<FunctionTester func={myFunction} schema={schema} />;

Driving the character

onLoad hands you a SandaiClient, which is your control surface for the avatar. The most common calls:

// Speak text — emotion + lip-sync are inferred automatically.
const { speechEndPromise } = await client.interactionManager.say("Hello!");
await speechEndPromise;

// Ask the on-device LLM for a reply, then speak it.
const reply = await client.interactionManager.respond("Tell me a joke about space.");

// Interrupt the current utterance.
await client.interactionManager.stop();

// Call any method on the character by typed dot-path (full autocomplete).
await client.rpcManager.call("setEmotion", "joy", 0.8);
await client.rpcManager.call("vrmManager.focusManager.focus");

The full SandaiClient API — interactionManager, loadManager, authManager, and the typed rpcManager — is documented in the sandai-core README.

Re-exports from sandai-core

sandai-react re-exports everything from sandai-core, so you can import the client, the URL builder, the voices map, and all the types straight from this package:

import {
  SandaiClient,
  UrlBuilder,
  voices,
  type VoiceNames,
  type Environments,
  type AllRpcMethods,
} from "sandai-react";

Contributing

Contributions are welcome! If you have suggestions or find issues, feel free to open an issue or a pull request.

License

MIT. See the LICENSE file for details.

Settings

Member Visibility

On This Page

sandai-react