Wireface for three.js

The Wireface SDK's three.js integration puts the Wireface face into a three.js scene. The face is the one in Wireface Desktop: the same blinks, breathing, expressions, gaze and lip sync, here moving real geometry that your lights and shadows fall on, as many faces as you like. This page is the guide for developers; the product page is Wireface for three.js, and the quickest way to see it is the Wireface Theatre demo.

Ways to give something a face#

All of them answer to the same methods (express, mood, state, say, hush, listenTo, gazeAt), so a character does not need to know which kind of face it has.

Set up#

Download the SDK from your account and add it to your project next to three.js (version 0.160 or later, a peer dependency): Get the SDK has the steps. With a bundler the imports just work. With no build step, serve the SDK's dist/ and assets/ folders together and map the module names in an import map (the paths here are wherever you serve them):

<script type="importmap">
{ "imports": {
  "three": "/vendor/three/build/three.module.js",
  "three/addons/": "/vendor/three/examples/jsm/",
  "@wireface/sdk/three": "/wireface/dist/three.js",
  "@wireface/sdk/three/characters": "/wireface/dist/three-characters.js",
  "@wireface/sdk/three/kit": "/wireface/dist/three-kit.js",
  "@wireface/sdk/2d": "/wireface/dist/2d.js",
  "@wireface/sdk/2d/live": "/wireface/dist/2d-live.js",
  "@wireface/sdk/create": "/wireface/dist/create.js"
} }
</script>

The three.js modules are @wireface/sdk/three (faces, screens and sound), @wireface/sdk/three/characters (the cast) and @wireface/sdk/three/kit (what the cast is made of).

A face on a head#

Make a FaceMesh, add it to a head with attachFace, and render as usual. The face updates itself as the scene renders.

import * as THREE from 'three';
import { FaceMesh, attachFace, unlockAudio } from '@wireface/sdk/three';

const head = new THREE.Group();            // or a mesh, or a bone
head.position.y = 1.08;
scene.add(head);

const face = new FaceMesh({ skin: 'sarah', size: 0.24 });   // 24 cm from chin to hairline
attachFace(head, face);                    // now it nods and sways `head` as it talks
face.gazeAt(camera);                       // the eyes first, then the head, follow the camera
face.express('happy', 3);

button.onclick = async () => {
  const audioContext = unlockAudio();      // on the click itself
  await face.say(await face.clip('sarah'), { audioContext });
};

renderer.setAnimationLoop(() => renderer.render(scene, camera));

size is the face's height in scene units (setSize changes it). The face brings the back of a head in its skin's colour, so a bare group is enough. attachFace(head, face, { at, rotation, size, motion }) places it, and motion: false leaves the head still. Other options include skull ('auto' gives skin styles a back of the head), headShape (width, height, depth, crown), headColor, shadows and autoUpdate: false if you would rather call face.update() yourself. Dress the head from what the face tells you:

const { skull, top } = face.bounds;      // positions in the face's own space: +z is forward, +y is up
const hat = new THREE.Mesh(new THREE.ConeGeometry(0.1, 0.2, 24), new THREE.MeshStandardMaterial({ color: '#7a3b8f' }));
hat.position.set(0, top + 0.1, skull.center.z);
face.head.add(hat);                            // face.head moves with every nod; hair and glasses go here too
hands.material = face.skinMaterial();          // a material in the head's colour, kept in step when the skin changes

The kit (below) has hairCap, hairFall and ears that fit a FaceMesh's head.

Skins, looks and materials#

Skins. The included skins are sarah, marcus, amara, walter, lucia, arjun, omar, ruth, robot, monster and dracula; the monster brings sharp teeth and Dracula fangs. Change one with await face.useSkin('robot', { sweep: 1.2 }), which scans the new skin down over the face (null takes the skin off). Your own photo becomes a skin on the device, with nothing uploaded, and a character from createCharacter fits the geometry to the person:

import { skinFromPhoto } from '@wireface/sdk/three';
import { createCharacter } from '@wireface/sdk/create';

await face.useSkin(await skinFromPhoto(photoFile));        // the photo mapped onto the face
await face.useCharacter(await createCharacter(photoFile)); // or the mesh itself fitted to the face (kind '3d')

Looks. style is photo (a skin shows in this), sculpt (shaded, lit by your scene), toon (cel-shaded) or hologram (a glowing wireframe, made for a bloom pass). The colour preset (cyan, ember, green, amber, violet, rose or ice) sets the hologram's colours, and skin_amount cross-fades from wireframe (0) to skin (1). Change any of it live:

new FaceMesh({ style: 'toon', colors: { skin: '#f2b38b', iris: '#2563eb' } });
new FaceMesh({ style: 'hologram', preset: 'violet', glow: 1.2 });
face.set({ skin_amount: 0.5 });                // half wireframe; set() throws ConfigError for a bad value

The skin styles are three.js's own lit materials, so your lights and shadows fall on the face; the hologram is drawn additively. The mouth group, eyes and surface meshes are there if you need to reach in.

The cast#

Eleven ready-made characters in @wireface/sdk/three/characters, each showing one way to put a face on something. They are short files to start from: copy the closest one.

import { buildCharacter, CAST } from '@wireface/sdk/three/characters';

const clock = new THREE.Clock();
const walter = buildCharacter('walter');
scene.add(walter.root);
walter.gazeAt(camera);
renderer.setAnimationLoop(() => {
  walter.update(clock.getDelta(), clock.elapsedTime);   // breathing, gestures, looking
  renderer.render(scene, camera);
});
await walter.speak({ audioContext: unlockAudio() });    // from a click: its own line, with its expressions

Each CAST entry has the fields blurb and code (the heart of how it is built), and build(options), whose options go to its face. The characters are made from @wireface/sdk/three/kit: toy-figure materials (clay, vinyl, metal, cloth, glow, glass), shapes (part, capsule, ball, box, lathe), toyBody, hairCap, hairFall, ears and the Character class, which gives a figure its idle life and talking gestures.

A face on a screen#

A FaceScreen runs a whole Wireface face on a hidden canvas and uses that canvas as a texture, so the face can sit on a monitor, a visor, a TV or a painting. engine: '3d' is the glowing-wireframe or skinned face; engine: '2d' is Wireface 2D making a picture talk. It draws a frame whenever the mesh it is bound to is drawn.

import { FaceScreen, screenGeometry, unlockAudio, voices } from '@wireface/sdk/three';

const screen = new FaceScreen({ engine: '3d', resolution: 512,
  config: { preset: 'cyan', skin_amount: 0, glow: 1 } });
const glass = new THREE.Mesh(
  screenGeometry({ width: 0.64, height: 0.48, curve: 0.02 }),   // visorGeometry() for a visor
  screen.material({ corner: 0.04, scanlines: 0.2 }));
monitor.add(screen.bind(glass));           // drawn whenever the glass is drawn
screen.gazeAt(camera);
await screen.say(await screen.clip('robot', voices()), { audioContext: unlockAudio() });   // from a click

Methods wait for the engine to load, so you can call them straight away; await screen.ready if you need the face itself (screen.face). Each FaceScreen is a WebGL context, so keep the number modest. For a 2D screen give character a bundled character's id (as listed in Wireface 2D) or the URL of a saved .wireface.json. dispose() frees the engine and the texture.

A face drawn on an object#

For a ghost, a pumpkin or a can, draw the face in the object's own colours as a 2D character, then let wrapGeometry curve its picture round the object and paintOn lay it into the object's material, so the metal or cloth still catches your light. Make the character once (see Make a character) and save it.

import { FaceScreen, wrapGeometry } from '@wireface/sdk/three';

const screen = new FaceScreen({ engine: '2d', character: '/faces/can-mint.wireface.json',
  config: { framing: 'contain', depth: 0.12, teeth_mode: 'none' } });
const geometry = wrapGeometry(new THREE.CylinderGeometry(0.2, 0.2, 0.32, 64, 8),
  { center: [0, 0, 0], radius: 0.2, width: 0.3, height: 0.3 });
const can = new THREE.Mesh(geometry, screen.paintOn(new THREE.MeshStandardMaterial({ color: '#5fb8a4', metalness: 0.45 })));
head.add(screen.bind(can));

wrapGeometry writes the geometry's UVs: the face goes round the front (+z) of the shape's cross-section, spanning width metres of surface at the given radius and height metres tall, centred on center.

On a rigged model#

For a glTF character, put a FaceMesh on the head bone. The face's nods and turns are added on top of your animations, so the model can wave, dance and talk at once.

import { FaceMesh, attachFace, findBone } from '@wireface/sdk/three';

const robot = (await new GLTFLoader().loadAsync('robot.glb')).scene;
attachFace(findBone(robot, /head/i), new FaceMesh({ skin: 'robot', size: 0.3 }));
mixer.update(dt);                          // your animations first; the face adds its motion as it renders

To animate a model's *own* face, ModelFace takes saved feature positions for its eyes and mouth and adds lids that blink, a glint that follows you and a mouth that lip-syncs, on the model's head: new ModelFace({ ...features, head: findBone(robot, /^head$/i) }). The SDK's examples/three/attach-to-gltf.html does this for a robot.

Speech, lip sync and sound#

Browsers keep a page silent until a click, so call unlockAudio() in a click handler (or onFirstGesture(fn) to wait for the first click or key). Without an audio context a face mimes its line.

• Recorded lines. face.clip(id, base?) loads a line aligned phoneme by phoneme, so the lips close on every m, b and p. A FaceMesh looks in the SDK's assets/voices folder (one for each included skin). The cast's lines and the 2D characters' lines are in other folders: voices('2d') gives the 2D folder, and buildCharacter(...).speak() picks the right one.
• Live audio. face.listenTo(audioContext, source, { gain }) reads the mouth from any sound as it plays: a microphone stream, a WebRTC call, text-to-speech in an <audio> element. It returns a function that stops it. Stop the stream's tracks yourself when you are done, so the recording light goes off.
• Streamed speech and conversations. StreamVoice plays PCM that streams in, and Conversation from @wireface/sdk/2d/live runs a Gemini Live chat against a FaceMesh. It needs a token endpoint on your server, where your Gemini key stays.
• Expressions. express(name, seconds), mood(name, intensity) and state('thinking') work as in the 3D face, and defineExpression adds your own for every face at once. face.level (0 to 1) says how much a face is talking, to drive lights or a body.

const audioContext = unlockAudio();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const stop = face.listenTo(audioContext, stream, { gain: 1.2 });
// ...later
stop();
stream.getTracks().forEach(track => track.stop());

You can look at things as well: face.gazeAt(target) takes a camera, an object or a world point; the eyes go first and converge on it, and the head turns for the rest, up to { turn } radians (null looks ahead).

Bundlers and assets#

The SDK's modules import three by name, so a bundler resolves them from node_modules with no settings. Two things need care:

1. Assets. Skins and recorded lines sit in the SDK's assets/ folder, beside the bundles. A bundler does not keep them there, so copy the folder into your public directory and pass assets to a face (new FaceMesh({ skin: 'sarah', assets: '/wireface/' })). For recorded lines call setVoiceFolders({ '3d': '/wireface/voices/', '2d': '/wireface/2d/voices/' }) once.
2. Dependency pre-bundling. If your bundler pre-bundles dependencies (Vite does), exclude @wireface/sdk so it can find its own files.

For createCharacter the MediaPipe paths are covered in the 2D guide.

The Wireface Theatre#

The Wireface Theatre puts the eleven cast characters on one stage in your browser. Click anyone to hear their line, change their looks and skins, and watch the others turn to look. The SDK download includes fourteen runnable examples in examples/three/ too: a face on a glTF robot, a can, a monitor, a crowd, a microphone, a Gemini Live chat and more.

Where next#

• Get the SDK and the developer guide, which covers character creation, the detection API and export.
• Wireface 2D: the engine behind faces drawn on objects, and how to make a 2D character from a picture.
• Wireface for three.js, the product page.

Adding Wireface to an app, game or website is free under the Personal Licence. If you charge for what you build, see Licensing for the licence you need.