Head and face
When the head turns, the face follows like a volume and not a flat picture. Eyes, brows, mouth and bangs each move at their own depth.
Mochi
An AI VTuber that speaks first and keeps the conversation going.
Mochi is a Korean-speaking AI VTuber that speaks first and carries a topic on, without waiting for a question. We are building two things: an avatar that moves, and the program that lets it talk on stream.
The avatar: from one drawing to a moving model
Claude turned a single drawing, split into 98 layers, into a moving Live2D model (rigging). It took 23 revisions over three days.
Eyes and mouth
Blinking, smiling eyes, gaze, and mouth shapes for speech.
Sway
Hair, ribbons, sleeves, wings and antennae swing after the movement (physics).
Expressions
Smile, smiling eyes, angry, troubled, surprised, shy: six expressions and an idle motion.
Three ways Claude works the Live2D editor
- 01
Operating the screen
It looks at the editor and clicks through the menus like a person, to build the structure.
- 02
Connecting to the editor
It reads and writes parts and values through the editor's own API.
- 03
Editing the model file
Hundreds of movement values are computed and written straight into the model file.
Claude also wrote a preview renderer to check poses without the editor. Changes to the drawing itself are made by a person.
The streaming program: read, choose, speak
It reads chat, chooses what to respond to, writes a line, and sends it out as voice and mouth movement.
Speaks first
With no chat at all it brings up a topic and carries the last one on.
Reacts to the stream
Donations, subscriptions and follows are answered with the right viewer's name.
Remembers what was said
Only lines that were really played as sound go into memory, and repeats are held back.
Operator console
A control page for the streamer: pause, quieter, chattier.
The circuit that chooses what to respond to
For the choosing step we experiment with a circuit derived from a public wiring map of the fruit fly brain (the FlyWire connectome). The circuit does not understand language. A language model reads the meaning and writes the lines; the circuit picks whom to respond to and whether to speak, from signals we designed. Our experiments have not shown the real wiring to be better than the alternatives.
What Claude does
- The whole Live2D rig: structure, movement values, physics, expressions, export
- The code of the streaming program: features, fixes, 365 automated tests
- Measuring instead of guessing: it measured why other languages leaked into Korean lines, found the cause and stopped it
Where it is
In development. Conversation, voice and the avatar's mouth movement run together on one PC. It is not yet connected to a streaming platform's chat, and the model's drawing is still being revised.