How Puppetry Works

How Puppetry Works

The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets.  A Puppetry Viewer runs a plug-in which supplies joint movement instructions.  The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server, and relayed to other nearby Viewers who apply the same animation conversion.

Puppetry is still an experimental feature and is not yet deployed everywhere.  It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region.  The Viewer can be obtained from the alternative viewer download page, or since it is open-source it can be built from scratch.  The Puppetry feature is only enabled on the Preview grid in a few regions: Bunraku, Marionette or Castelet.

Try It Out

Example Puppetry plugins can be found in the LEAP git repository.  Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the puppetry README.md for installing required Python modules.

Start up the Puppetry Viewer, log into the Preview grid and go to one of the aforementioned regions.   Open up the Advanced menu, and you should see Puppetry listed there.   Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)

Select Launch plug-in... and navigate to the puppetry modules in the LEAP repository you downloaded to your local filesystem earlier.   In  your copy of the LEAP repository find leap/puppetry/examples/ directory and select arm_wave.py.  Your avatar should raise up their left arm and move it slowly back and forth.

The arm_wave.py  movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level.   That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.

The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint.  The Viewer uses  Inverse Kinematics (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation.  The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view.  Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.

The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.

Some of the selections on the Puppetry menu are placeholders and don't do much now.  The "Face" and "Fingers" checks, for example, are intended to enable or disable Puppetry animations for those body parts but aren't functional yet.

The leap/puppetry/webcam/webcam_puppetry.py  script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data.  You may need to select different camera numbers from the menu to get this feature to start.   This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be.  It's a great hint, however, of the potential of this new feature.

Known Issues - The Broken Bits

This feature is experimental and sometimes experiments go wrong and fail and crash and burn

• Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
• The project viewer occasionally crashes when using Puppetry
• Correct hand tracking by webcam_puppetry.py is still a work in progress
• Lip, facial, and finger Puppetry requires a Bento avatar
• IK does not work well on fingers

LEAP Puppetry API

This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer.    The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.

Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.

Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the LLPuppetModule constructor.  There are only two:  set and get.

Outbound commands are sent by the viewer to the plug-in.  They are defined in puppetry.py with the @registerCommand decorator.  As of this writing, they are: stop, set_camera, enable_parts, and set_skeleton

Please note that plug-ins which issue blocking commands (taking too much time) will drop LEAP commands. In the examples, webcam_puppetry.py contains code working around this issue while waiting for the startup of the pose recognition software.

Inbound Commands (plug-in to viewer)

Command	Description
set	Update Puppetry configuration with new values. (e.g. {'command':'set','data':{'inverse_kinematics':{'mWristLeft':{'position':[x,y,z]}}}})
get	Query Puppetry for info: (e.g. {'command':'get','data':["camera", "skeleton"]})

There are two modes for setting joint transforms inverse_kinematics (avatar-frame) and joint_state (parent-frame). They can be used simultaneously, even on the same joint, because they represent different things.

Mode	Terse name	Description
inverse_kinematics	i	Use to set position and rotation of a Joint's IK target transform in the avatar-frame
joint_state	j	Specify position, rotation, and scale of Joint in its parent-frame

In either mode you can specify the position or rotation of the joint. In the parent-frame (joint_state) mode you can also adjust the scale.

Param	Terse name	Format	Description
position	p	[x, y, z]	Array of three floats representing Cartesian position
rotation	r	[x, y, z]	Array of three floats representing the imaginary (xyz) components of a unitary Quaternion whose real component (w) is positive
scale	s	[x, y, z]	Array of three floats representing Cartesian scale to be applied to Joint's position in its parent-frame, and also to Joint bone length

Outbound Commands (viewer to plug-in)

Command	Description
enable_parts	Passed an integer representing a mask of which parts of the skeleton are to be controlled by the script.  This matches the Puppetry menu selections.
Head	0x01
Face	0x02
Left Hand	0x04
Right Hand	0x08
Fingers	0x10
set_camera	Contains the tag camera_id which is an integer which identifies which camera to use.   This matches the Puppetry menu selection.
set_skeleton	Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end.  The trailing end position is typically but not always the parent-relative position of the immediate child. Note: mPelvis is the root of the avatar and thus does not have a parent-relative position or parent_id. Example: mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis) Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)
stop	No params, ends plug-in puppetry control of avatar.

Show Bones

The Develop Menu's Avatar options includes "Show Bones", which draws lines showing the position and length of the skeleton parts. Any bones that are moved with Puppetry data will be drawn with dark blue. By turning on this option, you can observe how the bones are actually moving. This helps demonstrate how your avatar's rigging works to alter the actual appearance.