How Puppetry Works

How Puppetry  Works

The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets.  A Puppetry Viewer runs a plug-in which supplies joint movement instructions.  The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server and relayed to other nearby Viewers who apply the same animation conversion.

Puppetry is still an experimental feature and is not yet deployed everywhere.  It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region.  The Viewer can be obtained from the alternative viewer download page, or since it is open-source it can be built from scratch.  The Puppetry feature is only enabled on the Preview grid in a few regions: Bunraku , Marionette or Castelet.

Try It Out

Example Puppetry plugins can be found in the LEAP git repository.  Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the puppetry README.md for installing required Python modules.

Start up the Puppetry Viewer, log into  the Preview grid, and go to one of the aforementioned regions.   Open up the Advanced menu, and you should see "Puppetry " listed there.   Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)

Select "Launch plug-in..." and navigate to the puppetry modules in the LEAP repository you downloaded to your local filesystem earlier.   In  your copy of the LEAP repository find leap/python/puppetry/examples/ directory and select arm_wave.py .  Your avatar should raise up their left arm, and move it slowly back and forth.

The arm_wave.py  movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level.   That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.

The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint.  The Viewer uses Inverse Kinematics (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation.  The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view.  Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.

The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.

Some of the selections on the Puppetry menu are placeholders and don't do much now.  The "Face " and "Fingers " checks, for example, are intended to enable or disable Puppetry animations for those body parts, but aren't functional yet.

The leap/python/puppetry/webcam/webcam_puppetry.py  script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data.  You may need to select different camera numbers from the menu to get this feature to start.   This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be.  It's a great hint, however, of the potential of this new feature.

Known Issues - The Broken Bits

This feature is experimental and sometimes experiments go wrong and fail and crash and burn

.  

• Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
• The project viewer occasionally crashes when using Puppetry
• Correct hand tracking by webcam_puppetry.py is still a work in progress
• Lip, facial, and finger Puppetry requires a bento avatar
• IK does not work well on fingers

LEAP Puppetry API

This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer.    The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.

Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.

Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the LLPuppetModule constructor.  Briefly, they are:  move, get_camera, set_camera, send_skeleton

Outbound commands are sent by the viewer to the plug-in.  They are defined in puppetry.py with the @registerCommand decorator.  As of this writing, they are: stop, log, set_camera, enable_parts, and set_skeleton

Please note that plug-ins which issue blocking commands (taking too much time) will drop leap commands. In the examples, webcam_puppetry.py contains code working around this issue while waiting for the startup of the pose recognition software.

Inbound Commands  (plug-in to viewer)

Command	Description
get_camera	Query the viewer as to the current camera.  Triggers a set_camera outbound message to relay the camera information to the plug-in
set_camera	Sets the webcam to the integer passed in as camera_id this setting is stored in the user's saved settings and will persist between viewer sessions
send_skeleton	Request the skeleton of the avatar from the viewer.  Issues a set_skeleton response to the plug-in
move	Move contains the data structure for a single frame of plug-in animation.  At the top level it is an array of joints using the standard mJoint (EX: mHead, mWristLeft) names used for traditional uploaded animations and defined in avatar_skeleton.xml.Each joint may contain the following tags:  rot, local_rot, pos, no_constraint.Move sub commands

Name	Params	Description
rot	3 floats	World space rotation.  The floats represent a compact quaternion.Rot and local_rot are incompatible with local_rot taking precedence.
local_rot	3 floats	Parent relative rotation.  The params represent a compact quaternion.Rot and local_rot are incompatible with local_rot taking precedence.
pos	3 floats	XYZ position of the effector target for this joint.  Note this is the trailing edge of the bone, not the leading edge thus, a position for mElbowLeft can be thought of as where the forearm ends, or as the start of the wrist.  Positions are in a normalized space (defined as the average length of the arms) relative to the pelvis.
no_constraint	boolean	True disables the humanoid constraint for this joint.  False is the default state.

Example structure:

• mHead
• Pos
• X offset from pelvis
• Y offset from pelvis
• Z offset from pelvis
• Rot
• Yaw component of compact quaternion
• Pitch component of compact quaternion
• Roll component of compact quaternion
• mElbowLeft
• ...Notes/Errata:Rot and local_rot are incompatible.  If both are issued on the same joint in a frame, local_rot will be used.If position is given for two sequential joints (EX, the mShoulderLeft and mElbowLeft) the viewer asserts priority to the child (the elbow in our example) and automatically adjusts the position of the parent (shoulder in this case) to length of the parent bone in the direction from child to parent If pos is specified for mShoulderLeft or mShoulderRight, the constraint is automatically disabled |

Outbound Commands (viewer to plug-in)

Command	Description
enable_parts	Passed an integer representing a mask of which parts of the skeleton are to be controlled by the script.  This matches the Puppetry menu selections.
Head	0x01
Face	0x02
Left Hand	0x04
Right Hand	0x08
Fingers	0x10
set_camera	Contains the tag camera_id which is an integer which identifies which camera to use.   This matches the Puppetry menu selection.
set_skeleton	Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end.  The trailing end position is typically but not always the parent-relative position of the immediate child.Note: mPelvis is the root of the avatar and thus does not have a parent-relative position or parent_id Example mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)
stop	No params, ends plug-in puppetry control of avatar.