How Puppetry Works
How Puppetry Works
The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets. A Puppetry Viewer runs a plug-in which supplies joint movement instructions. The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server and relayed to other nearby Viewers who apply the same animation conversion.
Puppetry is still an experimental feature and is not yet deployed everywhere. It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region. The Viewer can be obtained from the alternative viewer download page, or since it is open-source it can be built from scratch. The Puppetry feature is only enabled on the Preview grid in a few regions: Bunraku , Marionette or Castelet.
Try It Out
Example Puppetry plugins can be found in the LEAP git repository. Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the puppetry README.md for installing required Python modules.
Start up the Puppetry Viewer, log into the Preview grid, and go to one of the aforementioned regions. Open up the Advanced menu, and you should see Puppetry listed there. Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)
Select Launch plug-in... and navigate to the puppetry modules in the LEAP repository you downloaded to your local filesystem earlier. In your copy of the LEAP repository find
leap/puppetry/examples/ directory and select
arm_wave.py. Your avatar should raise up their left arm, and move it slowly back and forth.
arm_wave.py movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level. That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.
The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint. The Viewer uses Inverse Kinematics (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation. The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view. Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.
The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.
Some of the selections on the Puppetry menu are placeholders and don't do much now. The "Face " and "Fingers " checks, for example, are intended to enable or disable Puppetry animations for those body parts, but aren't functional yet.
leap/puppetry/webcam/webcam_puppetry.py script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data. You may need to select different camera numbers from the menu to get this feature to start. This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be. It's a great hint, however, of the potential of this new feature.
Known Issues - The Broken Bits
This feature is experimental and sometimes experiments go wrong and fail and crash and burn
- Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
- The project viewer occasionally crashes when using Puppetry
- Correct hand tracking by
webcam_puppetry.pyis still a work in progress
- Lip, facial, and finger Puppetry requires a bento avatar
- IK does not work well on fingers
LEAP Puppetry API
This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer. The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.
Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.
Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the constructor. Briefly, they are: move, get_camera, set_camera, send_skeleton
Outbound commands are sent by the viewer to the plug-in. They are defined in
puppetry.py with the @registerCommand decorator. As of this writing, they are: stop, set_camera, enable_parts, and set_skeleton
Please note that plug-ins which issue blocking commands (taking too much time) will drop leap commands. In the examples,
webcam_puppetry.py contains code working around this issue while waiting for the startup of the pose recognition software.
Inbound Commands (plug-in to viewer)
|get_camera||Query the viewer as to the current camera. Triggers a set_camera outbound message to relay the camera information to the plug-in|
|set_camera||Sets the webcam to the integer passed in as camera_id this setting is stored in the user's saved settings and will persist between viewer sessions|
|send_skeleton||Request the skeleton of the avatar from the viewer. Issues a set_skeleton response to the plug-in|
|move||Move contains the data structure for a single frame of plug-in animation. At the top level it is an array of joints using the standard mJoint (EX: mHead, mWristLeft) names used for traditional uploaded animations and defined in avatar_skeleton.xml.Each joint may contain the following tags: rot, local_rot, pos, no_constraint. Move sub commands|
|rot||3 floats||World space rotation. The floats represent a compact quaternion.Rot and local_rot are incompatible with local_rot taking precedence.|
|local_rot||3 floats||Parent relative rotation. The params represent a compact quaternion.Rot and local_rot are incompatible with local_rot taking precedence.|
|pos||3 floats||XYZ position of the effector target for this joint. Note this is the trailing edge of the bone, not the leading edge thus, a position for mElbowLeft can be thought of as where the forearm ends, or as the start of the wrist. Positions are in a normalized space (defined as the average length of the arms) relative to the pelvis.|
|no_constraint||boolean||True disables the humanoid constraint for this joint. False is the default state.|
- X offset from pelvis
- Y offset from pelvis
- Z offset from pelvis
- Yaw component of compact quaternion
- Pitch component of compact quaternion
- Roll component of compact quaternion
- ...Notes/Errata: Rot and are incompatible. If both are issued on the same joint in a frame, local_rot will be used.If position is given for two sequential joints (EX, the mShoulderLeft and mElbowLeft) the viewer asserts priority to the child (the elbow in our example) and automatically adjusts the position of the parent (shoulder in this case) to length of the parent bone in the direction from child to parent If pos is specified for mShoulderLeft or mShoulderRight, the constraint is automatically disabled
Outbound Commands (viewer to plug-in)
|enable_parts||Passed an integer representing a mask of which parts of the skeleton are to be controlled by the script. This matches the Puppetry menu selections.|
|set_camera||Contains the tag camera_id which is an integer which identifies which camera to use. This matches the Puppetry menu selection.|
|set_skeleton||Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end. The trailing end position is typically but not always the parent-relative position of the immediate child.Note: mPelvis is the root of the avatar and thus does not have a parent-relative position or parent_id Example mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)|
|stop||No params, ends plug-in puppetry control of avatar.|
The Develop Menu's Avatar options includes "Show Bones", which draws lines showing the position and length of the skeleton parts. Any bones that are moved with Puppetry data will be drawn with dark blue. By turning on this option, you can observe how the bones are actually moving. This helps demonstrate how your avatar's rigging works to alter the actual appearance.