Difference between revisions of "How Puppetry Works"

From Second Life Wiki
Jump to navigation Jump to search
m (added bitbucket links to all mentioned filenames)
m (No content changes, only formatting. Replaced Bitbucket links with GitHub links, to reflect LL's change of Git repository hosting.)
 
Line 1: Line 1:
[[Category:Puppetry]]
[[Category:Puppetry]]
{{distinguish|Puppeteering}}
{{distinguish|Puppeteering}}
<h2>How Puppetry Works</h2>
== How Puppetry Works ==


The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets.  A Puppetry Viewer runs a plug-in which supplies joint movement instructions.  The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server and relayed to other nearby Viewers who apply the same animation conversion.
The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets.  A Puppetry Viewer runs a plug-in which supplies joint movement instructions.  The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server, and relayed to other nearby Viewers who apply the same animation conversion.


Puppetry is still an experimental feature and is not yet deployed everywhere.  It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region.  The Viewer can be obtained from the [https://releasenotes.secondlife.com/viewer.html alternative viewer download page], or since it is open-source it can be [[Puppetry Development|built from scratch]].  The Puppetry feature is only enabled on [[Preview Grid|the Preview grid]] in a few regions: '''Bunraku''' , '''Marionette''' or '''Castelet'''.
Puppetry is still an experimental feature and is not yet deployed everywhere.  It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region.  The Viewer can be obtained from the [https://releasenotes.secondlife.com/viewer.html alternative viewer download page], or since it is open-source it can be [[Puppetry Development|built from scratch]].  The Puppetry feature is only enabled on [[Preview Grid|the Preview grid]] in a few regions: '''Bunraku''', '''Marionette''' or '''Castelet'''.


<h2>Try It Out</h2>
== Try It Out ==


Example Puppetry plugins can be found in the [https://bitbucket.org/lindenlab/leap/ LEAP git repository].  Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the [https://bitbucket.org/lindenlab/leap/src/main/puppetry/README.md puppetry README.md] for installing required Python modules.
Example Puppetry plugins can be found in the [https://github.com/secondlife/leap LEAP git repository].  Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the [https://github.com/secondlife/leap/blob/main/README.md puppetry README.md] for installing required Python modules.


Start up the Puppetry Viewer, log into [[Preview Grid|the Preview grid]], and go to one of the aforementioned regions.   Open up the '''Advanced''' menu, and you should see '''Puppetry''' listed there.   Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)
Start up the Puppetry Viewer, log into [[Preview Grid|the Preview grid]] and go to one of the aforementioned regions.   Open up the '''Advanced''' menu, and you should see '''Puppetry''' listed there.   Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)


[[File:howitworks.png|center]]
[[File:howitworks.png|center]]


Select '''Launch plug-in...''' and navigate to the puppetry modules in the {{abbr|LEAP|LLSD Event API Plug-in}} repository you downloaded to your local filesystem earlier.   In  your copy of the LEAP repository find <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/examples/ leap/puppetry/examples/]</code> directory and select <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/examples/arm_wave.py arm_wave.py]</code>.  Your avatar should raise up their left arm and move it slowly back and forth.


Select '''Launch plug-in...''' and navigate to the puppetry modules in the {{abbr|LEAP|LLSD Event API Plug-in}} repository you downloaded to your local filesystem earlier.   In  your copy of the LEAP repository find <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/examples/ leap/puppetry/examples/]</code> directory and select <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/examples/arm_wave.py arm_wave.py]</code>.  Your avatar should raise up their left arm, and move it slowly back and forth.
The <code>[https://github.com/secondlife/leap/blob/main/puppetry/examples/arm_wave.py arm_wave.py]</code>  movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level.   That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.
 
The <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/examples/arm_wave.py arm_wave.py]</code>  movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level.   That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.


The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint.  The Viewer uses {{Wikipedia|Inverse Kinematics}} (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation.  The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view.  Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.
The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint.  The Viewer uses {{Wikipedia|Inverse Kinematics}} (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation.  The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view.  Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.
Line 24: Line 23:
The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.
The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.


Some of the selections on the Puppetry menu are placeholders and don't do much now.  The "Face " and "Fingers " checks, for example, are intended to enable or disable Puppetry animations for those body parts, but aren't functional yet.
Some of the selections on the Puppetry menu are placeholders and don't do much now.  The "Face" and "Fingers" checks, for example, are intended to enable or disable Puppetry animations for those body parts but aren't functional yet.


The <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/webcam/webcam_puppetry.py leap/puppetry/webcam/webcam_puppetry.py]</code>  script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data.  You may need to select different camera numbers from the menu to get this feature to start.   This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be.  It's a great hint, however, of the potential of this new feature.
The <code>[https://github.com/secondlife/leap/blob/main/puppetry/webcam/webcam_puppetry.py leap/puppetry/webcam/webcam_puppetry.py]</code>  script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data.  You may need to select different camera numbers from the menu to get this feature to start.   This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be.  It's a great hint, however, of the potential of this new feature.


 
== Known Issues - The Broken Bits ==
<h2>Known Issues - The Broken Bits</h2>


This feature is experimental and sometimes experiments go wrong and fail and crash and burn
This feature is experimental and sometimes experiments go wrong and fail and crash and burn
.  


* Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
* Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
* The project viewer occasionally crashes when using Puppetry
* The project viewer occasionally crashes when using Puppetry
* Correct hand tracking by <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/webcam/webcam_puppetry.py webcam_puppetry.py]</code> is still a work in progress
* Correct hand tracking by <code>[https://github.com/secondlife/leap/blob/main/puppetry/webcam/webcam_puppetry.py webcam_puppetry.py]</code> is still a work in progress
* Lip, facial, and finger Puppetry requires a bento avatar
* Lip, facial, and finger Puppetry requires a [[Project Bento Skeleton Guide|Bento]] avatar
* IK does not work well on fingers
* IK does not work well on fingers


 
== LEAP Puppetry API ==
 
<h2>LEAP Puppetry API</h2>


This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer.    The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.
This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer.    The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.
Line 49: Line 43:
Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.
Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.


<br>
Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the {{code|LLPuppetModule}} constructor.  There are only two:  '''set''' and '''get'''.
Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the {{code|LLPuppetModule}} constructor.  There are only two:  '''set''' and '''get'''.


Outbound commands are sent by the viewer to the plug-in.  They are defined in <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/__init__.py puppetry.py]</code> with the @registerCommand decorator.  As of this writing, they are: '''stop''', '''set_camera''', '''enable_parts''', and '''set_skeleton'''
Outbound commands are sent by the viewer to the plug-in.  They are defined in <code>[https://github.com/secondlife/leap/blob/main/puppetry/__init__.py puppetry.py]</code> with the <code>@registerCommand</code> decorator.  As of this writing, they are: '''stop''', '''set_camera''', '''enable_parts''', and '''set_skeleton'''


Please note that plug-ins which issue blocking commands (taking too much time) will drop LEAP commands. In the examples, <code>[https://bitbucket.org/lindenlab/leap/src/main/puppetry/webcam/webcam_puppetry.py webcam_puppetry.py]</code> contains code working around this issue while waiting for the startup of the pose recognition software.
Please note that plug-ins which issue blocking commands (taking too much time) will drop LEAP commands. In the examples, <code>[https://github.com/secondlife/leap/blob/main/puppetry/webcam/webcam_puppetry.py webcam_puppetry.py]</code> contains code working around this issue while waiting for the startup of the pose recognition software.


<h2>Inbound Commands  (plug-in to viewer)</h2>
== Inbound Commands (plug-in to viewer) ==


{| class="wikitable" style="margin:auto"
{| class="wikitable" style="margin:auto"
Line 90: Line 83:
| '''scale''' || '''s''' || '''[x, y, z]''' || Array of three floats representing Cartesian scale to be applied to Joint's '''position''' in its parent-frame, and also to Joint bone length
| '''scale''' || '''s''' || '''[x, y, z]''' || Array of three floats representing Cartesian scale to be applied to Joint's '''position''' in its parent-frame, and also to Joint bone length
|}
|}
<h2>Outbound Commands (viewer to plug-in)</h2>


== Outbound Commands (viewer to plug-in) ==


{| class="wikitable" style="margin:auto"
{| class="wikitable" style="margin:auto"
Line 109: Line 102:
| Fingers || 0x10  
| Fingers || 0x10  
|-
|-
| '''set_camera''' || Contains the tag camera_id which is an integer which identifies which camera to use.   This matches the Puppetry menu selection.  
| '''set_camera''' || Contains the tag <code>camera_id</code> which is an integer which identifies which camera to use.   This matches the Puppetry menu selection.  
|-
|-
| '''set_skeleton''' || Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end.  The trailing end position is typically but not always the parent-relative position of the immediate child.Note: mPelvis is the root of the avatar and thus does not have a parent-relative position or parent_id Example mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)  
| '''set_skeleton''' || Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end.  The trailing end position is typically but not always the parent-relative position of the immediate child. '''Note:''' <code>mPelvis</code> is the root of the avatar and thus does not have a parent-relative position or <code>parent_id</code>. '''Example:''' mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis) Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)  
|-
|-
| '''stop''' || No params, ends plug-in puppetry control of avatar.  
| '''stop''' || No params, ends plug-in puppetry control of avatar.  
|}
|}


 
== Show Bones ==
<h2>Show Bones</h2>


The Develop Menu's Avatar options includes "Show Bones", which draws lines showing the position and length of the skeleton parts.  Any bones that are moved with Puppetry data will be drawn with dark blue.  By turning on this option, you can observe how the bones are actually moving.  This helps demonstrate how your avatar's rigging works to alter the actual appearance.
The Develop Menu's Avatar options includes "Show Bones", which draws lines showing the position and length of the skeleton parts.  Any bones that are moved with Puppetry data will be drawn with dark blue.  By turning on this option, you can observe how the bones are actually moving.  This helps demonstrate how your avatar's rigging works to alter the actual appearance.


[[File:ShowPuppetryBones.png|600px|center]]
[[File:ShowPuppetryBones.png|600px|center]]

Latest revision as of 03:00, 26 January 2024

Not to be confused with Puppeteering.

How Puppetry Works

The new Puppetry technology in Second Life is intended for real-time animation control of avatars without the need for pre-computed animation assets.  A Puppetry Viewer runs a plug-in which supplies joint movement instructions.  The movement data is converted to animation format and applied to the local Avatar, streamed to the Second Life server, and relayed to other nearby Viewers who apply the same animation conversion.

Puppetry is still an experimental feature and is not yet deployed everywhere.  It is only available when using a Puppetry Viewer connected to a Puppetry enabled Region.  The Viewer can be obtained from the alternative viewer download page, or since it is open-source it can be built from scratch.  The Puppetry feature is only enabled on the Preview grid in a few regions: Bunraku, Marionette or Castelet.

Try It Out

Example Puppetry plugins can be found in the LEAP git repository.  Clone that repo to your local filesystem, or download it as a zip file, and follow the instructions in the puppetry README.md for installing required Python modules.

Start up the Puppetry Viewer, log into the Preview grid and go to one of the aforementioned regions.   Open up the Advanced menu, and you should see Puppetry listed there.   Open that sub-menu (you may want to tear off that menu - click on the sub-menu top bar and drag it to a spot on your screen)

Howitworks.png

Select Launch plug-in... and navigate to the puppetry modules in the LEAP repository you downloaded to your local filesystem earlier.   In  your copy of the LEAP repository find leap/puppetry/examples/ directory and select arm_wave.py.  Your avatar should raise up their left arm and move it slowly back and forth.

The arm_wave.py  movement might not look exciting, and the wrist might be bent strangely, but it is new at a very fundamental level.   That Python program you selected is running on your local machine and is moving your avatar from outside the Second Life application.

The plug-in supplies data as a series of Puppetry events. Each event is a target position and/or orientation for a named joint.  The Viewer uses "Wikipedia logo"Inverse Kinematics (IK) to compute the transforms of the connecting bones that would allow the named joint to reach its goal (or get as close as possible) and blends that data onto the Avatar like an animation.  The Puppetry events are streamed to the Second Life server which relays them to other nearby Viewers who apply the same logic to animate the Avatars in view.  Anyone standing nearby will be able to see your Puppetry animation, but only if they are also running a Puppetry Viewer.

The "Send" and "Receive" menu checkboxes control whether your Viewer sends and receives Puppetry data: each direction of the data stream can be enabled/disabled independently.

Some of the selections on the Puppetry menu are placeholders and don't do much now.  The "Face" and "Fingers" checks, for example, are intended to enable or disable Puppetry animations for those body parts but aren't functional yet.

The leap/puppetry/webcam/webcam_puppetry.py  script is another plugin which uses several python modules to capture video from your webcam and try to compute Puppetry events to provide real-time Puppetry data.  You may need to select different camera numbers from the menu to get this feature to start.   This is a complex module, and the performance isn't as accurate or as smooth as we would like it to be.  It's a great hint, however, of the potential of this new feature.

Known Issues - The Broken Bits

This feature is experimental and sometimes experiments go wrong and fail and crash and burn

  • Your avatar may not look best as it's manipulated by a plug-in, or positioned with Inverse Kinematics.
  • The project viewer occasionally crashes when using Puppetry
  • Correct hand tracking by webcam_puppetry.py is still a work in progress
  • Lip, facial, and finger Puppetry requires a Bento avatar
  • IK does not work well on fingers

LEAP Puppetry API

This describes how a Puppetry plug-in connects and exchanges information with the Second Life viewer.    The LEAP (LLSD Event API Plug-in) layer provides event-driven communication between the Second Life Viewer and another application (the plug-in) on the same host.

Puppetry LEAP commands are defined in two locations which this document shall refer to as inbound and outbound communication relative to the viewer.

Inbound commands are sent by the plug-in to the viewer and may be found in the viewer source code in the LLPuppetModule constructor.  There are only two:  set and get.

Outbound commands are sent by the viewer to the plug-in.  They are defined in puppetry.py with the @registerCommand decorator.  As of this writing, they are: stop, set_camera, enable_parts, and set_skeleton

Please note that plug-ins which issue blocking commands (taking too much time) will drop LEAP commands. In the examples, webcam_puppetry.py contains code working around this issue while waiting for the startup of the pose recognition software.

Inbound Commands (plug-in to viewer)

Command Description
set Update Puppetry configuration with new values. (e.g. {'command':'set','data':{'inverse_kinematics':{'mWristLeft':{'position':[x,y,z]}}}})
get Query Puppetry for info: (e.g. {'command':'get','data':["camera", "skeleton"]})

There are two modes for setting joint transforms inverse_kinematics (avatar-frame) and joint_state (parent-frame). They can be used simultaneously, even on the same joint, because they represent different things.

Mode Terse name Description
inverse_kinematics i Use to set position and rotation of a Joint's IK target transform in the avatar-frame
joint_state j Specify position, rotation, and scale of Joint in its parent-frame

In either mode you can specify the position or rotation of the joint. In the parent-frame (joint_state) mode you can also adjust the scale.

Param Terse name Format Description
position p [x, y, z] Array of three floats representing Cartesian position
rotation r [x, y, z] Array of three floats representing the imaginary (xyz) components of a unitary Quaternion whose real component (w) is positive
scale s [x, y, z] Array of three floats representing Cartesian scale to be applied to Joint's position in its parent-frame, and also to Joint bone length

Outbound Commands (viewer to plug-in)

Command Description
enable_parts Passed an integer representing a mask of which parts of the skeleton are to be controlled by the script.  This matches the Puppetry menu selections.
Head 0x01
Face 0x02
Left Hand 0x04
Right Hand 0x08
Fingers 0x10
set_camera Contains the tag camera_id which is an integer which identifies which camera to use.   This matches the Puppetry menu selection.
set_skeleton Contains the list of joints in the current skeleton with the joint ID and parent ID, the normalized parent-relative position of this joint and the normalized position of its trailing end.  The trailing end position is typically but not always the parent-relative position of the immediate child. Note: mPelvis is the root of the avatar and thus does not have a parent-relative position or parent_id. Example: mJoint:Joint_id: int Parent_id: int Tip_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis) Parent_relative_position: 3 floats X,Y,Z (in arm-length normalized space relative the pelvis)
stop No params, ends plug-in puppetry control of avatar.

Show Bones

The Develop Menu's Avatar options includes "Show Bones", which draws lines showing the position and length of the skeleton parts. Any bones that are moved with Puppetry data will be drawn with dark blue. By turning on this option, you can observe how the bones are actually moving. This helps demonstrate how your avatar's rigging works to alter the actual appearance.

ShowPuppetryBones.png