UE 5 Hand Tracked Finger Texture Blending
In an effort to make VR hands a little more interactive I thought to try using hand tracking in conjunction with a material that could mask each finger on the hand and blend normal and displacement maps only for the fingers that are bending.
The main hurdle on the hand mesh was in masking 5 fingers with only 4 vertex colors, I achieved this by having both the thumb and pinky use vertex channel red but multiplying that by a vertex channel alpha on the pinky so that it could be masked separately from the thumb, I also chose to add each finger mask together to eventually blend the entire hand textures when all fingers and thumb are bent into a fist.
The material setup was straight forwards, using the result of all the masks to lerp between normal and displacement maps to visualise the creases deforming the glove as fingers bent. A nice detail was that the displacement maps could move the vertices along their normals both positive and negative to push the mesh out and in where creases formed.
Blueprints gathered the local rotation for the top bone of every finger and thumb on begin play, set dynamic material instances for each glove and then each frame compared the current local rotation of these bones against the idle rotations gathered on begin play, a remap value let me set a custom 0 to 1 range from custom rotation min and max angles, these were fed into the specific parameters for each finger in the dynamic material instances which in turn updated the texture blending in the material.
Though the effect is a little subtle in my example I'm sure it could be put to much more visual use on more wild designs, I'd also love to know if I could optimise this for better performance.
Captured footage of the hands showing the texture blending and the finger masks. Apologies for the less than stellar quality.
Raw capture from Unreal showing both hands tracked and texture blending in more detail.
Substance Painter renders of the gloves, open at the top and closed at the bottom.
In engine screenshot from the pawn blueprint showing the world position offset on the larger creases over the back of the hand.
Zbrush sculpt of the closed hand, it's a little odd sculpting creases formed only when a glove is closed to an open hand.
Zbrush sculpt of the open hand with creases common on the back of the hand where the fabric creases more as it tightens on the inside.
Vertex color red used to mask the thumb and pinky, Green for the index finger and blue for the middle finger.
Vertex color alpha used to mask the ring finger and the pinky finger by multiplying against vertex color red.
Material overview showing the vertex color masks in the bottom left taking in parameters for the strength of each mask, these all add together to lerp normal and displacement textures.
Creating dynamic material instances for both hands on event begin play to feed floats for the finger masks from the Pawn Blueprint to the materials.
On begin play getting and setting the original rotations of bones to compare later against the same bones to determine how much they have rotated.
Getting the local rotation of the top bone with inverse transform rotation compared against the original rotation and outputs a 0>1 to the material parameter needed.