Digital Human & Photorealistic Character

 

作为Epic的角色探索的一部分，Epic从照片真实感的角色（Photorealistic Character ）半身像开始开发数字人类，已经进行了许多渲染改进，以开发逼真可信的角色。

 

在开发这些角色的同时，对皮肤、眼睛、照明和次表面散射（Subsurface Scattering）进行了以下渲染改进。

• 使用双贝克曼双瓣法（Double Beckman Dual Lobe method）添加了一个新的镜面反射模型（Specular model ）。
• 利用背散射（Backscatter ）进行地下剖面的光传输（Light Transmission）。
• 使用边界出血颜色（Boundary Bleed Color）的次表面散射（Subsurface Scattering）提供更好的接触阴影（contact shadowing）。
• 通过后期处理材质（Post Process Materials）进行短距离动态全局照明（Short Distance Dynamic Global Illumination）。
• 使用单独的虹膜法线贴图（separate normal map for the Iris）为眼睛添加细节。

 

For additional information, see Digital Humans.

 

Images and characters provided by 3Lateral for Siren, Andy Serkis, and Osiris Black©.

Previously we demonstrated approaches to characters, like the ones used in Epic’s own game Paragon, with the Photorealistic Character example. Since that time, we’ve explored and improved on how we can design more believable characters through our approach to high-quality character shader techniques, using data scans of Actors, improving our geometry workflow, and ultimately improving Unreal Engine 4’s (UE4) rendering capabilities. 

As part of our character explorations, we developed several Digital Humans (and a Non-Human) to drive our workflow and rendering improvements to produce highly realistic and believable characters. Below you’ll find high-level overviews of our workflow processes and newly implemented features.

Also keep in mind that these features and techniques are not explicitly limited to only high-end machines. They can be applied to many real-time games and projects with good quality and FPS achievable.

To take a look at some of these improvements yourself, you can download the Digital Humans project from the Epic Games Launcher under the Learn tab. This project includes the original Photorealistic Character bust from Paragon along with the Mike Seymour Digital Human first shown at SIGGRAPH 2017.

 

Digital Human Performances

https://youtu.be/6MIkoLBWRv0

https://youtu.be/9owTAISsvwk

https://youtu.be/mkkWCmljMSA

Improved Character Rendering Features

Over the course of development of Digital Human characters, Epic has developed or improved the following features: 

• Using a new specular model.
• Light Transmission using Backscatter.
• Better contact shadowing for Subsurface Profiles.
• Short Distance Dynamic Irradiance for Screen Space Indirect Bounce approximation through Post Process Materials.
• Added detail for eyes using a separate normal map for the Iris.

Each of the sections below gives detail for these newly developed features or improvements with their own sections for Skin, Hair, and Eyes. 

 

Skin Shading

The character's skin uses UE4's Subsurface Profile shading model. 

Note that these projects use Material Functions to set up the base for the skin material. There are a couple of reasons for taking this approach:it provides reusable authoring for materials, and we can use masks to blend animations with the material as it changes. With Material Functions, artists can standardize their approach to development because when a change is made to a single function, all Materials that use that function automatically reflect the change.

You can take a look at some of the Material Functions for the Mike Seymour bust in /Content/Characters/Mike/Functions.

Skin Shading Techniques

To maintain believability and high-quality realism, the Digital Humans all use 4K texture resolution produced from facial scans of the subject, in this particular example, Mike Seymour. After acquiring the digital scans, the textures are cleaned up, tweaked, or generated by Artists at Epic.

 

The skin setup for characters can use various textures along with masks to achieve the desired look. The textures shown here are the core textures used to achieve the look of our Digital Humans: diffuse (albedo), roughness, specularity, scatter, and normal maps.

Texture	Name	Description
	Diffuse	The Diffuse map supplies the Base Color for the Material. At a 4K texture resolution, you can see the tiny capillaries just underneath the surface of the skin. Any darkening of wrinkles will accentuate the texture supplied by the Normal map. (T_head_diffuse_neutral)
	Roughness	The Roughness map is not painted by an Artist. Instead, we’re using a process in UE4 to convert the normal to roughness. (Toksvig_MikeMesoNormal) See Roughness Maps below for additional information about this process and how it benefits our animated digital humans.
	Specular	The Specular map scales the amount of specular highlight visible across the surface of the skin. It is important to note that the default value for specularity is 0.5. This map boosts specularity in areas where the skin may be stretched a bit tighter or dampens it in areas where we don’t want to see reflection, such as the center of the pores and within wrinkles. (Mike_T_specular_neutral)
	Scatter	The Scatter map controls how much light is going to be scattered through the surface of the skin. Areas that are dark will exhibit very little scatter, such as the cheeks, while lighter areas will have higher amounts of perceived scatter, such as the nose, eyes, and ears. You’ll notice that this is a simple mask to for areas around the ears, eyes, nose, and lips that we want to have the most amount of scattering. Other areas are darker but still light enough to have scattered, like real-world skin. The color of the scatter is managed by your Subsurface Profile asset assigned to your Skin material. (T_head_sss_ao_mask)
	Normal	The Normal map works by supplying the tactile texture to the surface by perturbing pixel normals. For our Digital Humans, there are several normal maps that are blended between and combined in the Material setup to capture the medium-sized and micro-sized detail of the skin. See Normal Maps below for additional information about how we use those and blend between different ones for different animation poses of the face. (head_normal_map_001)

Dual Lobe Specular

The Dual Lobe Specularity provides roughness values for two separate specular lobes that are combined for the final result. When these are combined these provide nice subpixel micro-frequencies in the skin for a natural look. 

 

 Softer Lobe

 Tighter Lobe

 Combined Lobes

 This view is using a single Light with a Black base color plugged into to see blended Lobes.

 In the Subsurface Profile Asset under the Dual Specular section, you can adjust the roughness of each specular lobe and how much they should mix.

 

Light Transmission using Back Scatter (Forward Scatter)

Lights now support Transmission through surfaces. This has been referred to as backscatter because it’s a backlight effect. However, it’s actually a “forward scatter” because the light is being scattered toward the view. We use a single scatter approach that samples depth from shadow maps with exponential falloff. Then a Henyey Greenstein phase function is used with the index of refraction (IOR) from the surface on that function to add a little more shaping to the scatter.

 
 
 Light Transmission Disabled                                     Light Transmission enabled

 Once you’ve set up your Subsurface Scattering profile and assigned it to your Material, you can enable Transmission on a Light to have a backlit effect by looking under the Light section in the Advanced rollout properties and enabling Transmission.

 

 
 

You can use masks in your Material to control how much transmission is allowed for scattering by blocking similarly to how you would use a mask for opacity or roughness areas. Then, in your Subsurface Scattering profile, you can control transmission with these properties:

 

Boundary Color Bleed

For areas around the mouth and eyes where we use scattering for the teeth and eyeballs, we use a new property called Boundary Bleed Color. It defines how one subsurface material blends into another one. 

 

 Without Boundary Bleed Color

 with Boundary Bleed Color

 Subsurface scattering is a screen space effect that would often cause halo-type artifacts in areas with other subsurface materials, similar to the example above where some light areas from the lips blend into areas around the teeth.

 

Short Distance Dynamic Irradiance

Another improvement for digital human characters was to reduce the amount of darkened colors (blacks and grays) there were in corners around the nose and eye sockets. To make the look more natural, a screen space indirect bounce approximation technique was used. A Post Process Material is used for short distance dynamic global illumination, improving the lighting and believability by capturing the indirect bounce from the cheeks to the nose to the corners of the eyes. 

 

 without Screen Space Irradiance

 with Screen Space Irradiance

 Screen Space Irradiance is set up as a Post Process Material, so it can be used for anything it affects (see example below). You can take a look at the Material in the Digital Humans project located in Content\Shared\UtilityMaterials in the ScreenSpaceRealIrr Material.

 

 

 Screen Space Irradiance Post Process Material casting short distance dynamic global illumination from emissive materials.

Hair Shading

Hair shading is handled by way of UE4’s Hair Shading Model. This shader is a physically-based shading model based on research by Eugene d'Eon, Steve Marschner, and Johannes Hanika and currently used by Weta Digital. The shader approximates the reflection of light from the surface of the hair with anisotropic specularity, the refraction of light through the surface, and the scatter of light through multiple strands. 

 

To use the UE4 Hair shader, set the Shading Model property of your Material to Hair.

 

 

 

Hair and Specularity

In the real world, hair tends to have multiple specular highlights: one representing the color of the light and another that is a mix of hair color and light color. For sake of this document, we will refer to these as primary and secondary specular highlights. The Hair shader approximates the same effects with highly realistic results.

 

 

 

1. The brighter brown hair represents the secondary specular highlight.
2. The lighter hair represents the primary specular highlight. 

 

The approximation algorithm used in UE4’s hair shader creates these effects similarly to how they are made in the real-world. As light strikes the surface of a hair, it does not simply bounce off. Hair is translucent, allowing some light to pass through it, to potentially bounce around inside it, then exit. The hair shader approximates this with three possible paths for the light to travel as it interacts with the hair, as shown in the diagram below:

 

 

 

 Cross section of single hair strand showing how the Hair shader approximates interaction with light, demonstrating the three primary types of paths light will take. See the table below for descriptions of each part of the process 

 

 

Number	Description
0	The growth (root to tip) direction of the hair follicle.
1	Reflection-only path, with light bouncing off the surface of the hair. This produces the Primary specular highlight.
2	Transmission-Transmission path, in which light passes into the hair follicle and back out the other side. This is how light scatters within a volume of hair.
3	Transmission-Reflection-Transmission path, in which light enters the hair follicle, reflects off the inside boundary of the surface and then exits. This produces the Secondary specular highlight.

 

As shown in the diagram, a strand of hair is not a perfect cylinder or tube. In actuality, hair appears more as a series of stacked cones. This means light bouncing off the surface of the hair will be scattered differently than it would be if the hair were perfectly smooth. Further, because each strand of hair is generally pointing in a different direction, the specular highlight will not be unified but will be independently placed based on the direction the hair is pointing. This is often referred to as anisotropic specularity, which is also supported by the UE4 Hair shader. 

Eye Shading

  

 

 Eyes are often referred to as “the windows to the soul.” Few aspects of a character have as much influence on the viewer’s perception of a character as a real person as the character’s eyes. The Eye shader in UE4 is designed to bring realistic rendering of an eye surface, exposing artistic control over each of the biological parts of the eye itself.

 The Eye shader in UE4 is highly technical and has been developed such that there are very strong dependencies between the shader code, the Material, the shape of the geometry, and the geometry’s UV layout. Building an eye from scratch using this shader is not recommended without extensive experience in shader development. To that end, we strongly recommend, for those creating realistic humanoid eyes, that you extract the eye geometry from this example and use the eye Material as-is, replacing the necessary textures to suit your purpose.

 Located within the Photoreal Character Project is a Static Mesh named PhotorealEyeGeo. This can be exported from the project into the DCC app of your choice via FBX to be used in your own projects.

 

Eye Biology

In order to fully understand the features of the eye, it is best to know a little bit of eye biology. Each of the parts you see here is represented in the eye setup, and all can be modified in some way using the exposed parameters in the Material Instance applied to the example character’s eyes (MI_EyeRefractive_Bust).

Here’s a quick refresher:

 

Number	Name	Description
1	Sclera	The sclera is also known as the “white of the eye.” This area of the eye is generally very wet and has a slight amount of tactile textures.
2	Limbus	The limbus is the dark ring that is present between the iris and the sclera. It is more dominant in some eyes than others, and it will tend to fade away when viewed from edge-on.
3	Iris	The iris is the ring of color around the center of the eye. When one is said to have “green” eyes, it is because their iris is predominantly green. In a real eye, the iris is a ring of muscle-like fibers that expand and contract, allowing more light into – or closing light out of – the pupil. It is also worth noting that in the real world, the iris is actually more of a disc or cone shape, and does not bulge outward with the rest of the eye.
4	Pupil	The pupil is the dark spot at the center of the eye. It is the hole through which light passes so it can be picked up by the rods and cones of the retina.
5	Cornea	The cornea is the clear, fluid-filled dome that rests over the surface of the iris.

Refraction within the Eye - Then and Now

Because the eyeball is filled with fluid, it is given to refracting any light that passes through it. This can be seen in the real world when looking at an eye from multiple angles. The iris and pupil will be distorted by refraction as they are viewed through the cornea. The traditional approach to solving this for games and film has been to create two separate eye surfaces; one that supplied the sclera, iris, and pupil; and another surface on top that supplied the cornea and overall wetness of the eye. This allowed for refraction as the underlying surface was viewed through the wet layer shell. Such an approach was used in Epic’s A Boy and His Kite tech demo on the boy’s eyes. The diagram below shows a representation of the two surfaces.

 

By using the Eye shading model, an even more realistic effect can be achieved with a single surface. The refraction of light through the surface of the cornea is now handled entirely within the shader, alleviating the need for underlying geometry. You can experiment with this yourself by opening M_EyeRefractive Material Instance and adjusting the Refraction On/Off property.

 

Eye Shader Parameters

Due to the complexity of the Eye shader and our recommendation to use it as-is, we have included a breakdown of the artistic parameters that are built into the Material. The following are the eye shader parameters that can be tweaked from within MI_EyeRefractiveMike_R Material Instances. Wherever possible, we will also include suggested parameter ranges.

 

Name	Description
Static Switch Parameter Values
Use Eye Bulge	Adjusts the refraction properties of the shader based on whether or not the eyeball geometry uses a physically modeled bulge for the cornea.
Vector Parameter Values
Eye Corner Darkness Color	Sets a color value for the eye color darkness, which is controlled using the scalar variables for Shadow Hardness and Shadow Radius.
Scalar Parameter Values
Depth Scale	This controls the depth of the refraction of the iris underneath the cornea.
Flatten Normal	This value controls how much flattening of the eye's normal map is taking place, focused particularly on the sclera.
Index of Refraction (IOR)	Index of refraction of the fluids underneath the cornea. Controls how much refraction takes place.
Iris Concavity Power	Used alongside Iris Concavity Scale to control the shape and amount of light caustics that are calculated on the surface of the iris as light passes through the cornea. This will generally only be visible in an actual lit scene, and can be difficult to visualize within the Material Instance Editor.
Iris Concavity Scale	Used alongside Iris Concavity Power to control the shape and amount of light caustics that are calculated on the surface of the iris as light passes through the cornea. This will generally only be visible in an actual lit scene, and can be difficult to visualize within the Material Instance Editor.
Iris UV Radius	Controls the overall size of the iris on the eyeball.
Iris Brightness	Controls the brightness of the iris.
IrisDispScaleUV	Scales the displacement of the iris when using an iris normal map.
Iris Roughness	This value drives how shiny the cornea is (the surface directly over the iris).
Limbus Dark Scale	Controls the size of the darkening ring of the limbus.
Limbus Power	Controls the overall darkening of the limbus area. Overdriven values will darken the entire corneal area.
Limbus UV Width Color	Controls the sample size for the limbus, or how much of the eye surface will be alloted for showing the limbus. This is a fairly delicate setting and should be adjusted with care.
Limbus UV Width Shading	Controls how much light will affect the shading of the limbus. Very delicate setting, adjust with care.
Normal UV Scale	Controls the scale of the normal map texture used across the surface of the eye.
Pupil Scale	Controls the size of the pupil. This is the setting you would use to dilate the eyes.
Refraction On/Off	Blends between refracting and non-refracting versions of the shader.
Scale By Center	Adjusts the scale of the entire iris/pupil area from its center.
Sclera Brightness	Controls the brightness of the sclera, or white of the eye.
Sclera Roughness	Controls the Material Roughness value of the sclera.
Shadow Hardness	Controls the sharpness of the blend between the sclera inner and outer color. Used with Shadow Radius to help drive shading across the surface of the sclera, approximating the effect of the eyelid casting subsurface scattered shadows across the surface of the eyeball.
Shadow Radius	Controls the size of the blend between the inner and outer colors of the sclera. Used with Shadow Hardness.
Specularity Iris	Controls specularity level across the cornea (iris and pupil).
Specularity Sclera	Controls the specularity level across the sclera.
Texture Parameter Values
Iris Color	See Eye Textures below.
Iris Normal Map	See Eye Textures below.
Sclera Color	See Eye Textures below.

 

 

Eye Iris Normal

For eyes, there is now support for a Dual Normal to add additional detail and light directionality to the eye through the Iris Normal.

 iris normal disabled

 iris normal enable

 

The eye in this shot is also using Boundary Bleed Color improvements to soften the illumination and light leaking between materials.

The iris normal can be enabled by adding the following console variable to your DefaultEngine.ini:

r.IrisNormal=1

 

The normal for the iris is fed into the Clear Coat Bottom Normal node in the Material. 

 

Eye Textures

Many of the textures you’ll create for the eyes will be based on the UV layout created above, with the exception of the Iris map. The list of primary editable textures includes:

 

Texture	Name	Description
	Sclera Map	The sclera map controls the colors of the eye whites, as well as the vein and tissue color that will be present at the edges of the eye. If you wanted your character to be able to have bloodshot eyes, for example, this is the texture you would modify. (Sclera_Tweak_R)
	Mid Plane Displacement Map	The mid-plane displacement map is used to lock down a plane that cuts through the center of the eye, and is used as a basis for offsetting the depth of the iris. (T_EyeMidPlaneDisplacement)
	Normal Map	The normal map does the same job as normal maps in all shaders – it supplies tactile texture variation. In the case of the example character eye setup, it produces tiny lumps and bumps along the wet surface of the eye. (iris08_leftEye_nml)
	Tangent Map	The tangent map is used to control the direction the surface is flowing in tangent space. In the case of the example character’s eye setup, this is used to accentuate the change in direction where the cornea meets the sclera. (T_Eye_N)
	Iris Map	The iris texture is unique in that it does not match the UV layout. Instead, the texture fills the entirety of the resolution. UVs and Alpha masks control the overall iris size, as well as the size of the pupil. If making your own iris maps, try to keep the pupil size reasonably close to the proportions in the supplied texture. (eye_iris_clr)

 

Note that the Tangent map is not fed directly into the Normal map input. Instead, it is connected to the Tangent Output node which must be created separately, as seen in the master Material.

 

Eye Soft Ambient Shadowing

There are two ways to approach soft ambient shadowing for the eye so that it feels natural and believable. In the real world – wherever you eye comes in contact with other tissues, such as the eyelid or the tear duct (caruncle) – you will see very soft shadowing. This softness is due to the fact that the eye itself is translucent, allowing light to scatter through it. To help simulate this effect, you can use one of the following methods using a thin piece of geometry under the eyelid or enable contact shadowing for translucent materials. You may find that one method works better than the other depending on your own characters and the look you are developing.  

Note that both of these methods use Boundary Bleed Color to soften areas around the eye - just like the mouth - to reduce haloing artifacts from subsurface scattering screen space limitations.

Eye Ambient Occlusion

One way to help simulate realistic shadowing for the eye is to place a thin sheet of geometry that covers the eye. It provides approximated occlusion, creating a richer degree of realism across the eye.

The Material applied to this sheet is named MI_Eye_Occlusion_Mike and can be found in Material Element 17 slot of the example character model.

 Note that we did not use this method with Mike’s bust. Instead we developed and used Contact Shadows for Translucent Materials, which we felt worked better for this character. You can enable Contact Shadows by applying a value to the variable Shadow Amount in the Material Instance.

 

 

 The eye geometry used for ambient shadowing.

 If authoring your own content for character eyes, you will very likely need to build this piece of geometry yourself. Note that it should be made to fit the area of the open eyelid and that its UVs should cleanly fill the 0-1 UV space in your 3D modeling application.

 

Contact Shadows for Translucent Materials

Translucent Materials can enable Contact Shadows. This uses similar functionality as the Contact Shadows for Lights but it's not linked to the light's contact shadow parameter. This is a screen space effect and can be used in addition to or in place of geometry to make the eye feel seated in the socket for believability. 

 

 

 

 You can enable Contact Shadows in your material Details panel under the main Material node's Translucency tab:

 

 

Material Techniques

Below are some key takeaways that we learned while working on creating the look of our digital humans. 

 

Normal Maps

There are two types of Normal maps used to create the look of realistic and believable skin for characters; Meso (or medium-sized) normals and Micro normals.

 

Meso Normals

Meso normals represent the medium-sized details of the skin like the primary wrinkles around the eyes, lips, and forehead. Multiple meso normals are used and animated with different animated poses.

 
The poses are split into several different normal maps (see below). They are driven by the animation so that when the subject is acting or running an animation, the correct normal is triggered and blended between the different poses. This helps to create a realistic and believable result in the process. 
 

 

 The poses and masked normal maps driven by Andy Serkis’ facial capture performance.

Micro Normals

Micro Normals represent the small details of the skin such as pores and small imperfections. These details are extracted in one of two ways: from scan data of the subject, or from the Specular map.

 

Roughness Maps

The Roughness map was generated from the Normal map, using a technique called "Normal to Roughness." Inside the Texture Editor, you can apply any texture to the Composite Texture slot and set the mode of the texture so that it becomes the basis for the roughness.