GDC 2014 - Deformable Snow Rendering in Batman: Arkham Origins

1 / 43Deformable Snow Rendering in BatmanTM: Arkham Origins Colin Barré-Brisebois (Lead Rendering Programmer)

2 / 43Agenda  Motivations  Deformable Snow ●くろまる Novel technique for rendering of surfaces covered with fallen deformable snow ●くろまる For consoles and enhanced for PC (DX11 tessellation)  Q&A

Most read

3 / 43

4 / 43

5 / 43Motivations  Enhance the world with dynamics of deformable snow  Three requirements: 1. Iconic visuals of deformable snow 2. Organic deformation from walking, falling, sliding, fighting and more 3. Low memory usage and low performance cost for an open world game

6 / 43Iconic / Organic Deformable Snow From Google Images - http://bit.ly/M7T9kV (footsteps in snow, left) and http://bit.ly/M7TbJB (snow angel, right)

7 / 43Previous Work? [St-Amour 2013] (Assassin’s Creed 3) [Edwards 2012] (Journey)  Raycast on a terrain / Modify terrain mesh. - We don’t have terrain. We have rooftops and streets. - Besides, we don’t want to add raycasts.  Requires variable triangle density for visually convincing vertex displacement in all cases - PC DX11 with tessellation is great... but what about consoles? 

8 / 43Our Approach (1/)  Generate displacement heightmaps at runtime ●くろまる Snow prints are a semi-low frequency detail effect ●くろまる Cheap approximation works with footsteps & more ●くろまる Great performance, and low memory usage  Consoles: virtual displacement via Relief Mapping ●くろまる Minimal taps. No "swimming" ●くろまる Independent of triangle density  PC: DirectX 11 version with tessellation

Most read

9 / 43Our Approach (2/)  Gotham has many rooftops and streets  Dynamically alloc/dealloc heightmaps based on size, player/AIs and visibility Heightmaps

10 / 43Feet Cape

11 / 43Feet Cape

12 / 43 Render snow-affecting objects looking from under the surface using an ankle-high orthogonal frustum 1. Clear to black 2. Render actors in white 3. Filter and accumulate (ping/pong) in a texture  Anything in that zone will affect the heightmap (feet, hands, sliding, throwing a thug to the ground...) Generating the Heightmap ?

13 / 43Ankle-high Orthogonal Frustum

14 / 43

15 / 43

16 / 43

17 / 43Let’s see what it looks like at runtime!

18 / 43Update Loop For every active* snow surface 1. Figure out if surface-affecting object is on the surface -We use a quad tree look-up rather than keeping an actor list for each surface 2. Override materials on all parts - Simple white material 3. Render actors 4. Process/Accumulate with custom post-process chain

19 / 43Heightmap Accumulation & Render  Stage 1 – Get results & small blur ●くろまる 4-tap bilinear Poisson  Stage 2 – Add to existing heightmap ●くろまる During this stage, you can also subtract a small value to the heightmap to make snow gradually replenish (since it’s snowing)   Stage 3 – Shading

20 / 43Stage 3 - Shading (1/)  Snow surfaces have 2 material options 1. Basic Snow Material o Active when surface is not being deformed o Shows new / clean / untouched snow, cheaper 2. Deformable Snow Material o Two stages: non-deformed or fully flattened snow o Non-deformed part the same as Basic Snow Material o Fully flattened shows rooftop tiles / concrete. o Blends both stages using heightmap & Relief Mapping

21 / 43Stage 3 - Shading (2/) Non-deformed Snow Flattened Snow

22 / 43Stage 3 - Shading (3/)  Blending Material Stages ●くろまる For diffuse & spec, simple lerp o Also, tint diffuse with sky color in transition area to fake SSS ●くろまる For normals, blend using Reoriented Normal Mapping [Barré-Brisebois & Hill 2012] o Normals are not colors. o You can’t lerp/overlay between directions! o Used in game to: ●くろまる Blend the snow detail normal and the macro "wave" snow normal ●くろまる Add detail normal maps everywhere

23 / 43Stage 3 - Shading (4/) float3 t = tex2D(BaseNormal, uv) * float3(2, 2, 2) + float3(-1, -1, 0); float3 u = tex2D(DetailNormal, uv) * float3(-2, -2, 2) + float3(1, 1, -1); float3 r = t * dot(t, u) / t.z – u; [Barré-Brisebois & Hill 2012]

24 / 43Linear Interpolation

25 / 43Overlay

26 / 43[Barré-Brisebois & Hill 2012] Reoriented Normal Mapping

27 / 43Add. Implementation Details (1/)  Surface UVs align with ortho frustum ●くろまる 0-1 range, simplifies heightmap-to-displacement  Scaled world-space heightmap res. ●くろまる Min(512, 1⁄4 * (SurfaceX, SurfaceY)) ●くろまる Tries to keep texels "square" ●くろまる Doesn’t need to be high-res, looks better in lower resolutions ●くろまる Must scale Relief Mapping parameters

28 / 43Add. Implementation Details (2/)  Split render & tick of active surfaces ●くろまる Snow surface where Batman stands has priority ●くろまる Only render 2 surfaces/frame (tweakable but good enough, with distance-based priorities)  Reuse memory from old heightmaps ●くろまる Not active/visible (max distance from sphere bounds) ●くろまる Un-streamed open-world zones

29 / 43DirectX 11 With Tessellation (1/)  Feature developed with our friends @ NVIDIA (Evgeny Makarov)  Accurate displacement based on depth ●くろまる Capture the height field like a z-buffer ●くろまる Two channels: o Minimum height field o Projected displacement ●くろまる Allows for additive capture & smoother results. ●くろまる Also allows for deformable snow banks! 

Most read

30 / 43Rooftop Minimum Height field Orthogonal Capture Frustum Projected Displacement Final Surface (displaced) DirectX 11 With Tessellation (2/)

31 / 43DirectX 11 With Tessellation (3/)  Tessellated version adds detailed displacement calculated from the normal map, globally applied to the snow surface ●くろまる Extra detail compared to the relief-mapped version ●くろまる Takes the macro normal map to add additional "macro waves"

32 / 43Without Tessellation (No Macro Deformation)

33 / 43With Tessellation (Macro Deformation)

34 / 43DirectX 11 With Tessellation (4/)  Runtime dicing of snow meshes  Real geometry means: ●くろまる Works with Dynamic Shadows o Character shadows now follows the surface and shift with the deformation o Self shadowing & self-shading ●くろまる Works with dynamic ambient occlusion o AO fills-in the trails

35 / 43

36 / 43Performance & Memory  Performance ●くろまる Heightmaps update < 1.0ms GPU on PS3/360  Memory ●くろまる 2 MB (360 / PS3 / WiiU) o Since we’re using low resolution heightmaps o This is flexible, but sufficient for our needs since we allocate/deallocate as the player flies in the world ●くろまる 2-4 MB (FP16 vs FP32 on PC)

37 / 43Caveats / Issues ?  Relief-Mapped Approach ●くろまる Deformation looks great, but will never be as thick as tessellation. Replace with Parallax Occlusion Mapping? ●くろまる Derive parametric AO from the heightmap?  Tessellated Approach ●くろまる When artists were working on content creation, displacement wasn't taken into account (pre-pass actors, open edges being visible, etc...) ●くろまる Some meshes couldn't use tessellation as there were parts of geometry right under the snow, not supposed to be visible

38 / 43Future Endeavours...  Save the heightmaps and reload them?  Use this technique for other cases, such as sand, mud, etc...

39 / 43Summary  A fast and low-memory footprint technique to render deformable snow surfaces ●くろまる Adds a really nice level of interaction between players and the world ●くろまる Depics iconic & organic visuals of deformable snow  A good tessellation case for your DX11 game using minimal editing and art tweaks

40 / 43Thank You! Érick Bilodeau David Massicotte Sébastien Turcotte Jimmy Béliveau Olivier Pomarez Philippe Bernard Ryan Lewis Marc Bouchard Jean-Noé Morissette Pierric Gimmig Patrick Dubuc Reid Schneider Maggy Larouche Miguel Sainz Evgeny Makarov Jon Jansen Christina Coffin Jon Greenberg NVIDIA Questions? colin.barrebrisebois@wbgames.com / @ZigguratVertigo

41 / 43http://www.wbgamesmontreal.com

42 / 43References [Barré-Brisebois & Hill 2012] Barré-Brisebois, Colin and Hill, Stephen. "Blending in Detail - Reoriented Normal Mapping", 2012. http://bit.ly/Mf2UH0 [Edwards 2013] Edwards, John. "Sand Rendering in Journey", Advances in Real-Time Rendering, SIGGRAPH, 2012. http://advances.realtimerendering.com/s2012/index.html [Policarpo & Oliveira 2006] Policarpo, Fabio and Oliveira, Manuel M. Rendering Surface Details in Games with Relief Mapping Using a Minimally Invasive Approach. In:Wolfgang Engel (ed.). SHADER X4: Lighting & Rendering. Charles River Media, Inc., Hingham, Massachusetts, 2006 (ISBN 1-58450-425-0), pp. 109-119. [St-Amour 2013] St-Amour, Jean-François. "Rendering Assassin's Creed", Game Developers Conference, 2013.

43 / 43GDC 2014 - Deformable Snow Rendering in Batman: Arkham Origins

Deformable Snow Rendering in BatmanTM: Arkham Origins Colin Barré-Brisebois (Lead Rendering Programmer)

Agenda  Motivations  Deformable Snow ●くろまる Novel technique for rendering of surfaces covered with fallen deformable snow ●くろまる For consoles and enhanced for PC (DX11 tessellation)  Q&A

Motivations  Enhance the world with dynamics of deformable snow  Three requirements: 1. Iconic visuals of deformable snow 2. Organic deformation from walking, falling, sliding, fighting and more 3. Low memory usage and low performance cost for an open world game

Iconic / Organic Deformable Snow From Google Images - http://bit.ly/M7T9kV (footsteps in snow, left) and http://bit.ly/M7TbJB (snow angel, right)

Previous Work? [St-Amour 2013] (Assassin’s Creed 3) [Edwards 2012] (Journey)  Raycast on a terrain / Modify terrain mesh. - We don’t have terrain. We have rooftops and streets. - Besides, we don’t want to add raycasts.  Requires variable triangle density for visually convincing vertex displacement in all cases - PC DX11 with tessellation is great... but what about consoles? 

Our Approach (1/)  Generate displacement heightmaps at runtime ●くろまる Snow prints are a semi-low frequency detail effect ●くろまる Cheap approximation works with footsteps & more ●くろまる Great performance, and low memory usage  Consoles: virtual displacement via Relief Mapping ●くろまる Minimal taps. No "swimming" ●くろまる Independent of triangle density  PC: DirectX 11 version with tessellation

Our Approach (2/)  Gotham has many rooftops and streets  Dynamically alloc/dealloc heightmaps based on size, player/AIs and visibility Heightmaps

Feet Cape

Feet Cape

 Render snow-affecting objects looking from under the surface using an ankle-high orthogonal frustum 1. Clear to black 2. Render actors in white 3. Filter and accumulate (ping/pong) in a texture  Anything in that zone will affect the heightmap (feet, hands, sliding, throwing a thug to the ground...) Generating the Heightmap ?

Ankle-high Orthogonal Frustum

Let’s see what it looks like at runtime!

Update Loop For every active* snow surface 1. Figure out if surface-affecting object is on the surface -We use a quad tree look-up rather than keeping an actor list for each surface 2. Override materials on all parts - Simple white material 3. Render actors 4. Process/Accumulate with custom post-process chain

Heightmap Accumulation & Render  Stage 1 – Get results & small blur ●くろまる 4-tap bilinear Poisson  Stage 2 – Add to existing heightmap ●くろまる During this stage, you can also subtract a small value to the heightmap to make snow gradually replenish (since it’s snowing)   Stage 3 – Shading

Stage 3 - Shading (1/)  Snow surfaces have 2 material options 1. Basic Snow Material o Active when surface is not being deformed o Shows new / clean / untouched snow, cheaper 2. Deformable Snow Material o Two stages: non-deformed or fully flattened snow o Non-deformed part the same as Basic Snow Material o Fully flattened shows rooftop tiles / concrete. o Blends both stages using heightmap & Relief Mapping

Stage 3 - Shading (2/) Non-deformed Snow Flattened Snow

Stage 3 - Shading (3/)  Blending Material Stages ●くろまる For diffuse & spec, simple lerp o Also, tint diffuse with sky color in transition area to fake SSS ●くろまる For normals, blend using Reoriented Normal Mapping [Barré-Brisebois & Hill 2012] o Normals are not colors. o You can’t lerp/overlay between directions! o Used in game to: ●くろまる Blend the snow detail normal and the macro "wave" snow normal ●くろまる Add detail normal maps everywhere

Stage 3 - Shading (4/) float3 t = tex2D(BaseNormal, uv) * float3(2, 2, 2) + float3(-1, -1, 0); float3 u = tex2D(DetailNormal, uv) * float3(-2, -2, 2) + float3(1, 1, -1); float3 r = t * dot(t, u) / t.z – u; [Barré-Brisebois & Hill 2012]

Linear Interpolation

Overlay

[Barré-Brisebois & Hill 2012] Reoriented Normal Mapping

Add. Implementation Details (1/)  Surface UVs align with ortho frustum ●くろまる 0-1 range, simplifies heightmap-to-displacement  Scaled world-space heightmap res. ●くろまる Min(512, 1⁄4 * (SurfaceX, SurfaceY)) ●くろまる Tries to keep texels "square" ●くろまる Doesn’t need to be high-res, looks better in lower resolutions ●くろまる Must scale Relief Mapping parameters

Add. Implementation Details (2/)  Split render & tick of active surfaces ●くろまる Snow surface where Batman stands has priority ●くろまる Only render 2 surfaces/frame (tweakable but good enough, with distance-based priorities)  Reuse memory from old heightmaps ●くろまる Not active/visible (max distance from sphere bounds) ●くろまる Un-streamed open-world zones

DirectX 11 With Tessellation (1/)  Feature developed with our friends @ NVIDIA (Evgeny Makarov)  Accurate displacement based on depth ●くろまる Capture the height field like a z-buffer ●くろまる Two channels: o Minimum height field o Projected displacement ●くろまる Allows for additive capture & smoother results. ●くろまる Also allows for deformable snow banks! 

Rooftop Minimum Height field Orthogonal Capture Frustum Projected Displacement Final Surface (displaced) DirectX 11 With Tessellation (2/)

DirectX 11 With Tessellation (3/)  Tessellated version adds detailed displacement calculated from the normal map, globally applied to the snow surface ●くろまる Extra detail compared to the relief-mapped version ●くろまる Takes the macro normal map to add additional "macro waves"

Without Tessellation (No Macro Deformation)

With Tessellation (Macro Deformation)

DirectX 11 With Tessellation (4/)  Runtime dicing of snow meshes  Real geometry means: ●くろまる Works with Dynamic Shadows o Character shadows now follows the surface and shift with the deformation o Self shadowing & self-shading ●くろまる Works with dynamic ambient occlusion o AO fills-in the trails

Performance & Memory  Performance ●くろまる Heightmaps update < 1.0ms GPU on PS3/360  Memory ●くろまる 2 MB (360 / PS3 / WiiU) o Since we’re using low resolution heightmaps o This is flexible, but sufficient for our needs since we allocate/deallocate as the player flies in the world ●くろまる 2-4 MB (FP16 vs FP32 on PC)

Caveats / Issues ?  Relief-Mapped Approach ●くろまる Deformation looks great, but will never be as thick as tessellation. Replace with Parallax Occlusion Mapping? ●くろまる Derive parametric AO from the heightmap?  Tessellated Approach ●くろまる When artists were working on content creation, displacement wasn't taken into account (pre-pass actors, open edges being visible, etc...) ●くろまる Some meshes couldn't use tessellation as there were parts of geometry right under the snow, not supposed to be visible

Future Endeavours...  Save the heightmaps and reload them?  Use this technique for other cases, such as sand, mud, etc...

Summary  A fast and low-memory footprint technique to render deformable snow surfaces ●くろまる Adds a really nice level of interaction between players and the world ●くろまる Depics iconic & organic visuals of deformable snow  A good tessellation case for your DX11 game using minimal editing and art tweaks

Thank You! Érick Bilodeau David Massicotte Sébastien Turcotte Jimmy Béliveau Olivier Pomarez Philippe Bernard Ryan Lewis Marc Bouchard Jean-Noé Morissette Pierric Gimmig Patrick Dubuc Reid Schneider Maggy Larouche Miguel Sainz Evgeny Makarov Jon Jansen Christina Coffin Jon Greenberg NVIDIA Questions? colin.barrebrisebois@wbgames.com / @ZigguratVertigo

http://www.wbgamesmontreal.com

References [Barré-Brisebois & Hill 2012] Barré-Brisebois, Colin and Hill, Stephen. "Blending in Detail - Reoriented Normal Mapping", 2012. http://bit.ly/Mf2UH0 [Edwards 2013] Edwards, John. "Sand Rendering in Journey", Advances in Real-Time Rendering, SIGGRAPH, 2012. http://advances.realtimerendering.com/s2012/index.html [Policarpo & Oliveira 2006] Policarpo, Fabio and Oliveira, Manuel M. Rendering Surface Details in Games with Relief Mapping Using a Minimally Invasive Approach. In:Wolfgang Engel (ed.). SHADER X4: Lighting & Rendering. Charles River Media, Inc., Hingham, Massachusetts, 2006 (ISBN 1-58450-425-0), pp. 109-119. [St-Amour 2013] St-Amour, Jean-François. "Rendering Assassin's Creed", Game Developers Conference, 2013.

GDC 2014 - Deformable Snow Rendering in Batman: Arkham Origins