So, this is not directly to do with graphics, or shading. But I’m just finishing up my postdoc and the last year or so I’ve been working on this real-time depth-camera based tracking system. For want of another place to publish this, I decided to make a short blog post.
The system came to be as part of a project to create a robotic system to interact with rats, the Pirat. The goal is to be able to study social interaction, in a rat/robot perspective. To support this the tracking system ideally must be able to track the position and pose of the rat, such that the robot can respond autonomously to such stimuli.
Video showing the Pirat system accompanying the IROS 2018 paper. This paper describes the whole system which used an early version of tracker as one part. The published code is just the tracker, but much more developed.
Here’s another video that showcases what the system is capable of and how it works. It’s an extended version of a video that was originally submitted with a paper to ICRA 2019 (but which was rejected). I might put up a pre-print at some point.
I’ve now published it as open source on gitlab. The system uses CUDA to be able to track the pose of rats in real-time, but can also be run using only the CPU. The basic tracking algorithm is based on ICP – finding correspondences in the point cloud to some template and then solving for translation and rotation and repeating – with a bunch of modifications. The system is optimized for latency, to enable closed-loop control of a robot (which is what the system was designed for). There is a lot more documentation about the system in the readme.
This year, first Oculus, then ID Software, have detailed successful applications of Clustered (Forward) Shading, adding to the existing implementations in Just Cause 3 and Forza Horizon 2. There are doubtless other implementations that have not become public yet, but this already builds a strong case for clustered shading in production. The genres are diverse with open-world sandbox, FPS, racing and Virtual Reality, providing evidence that this technique really is robust for many use-cases.
Three out of four use Clustered Forward Shading, while Avalanche (Just Cause), use a mix, where the main opaque geometry is rendered using Clustered Deferred and transparent geometry uses a Clustered Forward path.
The use in Doom is by far the most advanced yet, storing not only lights in the cluster grid, but also using is to store reflection probes and decals. These are then applied in a main forward shading pass.
These additions demonstrate the power of the technique, highlighting a key property of Clustered Shading: making it possible to turn ‘splatting’ techniques into gathering. This, in turn, allows the freedom to choose forward or deferred shading as desired. The flexibility afforded by this means that game engines can finally support transparent geometry with a consistent look again.
In principle, anything that can be splatted could be gathered efficiently using Clustered Shading. As developers gain experience using the technique I’m sure we will see many new exciting takes on this approach.
- Adrian Courrèges’ Blog provides details on the DOOM rendering pipeline, but sources most of it’s information about clustered shading from:
- Tiago Sousa’s presentation from Advances in Real-time Rendering at SIGGRAPH 2016 titled ‘The devil is in the details: idTech666’, which also gives details on their GCN specific optimizations.
- Some more cover (but no added information) in EuroGamer.
- EuroGamer also provides the article mentioning the use of Clustered Forward Shading in Forza Horizon 2. (Though they mix in the ‘Forward+’ terminology to confuse matters, thanks again AMD).
- Oculus developer Pete Demoreuille, describes their implementation of Clustered Forward Shading in the Unreal Engine. Full source code is available on github to registered Unreal Developers (it’s free!). Their focus is on achieving 90fps on VR devices.
- The SIGGRAPH 2015 course organized by myself together with Emil Persson and Markus Billeter provides overview material and advanced topics, also covering shadow management and mobile hardware.
- Intel demo implementation ‘Forward Clustered Shading‘, uses ISPC to vectorize the grid construction on Intel CPUs. Intel also hosts a sample implementation for android.
- Gareth Thomas from AMD gave a presentation titled ‘Advancements in Tile-Based Compute Rendering‘ at GDC 2015. He provides some details and comparison to modifications of tiled shading (e.g., Half-Z and 2.5D culling). Though he does incorrectly state that our original paper does all culling on the CPU, whereas we do everything on the GPU.
- Matt Pettineo has a blog post and a demo where he implements clustered shading including (clustered) decals. He compares deferred texturing and clustered forward shading.
- My original paper at HPG which got the ball rolling (non-pay walled author copy, video, and demo are available here).
If you are interested in evaluating Clustered Shading in your engine, our consultant services may be of help. We have in-house robust and high-performance implementations that we can integrate into your engine and license. The current implementation has been tested in the Unreal Engine with full compute shader implementation.
I recently did a talk at the Game Technology Brisbane Meetup. The presentation roughly covered the same ground as my two parts of the SIGGRAPH 2015 course, with some fairly minor modifications (the actual talk is a little more different than the notes and slides actually).
The presentation is available on youtube, making it more accessible than most of my other talks that require ACM digital library access. I have done a few presentations before at Game Technology Brisbane, and it is great to present to an interested and engaged audience like this, with interesting discussion and good questions!
introduction (pdf | pptx)
shadows (pdf | pptx)