So, this is not directly to do with graphics, or shading. But I’m just finishing up my postdoc and the last year or so I’ve been working on this real-time depth-camera based tracking system. For want of another place to publish this, I decided to make a short blog post.
The system came to be as part of a project to create a robotic system to interact with rats, the Pirat. The goal is to be able to study social interaction, in a rat/robot perspective. To support this the tracking system ideally must be able to track the position and pose of the rat, such that the robot can respond autonomously to such stimuli.
Video showing the Pirat system accompanying the IROS 2018 paper. This paper describes the whole system which used an early version of tracker as one part. The published code is just the tracker, but much more developed.
Here’s another video that showcases what the system is capable of and how it works. It’s an extended version of a video that was originally submitted with a paper to ICRA 2019 (but which was rejected). I might put up a pre-print at some point.
I’ve now published it as open source on gitlab. The system uses CUDA to be able to track the pose of rats in real-time, but can also be run using only the CPU. The basic tracking algorithm is based on ICP – finding correspondences in the point cloud to some template and then solving for translation and rotation and repeating – with a bunch of modifications. The system is optimized for latency, to enable closed-loop control of a robot (which is what the system was designed for). There is a lot more documentation about the system in the readme.