Learn more about Russian war crimes in Ukraine.

How to make a floating head

In a previous post. I showed how to run BodyPix on a video stream and access the segmentation from your shader. In this post, I demo the segmentPersonParts method, using it to make a floating head. You can run it on your own webcam;

A call to net.segmentPerson returns a Uint8Array where each value is either 1 (part of a person) or 0 (not part of a person). But the API also provides net.segmentPersonParts, which returns an Int32Array, but where the values are between -1 and 23, with -1 being not part of a person, and the other numbers being various parts. Here I’m just interested in values 0 and 1, which represent parts “left side of face” and “right side of face”.

To access this data in a WebGL shader, we need to get it into a texture using gl.texImage2D. When you pass an array to gl.texImage2D, you tell it which format to interpret it as. We’ll use gl.ALPHA, which has one unsigned byte per pixel. To convert to this format, we can use new Uint8Array(segmentation.data). The -1 values in signed ints get converted to 255 in unsigned ints. This byte interpreted as the alpha channel when the texture is accessed by a shader. Then in the pixel shader, these values between 0 and 255 are squeezed into the float range from 0 to 1. So we can check for the original values 1 and 2 with alpha <= 2./255..

BodyPix seems to include “neck” as part of the face. This is annoying for my purposes; I wanted to crop the face at the chin. I don’t see an easy way to hack around this.

Finally, here’s what I get when I run the demo against my own webcam feed:

What can computers do? What are the limits of mathematics? And just how busy can a busy beaver be? This year, I’m writing Busy Beavers, a unique interactive book on computability theory. You and I will take a practical and modern approach to answering these questions — or at least learning why some questions are unanswerable!

It’s only $19, and you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

After months of secret toil, I and Andrew Carr released Everyday Data Science, a unique interactive online course! You’ll make the perfect glass of lemonade using Thompson sampling. You’ll lose weight with differential equations. And you might just qualify for the Olympics with a bit of statistics!

It’s $29, but you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

More by Jim

Tagged #programming, #web, #webgl, #ml. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.