How to make a floating head

In a previous post. I showed how to run BodyPix on a video stream and access the segmentation from your shader. In this post, I demo the segmentPersonParts method, using it to make a floating head. You can run it on your own webcam;

A call to net.segmentPerson returns a Uint8Array where each value is either 1 (part of a person) or 0 (not part of a person). But the API also provides net.segmentPersonParts, which returns an Int32Array, but where the values are between -1 and 23, with -1 being not part of a person, and the other numbers being various parts. Here I’m just interested in values 0 and 1, which represent parts “left side of face” and “right side of face”.

To access this data in a WebGL shader, we need to get it into a texture using gl.texImage2D. When you pass an array to gl.texImage2D, you tell it which format to interpret it as. We’ll use gl.ALPHA, which has one unsigned byte per pixel. To convert to this format, we can use new Uint8Array(segmentation.data). The -1 values in signed ints get converted to 255 in unsigned ints. This byte interpreted as the alpha channel when the texture is accessed by a shader. Then in the pixel shader, these values between 0 and 255 are squeezed into the float range from 0 to 1. So we can check for the original values 1 and 2 with alpha <= 2./255..

BodyPix seems to include “neck” as part of the face. This is annoying for my purposes; I wanted to crop the face at the chin. I don’t see an easy way to hack around this.

Finally, here’s what I get when I run the demo against my own webcam feed:

Tagged #programming, #web, #webgl, #machine-learning.

👋 I'm Jim, a full-stack product engineer. Want to build an amazing product and a profitable business? Read more about me or Get in touch!

Similar posts

Using BodyPix segmentation in a WebGL shader

A WebGL shader that uses BodyPix segmentation to set the alpha channel. 2020-09-24

What is a Single-Shot Multibox Detector?

A neural network architecture that can output bounding boxes. It uses a fixed set of “anchor points” and predicts offsets and sizes for each anchor. 2021-02-16

How to run a pre-trained model in TensorFlow.js

Load the model, convert the input to a tensor, preprocess the tensor to match the model’s expected format, run inference with .predict(), and decode the prediction tensor. 2021-02-15

A head in a box with BlazeFace

A method for calculating a bounding circle around a head, using facial landmarks from BlazeFace. Plus a live demo that you can run on your own face. 2020-10-12

Head tracking with BlazeFace

A method for calculating a bounding circle around a head, using facial landmarks from BlazeFace. Plus a live demo that you can run on your own face. 2020-10-11

How to self-host a TensorFlow.js model

Involves downloading the model manifest file and associated binary weight files, to avoid downloading large models every application launch. 2020-10-07

Similar posts

Using BodyPix segmentation in a WebGL shader

A WebGL shader that uses BodyPix segmentation to set the alpha channel. 2020-09-24

What is a Single-Shot Multibox Detector?

A neural network architecture that can output bounding boxes. It uses a fixed set of “anchor points” and predicts offsets and sizes for each anchor. 2021-02-16

How to run a pre-trained model in TensorFlow.js

Load the model, convert the input to a tensor, preprocess the tensor to match the model’s expected format, run inference with .predict(), and decode the prediction tensor. 2021-02-15

A head in a box with BlazeFace

A method for calculating a bounding circle around a head, using facial landmarks from BlazeFace. Plus a live demo that you can run on your own face. 2020-10-12

Head tracking with BlazeFace

A method for calculating a bounding circle around a head, using facial landmarks from BlazeFace. Plus a live demo that you can run on your own face. 2020-10-11

How to self-host a TensorFlow.js model

Involves downloading the model manifest file and associated binary weight files, to avoid downloading large models every application launch. 2020-10-07

More by Jim

What does the dot do in JavaScript?

foo.bar, foo.bar(), or foo.bar = baz - what do they mean? A deep dive into prototypical inheritance and getters/setters. 2020-11-01

Smear phishing: a new Android vulnerability

Trick Android to display an SMS as coming from any contact. Convincing phishing vuln, but still unpatched. 2020-08-06

A probabilistic pub quiz for nerds

A “true or false” quiz where you respond with your confidence level, and the optimal strategy is to report your true belief. 2020-04-26

Time is running out to catch COVID-19

Simulation shows it’s rational to deliberately infect yourself with COVID-19 early on to get treatment, but after healthcare capacity is exceeded, it’s better to avoid infection. Includes interactive parameters and visualizations. 2020-03-14

The inception bar: a new phishing method

A new phishing technique that displays a fake URL bar in Chrome for mobile. A key innovation is the “scroll jail” that traps the user in a fake browser. 2019-04-27

The hacker hype cycle

I got started with simple web development, but because enamored with increasingly esoteric programming concepts, leading to a “trough of hipster technologies” before returning to more productive work. 2019-03-23

Project C-43: the lost origins of asymmetric crypto

Bob invents asymmetric cryptography by playing loud white noise to obscure Alice’s message, which he can cancel out but an eavesdropper cannot. This idea, published in 1944 by Walter Koenig Jr., is the forgotten origin of asymmetric crypto. 2019-02-16

How Hacker News stays interesting

Hacker News buried my post on conspiracy theories in my family due to overheated discussion, not censorship. Moderation keeps the site focused on interesting technical content. 2019-01-26

My parents are Flat-Earthers

For decades, my parents have been working up to Flat-Earther beliefs. From Egyptology to Jehovah’s Witnesses to theories that human built the Moon billions of years in the future. Surprisingly, it doesn’t affect their successful lives very much. For me, it’s a fun family pastime. 2019-01-20

The dots do matter: how to scam a Gmail user

Gmail’s “dots don’t matter” feature lets scammers create an account on, say, Netflix, with your email address but different dots. Results in convincing phishing emails. 2018-04-07

The sorry state of OpenSSL usability

OpenSSL’s inadequate documentation, confusing key formats, and deprecated interfaces make it difficult to use, despite its importance. 2017-12-02

I hate telephones

I hate telephones. Some rational reasons: lack of authentication, no spam filtering, forced synchronous communication. But also just a visceral fear. 2017-11-08

The Three Ts of Time, Thought and Typing: measuring cost on the web

Businesses often tout “free” services, but the real costs come in terms of time, thought, and typing required from users. Reducing these “Three Ts” is key to improving sign-up flows and increasing conversions. 2017-10-26

Granddad died today

Granddad died. The unspoken practice of death-by-dehydration in the NHS. The Liverpool Care Pathway. Assisted dying in the UK. The importance of planning in end-of-life care. 2017-05-19

How do I call a program in C, setting up standard pipes?

A C function to create a new process, set up its standard input/output/error pipes, and return a struct containing the process ID and pipe file descriptors. 2017-02-17

Your syntax highlighter is wrong

Syntax highlighters make value judgments about code. Most highlighters judge that comments are cruft, and try to hide them. Most diff viewers judge that code deletions are bad. 2014-05-11

This page copyright James Fisher 2020. Content is not associated with my employer. Found an error? Edit this page.