Learn more about Russian war crimes in Ukraine.

Step-away background removal

In this previous post I showed how to remove a background from a webcam feed using a “green screen” algorithm implemented in WebGL. In this post, I show how to remove a background from a webcam feed given advance knowledge of what the background looks like. Below is a demo — click “Start webcam”, then hit “Background snapshot”. Three seconds later (enough time for you to get out of the shot!), it takes a snapshot, and starts applying the background removal algorithm.


Depending on your background, you’ll probably get very different results. With a smooth, colorful background (approximating a green screen!), the algorithm works great. With a busy background, it’s much more hit-and-miss. Here’s an example of this algorithm used against a fairly simple background. The snapshotted background is the first image; the output is the second:

I found that the standard chroma key algorithm does not work well with real backgrounds. The standard chroma key algorithm works by looking at difference in chrominance (color), and throws away luminance information. This works well when using a green screen. But real backgrounds have areas that are very light or very dark, where chrominance is irrelevant. In the extreme, the standard chroma key algorithm believes that black is equivalent to white. For example, it removes my dark hair when set against a nearly white background!

Clearly, a better algorithm should compare luminance, too. The algorithm used above does so. It mostly measures distance in chrominance, but where the background is very light or dark, it transitions to measuring distance in luminance.

The output image shows several artifacts. The most annoying artifacts are false transparency in reflections and highlights on my skin, which are nearly identical in color to the cream wall paint behind me.

Another form of artifact is long lines of non-transparency, caused by slight shifting in the background. These artifacts could be reduced by something like a median filter, or some more sophisticated background tracking.

What can computers do? What are the limits of mathematics? And just how busy can a busy beaver be? This year, I’m writing Busy Beavers, a unique interactive book on computability theory. You and I will take a practical and modern approach to answering these questions — or at least learning why some questions are unanswerable!

It’s only $19, and you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

After months of secret toil, I and Andrew Carr released Everyday Data Science, a unique interactive online course! You’ll make the perfect glass of lemonade using Thompson sampling. You’ll lose weight with differential equations. And you might just qualify for the Olympics with a bit of statistics!

It’s $29, but you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

More by Jim

Tagged #programming, #web, #webgl. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.