Step-away background removal

In this previous post I showed how to remove a background from a webcam feed using a “green screen” algorithm implemented in WebGL. In this post, I show how to remove a background from a webcam feed given advance knowledge of what the background looks like. Below is a demo — click “Start webcam”, then hit “Background snapshot”. Three seconds later (enough time for you to get out of the shot!), it takes a snapshot, and starts applying the background removal algorithm.

	Start webcam
	Background snapshot
Similarity
Smoothness
Spill

Depending on your background, you’ll probably get very different results. With a smooth, colorful background (approximating a green screen!), the algorithm works great. With a busy background, it’s much more hit-and-miss. Here’s an example of this algorithm used against a fairly simple background. The snapshotted background is the first image; the output is the second:

I found that the standard chroma key algorithm does not work well with real backgrounds. The standard chroma key algorithm works by looking at difference in chrominance (color), and throws away luminance information. This works well when using a green screen. But real backgrounds have areas that are very light or very dark, where chrominance is irrelevant. In the extreme, the standard chroma key algorithm believes that black is equivalent to white. For example, it removes my dark hair when set against a nearly white background!

Clearly, a better algorithm should compare luminance, too. The algorithm used above does so. It mostly measures distance in chrominance, but where the background is very light or dark, it transitions to measuring distance in luminance.

The output image shows several artifacts. The most annoying artifacts are false transparency in reflections and highlights on my skin, which are nearly identical in color to the cream wall paint behind me.

Another form of artifact is long lines of non-transparency, caused by slight shifting in the background. These artifacts could be reduced by something like a median filter, or some more sophisticated background tracking.

Tagged #programming, #web, #webgl. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.