Step-away background removal
In this previous post I showed how to remove a background from a webcam feed using a “green screen” algorithm implemented in WebGL. In this post, I show how to remove a background from a webcam feed given advance knowledge of what the background looks like. Below is a demo — click “Start webcam”, then hit “Background snapshot”. Three seconds later (enough time for you to get out of the shot!), it takes a snapshot, and starts applying the background removal algorithm.
| Similarity | |
|---|---|
| Smoothness | |
| Spill |
Depending on your background, you’ll probably get very different results. With a smooth, colorful background (approximating a green screen!), the algorithm works great. With a busy background, it’s much more hit-and-miss. Here’s an example of this algorithm used against a fairly simple background. The snapshotted background is the first image; the output is the second:
I found that the standard chroma key algorithm does not work well with real backgrounds. The standard chroma key algorithm works by looking at difference in chrominance (color), and throws away luminance information. This works well when using a green screen. But real backgrounds have areas that are very light or very dark, where chrominance is irrelevant. In the extreme, the standard chroma key algorithm believes that black is equivalent to white. For example, it removes my dark hair when set against a nearly white background!
Clearly, a better algorithm should compare luminance, too. The algorithm used above does so. It mostly measures distance in chrominance, but where the background is very light or dark, it transitions to measuring distance in luminance.
The output image shows several artifacts. The most annoying artifacts are false transparency in reflections and highlights on my skin, which are nearly identical in color to the cream wall paint behind me.
Another form of artifact is long lines of non-transparency, caused by slight shifting in the background. These artifacts could be reduced by something like a median filter, or some more sophisticated background tracking.
With Vidrio
With generic competitor
More by Jim
- Your syntax highlighter is wrong
- Granddad died today
- The Three Ts of Time, Thought and Typing: measuring cost on the web
- I hate telephones
- The sorry state of OpenSSL usability
- The dots do matter: how to scam a Gmail user
- My parents are Flat-Earthers
- How Hacker News stays interesting
- Project C-43: the lost origins of asymmetric crypto
- The hacker hype cycle
- The inception bar: a new phishing method
- Time is running out to catch COVID-19
- A probabilistic pub quiz for nerds
- Smear phishing: a new Android vulnerability
Tagged #programming, #web, #webgl. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.