Step-away background removal
In this previous post I showed how to remove a background from a webcam feed using a “green screen” algorithm implemented in WebGL. In this post, I show how to remove a background from a webcam feed given advance knowledge of what the background looks like. Below is a demo — click “Start webcam”, then hit “Background snapshot”. Three seconds later (enough time for you to get out of the shot!), it takes a snapshot, and starts applying the background removal algorithm.
| Similarity | |
|---|---|
| Smoothness | |
| Spill |
Depending on your background, you’ll probably get very different results. With a smooth, colorful background (approximating a green screen!), the algorithm works great. With a busy background, it’s much more hit-and-miss. Here’s an example of this algorithm used against a fairly simple background. The snapshotted background is the first image; the output is the second:
I found that the standard chroma key algorithm does not work well with real backgrounds. The standard chroma key algorithm works by looking at difference in chrominance (color), and throws away luminance information. This works well when using a green screen. But real backgrounds have areas that are very light or very dark, where chrominance is irrelevant. In the extreme, the standard chroma key algorithm believes that black is equivalent to white. For example, it removes my dark hair when set against a nearly white background!
Clearly, a better algorithm should compare luminance, too. The algorithm used above does so. It mostly measures distance in chrominance, but where the background is very light or dark, it transitions to measuring distance in luminance.
The output image shows several artifacts. The most annoying artifacts are false transparency in reflections and highlights on my skin, which are nearly identical in color to the cream wall paint behind me.
Another form of artifact is long lines of non-transparency, caused by slight shifting in the background. These artifacts could be reduced by something like a median filter, or some more sophisticated background tracking.
With Vidrio
With generic competitor
More by Jim
- Smear phishing: a new Android vulnerability
- A probabilistic pub quiz for nerds
- Time is running out to catch COVID-19
- The inception bar: a new phishing method
- The hacker hype cycle
- Project C-43: the lost origins of asymmetric crypto
- How Hacker News stays interesting
- My parents are Flat-Earthers
- The dots do matter: how to scam a Gmail user
- The sorry state of OpenSSL usability
- I hate telephones
- The Three Ts of Time, Thought and Typing: measuring cost on the web
- Granddad died today
- Your syntax highlighter is wrong
Tagged #programming, #web, #webgl. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.