Step-away background removal

In this previous post I showed how to remove a background from a webcam feed using a “green screen” algorithm implemented in WebGL. In this post, I show how to remove a background from a webcam feed given advance knowledge of what the background looks like. Below is a demo — click “Start webcam”, then hit “Background snapshot”. Three seconds later (enough time for you to get out of the shot!), it takes a snapshot, and starts applying the background removal algorithm.



Similarity
Smoothness
Spill

Depending on your background, you’ll probably get very different results. With a smooth, colorful background (approximating a green screen!), the algorithm works great. With a busy background, it’s much more hit-and-miss. Here’s an example of this algorithm used against a fairly simple background. The snapshotted background is the first image; the output is the second:

I found that the standard chroma key algorithm does not work well with real backgrounds. The standard chroma key algorithm works by looking at difference in chrominance (color), and throws away luminance information. This works well when using a green screen. But real backgrounds have areas that are very light or very dark, where chrominance is irrelevant. In the extreme, the standard chroma key algorithm believes that black is equivalent to white. For example, it removes my dark hair when set against a nearly white background!

Clearly, a better algorithm should compare luminance, too. The algorithm used above does so. It mostly measures distance in chrominance, but where the background is very light or dark, it transitions to measuring distance in luminance.

The output image shows several artifacts. The most annoying artifacts are false transparency in reflections and highlights on my skin, which are nearly identical in color to the cream wall paint behind me.

Another form of artifact is long lines of non-transparency, caused by slight shifting in the background. These artifacts could be reduced by something like a median filter, or some more sophisticated background tracking.

Tagged #programming, #web, #webgl.

👋 I'm Jim, a full-stack product engineer. Want to build an amazing product and a profitable business? Read more about me or Get in touch!

Similar posts

Edge detection with Sobel filters

A WebGL fragment shader that implements the Sobel filter, a method for detecting edges in an image. A demo processes your webcam video stream. 2020-08-31

Why does my WebGL alpha-transparency look wrong?

WebGL alpha-transparency may appear wrong because of premultiplied alpha. Fix it by setting premultipliedAlpha: false when creating the WebGL context. 2020-08-12

Production-ready green screen in the browser

A green screen implementation in the browser using WebGL and chroma key. Includes a live demo. 2020-08-11

How to implement green screen in WebGL

An green screen implementation in the browser using WebGL and a chroma key shader. Includes a demo that replaces sufficiently green pixels with magenta. 2020-08-10

How to make a floating head

Creating a “floating head” effect using WebGL and the BodyPix machine learning model to segment the face from a webcam feed. 2020-10-06

Using BodyPix segmentation in a WebGL shader

A WebGL shader that uses BodyPix segmentation to set the alpha channel. 2020-09-24

Similar posts

Edge detection with Sobel filters

A WebGL fragment shader that implements the Sobel filter, a method for detecting edges in an image. A demo processes your webcam video stream. 2020-08-31

Why does my WebGL alpha-transparency look wrong?

WebGL alpha-transparency may appear wrong because of premultiplied alpha. Fix it by setting premultipliedAlpha: false when creating the WebGL context. 2020-08-12

Production-ready green screen in the browser

A green screen implementation in the browser using WebGL and chroma key. Includes a live demo. 2020-08-11

How to implement green screen in WebGL

An green screen implementation in the browser using WebGL and a chroma key shader. Includes a demo that replaces sufficiently green pixels with magenta. 2020-08-10

How to make a floating head

Creating a “floating head” effect using WebGL and the BodyPix machine learning model to segment the face from a webcam feed. 2020-10-06

Using BodyPix segmentation in a WebGL shader

A WebGL shader that uses BodyPix segmentation to set the alpha channel. 2020-09-24

More by Jim

What does the dot do in JavaScript?

foo.bar, foo.bar(), or foo.bar = baz - what do they mean? A deep dive into prototypical inheritance and getters/setters. 2020-11-01

Smear phishing: a new Android vulnerability

Trick Android to display an SMS as coming from any contact. Convincing phishing vuln, but still unpatched. 2020-08-06

A probabilistic pub quiz for nerds

A “true or false” quiz where you respond with your confidence level, and the optimal strategy is to report your true belief. 2020-04-26

Time is running out to catch COVID-19

Simulation shows it’s rational to deliberately infect yourself with COVID-19 early on to get treatment, but after healthcare capacity is exceeded, it’s better to avoid infection. Includes interactive parameters and visualizations. 2020-03-14

The inception bar: a new phishing method

A new phishing technique that displays a fake URL bar in Chrome for mobile. A key innovation is the “scroll jail” that traps the user in a fake browser. 2019-04-27

The hacker hype cycle

I got started with simple web development, but because enamored with increasingly esoteric programming concepts, leading to a “trough of hipster technologies” before returning to more productive work. 2019-03-23

Project C-43: the lost origins of asymmetric crypto

Bob invents asymmetric cryptography by playing loud white noise to obscure Alice’s message, which he can cancel out but an eavesdropper cannot. This idea, published in 1944 by Walter Koenig Jr., is the forgotten origin of asymmetric crypto. 2019-02-16

How Hacker News stays interesting

Hacker News buried my post on conspiracy theories in my family due to overheated discussion, not censorship. Moderation keeps the site focused on interesting technical content. 2019-01-26

My parents are Flat-Earthers

For decades, my parents have been working up to Flat-Earther beliefs. From Egyptology to Jehovah’s Witnesses to theories that human built the Moon billions of years in the future. Surprisingly, it doesn’t affect their successful lives very much. For me, it’s a fun family pastime. 2019-01-20

The dots do matter: how to scam a Gmail user

Gmail’s “dots don’t matter” feature lets scammers create an account on, say, Netflix, with your email address but different dots. Results in convincing phishing emails. 2018-04-07

The sorry state of OpenSSL usability

OpenSSL’s inadequate documentation, confusing key formats, and deprecated interfaces make it difficult to use, despite its importance. 2017-12-02

I hate telephones

I hate telephones. Some rational reasons: lack of authentication, no spam filtering, forced synchronous communication. But also just a visceral fear. 2017-11-08

The Three Ts of Time, Thought and Typing: measuring cost on the web

Businesses often tout “free” services, but the real costs come in terms of time, thought, and typing required from users. Reducing these “Three Ts” is key to improving sign-up flows and increasing conversions. 2017-10-26

Granddad died today

Granddad died. The unspoken practice of death-by-dehydration in the NHS. The Liverpool Care Pathway. Assisted dying in the UK. The importance of planning in end-of-life care. 2017-05-19

How do I call a program in C, setting up standard pipes?

A C function to create a new process, set up its standard input/output/error pipes, and return a struct containing the process ID and pipe file descriptors. 2017-02-17

Your syntax highlighter is wrong

Syntax highlighters make value judgments about code. Most highlighters judge that comments are cruft, and try to hide them. Most diff viewers judge that code deletions are bad. 2014-05-11

This page copyright James Fisher 2020. Content is not associated with my employer. Found an error? Edit this page.