Learn more about Russian war crimes in Ukraine.

How to implement green screen in the browser

You’re making a web app that captures a user’s webcam, your user has a green screen behind them, and you want to “remove the background” from the webcam video in realtime. This post shows one way to do so. First, here’s the live demo, which replaces green pixels with magenta:

The “pipeline” for this demo is:

There are two big deficiencies in this demo, as a result of the naivety of the approach. First, it’s pretty inefficient. For efficiency, everything should happen on the GPU, but this demo does most processing on the CPU. It uses getImageData and putImageData to process frames in JavaScript as ImageData objects. In the next post, I show how to avoid this inefficiency by using a WebGL shader..

The second deficiency here is the naivety of the green screen algorithm. The demo sets a pixel transparent if g > 100 && r < 100. There exist more sophisticated methods to decide whether a pixel should be transparent, or how transparent it should be. There are also algorithms for “color spill reduction”, removing green light reflected from the subject. I’ll also show these in a future post.

Finally, here’s the complete HTML for this example.

<!DOCTYPE html>
<html>
  <body>
    <video id="webcamVideo" style="display: none;"></video>
    <canvas id="displayCanvas" style="background-color: magenta;"></canvas>
    <button onclick="startWebcam(); this.parentElement.removeChild(this)">Start webcam</button>
    <script type="text/javascript">
      function startWebcam() {
        const webcamVideoEl = document.getElementById("webcamVideo");
        const blitCanvas = new OffscreenCanvas(0, 0);  // size dynamically assigned per frame
        const blitCtx = blitCanvas.getContext("2d");
        const displayCanvasEl = document.getElementById("displayCanvas");
        const displayCtx = displayCanvasEl.getContext("2d");
        navigator.mediaDevices.getUserMedia({ video: { facingMode: "user" } }).then(stream => {
            webcamVideoEl.srcObject = stream;
            webcamVideoEl.play();
            function processFrame(now, metadata) {
              // downsample to this width (more sophisticated could dynamically choose size)
              const canvasWidth = 320;

              // use aspect ratio of latest frame
              const height = canvasWidth * metadata.height/metadata.width;
  
              // note this clears the canvases (at least in Chrome)
              blitCanvas.width = canvasWidth;
              blitCanvas.height = height;
              displayCanvasEl.width = canvasWidth;
              displayCanvasEl.height = height;
  
              // Downsamples video to canvas size
              blitCtx.drawImage(webcamVideoEl, 0, 0, canvasWidth, height);
              const imageData = blitCtx.getImageData(0, 0, canvasWidth, height);
              
              const numPixels = imageData.data.length / 4;
              for (let i = 0; i < numPixels; i++) {
                const r = imageData.data[i * 4 + 0];
                const g = imageData.data[i * 4 + 1];
                const b = imageData.data[i * 4 + 2];
                if (g > 100 && r < 100) imageData.data[i * 4 + 3] = 0;  // crude green screen
              }
              displayCtx.putImageData(imageData, 0, 0);
              webcamVideoEl.requestVideoFrameCallback(processFrame);
            }
            webcamVideoEl.requestVideoFrameCallback(processFrame);
        }).catch(error => {
          console.error(error);
        });
      }
    </script>
  </body>
</html>

What can computers do? What are the limits of mathematics? And just how busy can a busy beaver be? This year, I’m writing Busy Beavers, a unique interactive book on computability theory. You and I will take a practical and modern approach to answering these questions — or at least learning why some questions are unanswerable!

It’s only $19, and you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

After months of secret toil, I and Andrew Carr released Everyday Data Science, a unique interactive online course! You’ll make the perfect glass of lemonade using Thompson sampling. You’ll lose weight with differential equations. And you might just qualify for the Olympics with a bit of statistics!

It’s $29, but you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

More by Jim

Tagged #programming, #web. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.