How to implement green screen in the browser

You’re making a web app that captures a user’s webcam, your user has a green screen behind them, and you want to “remove the background” from the webcam video in realtime. This post shows one way to do so. First, here’s the live demo, which replaces green pixels with magenta:

The “pipeline” for this demo is:

There are two big deficiencies in this demo, as a result of the naivety of the approach. First, it’s pretty inefficient. For efficiency, everything should happen on the GPU, but this demo does most processing on the CPU. It uses getImageData and putImageData to process frames in JavaScript as ImageData objects. In the next post, I show how to avoid this inefficiency by using a WebGL shader..

The second deficiency here is the naivety of the green screen algorithm. The demo sets a pixel transparent if g > 100 && r < 100. There exist more sophisticated methods to decide whether a pixel should be transparent, or how transparent it should be. There are also algorithms for “color spill reduction”, removing green light reflected from the subject. I’ll also show these in a future post.

Finally, here’s the complete HTML for this example.

<!DOCTYPE html>
<html>
  <body>
    <video id="webcamVideo" style="display: none;"></video>
    <canvas id="displayCanvas" style="background-color: magenta;"></canvas>
    <button onclick="startWebcam(); this.parentElement.removeChild(this)">Start webcam</button>
    <script type="text/javascript">
      function startWebcam() {
        const webcamVideoEl = document.getElementById("webcamVideo");
        const blitCanvas = new OffscreenCanvas(0, 0);  // size dynamically assigned per frame
        const blitCtx = blitCanvas.getContext("2d");
        const displayCanvasEl = document.getElementById("displayCanvas");
        const displayCtx = displayCanvasEl.getContext("2d");
        navigator.mediaDevices.getUserMedia({ video: { facingMode: "user" } }).then(stream => {
            webcamVideoEl.srcObject = stream;
            webcamVideoEl.play();
            function processFrame(now, metadata) {
              // downsample to this width (more sophisticated could dynamically choose size)
              const canvasWidth = 320;

              // use aspect ratio of latest frame
              const height = canvasWidth * metadata.height/metadata.width;
  
              // note this clears the canvases (at least in Chrome)
              blitCanvas.width = canvasWidth;
              blitCanvas.height = height;
              displayCanvasEl.width = canvasWidth;
              displayCanvasEl.height = height;
  
              // Downsamples video to canvas size
              blitCtx.drawImage(webcamVideoEl, 0, 0, canvasWidth, height);
              const imageData = blitCtx.getImageData(0, 0, canvasWidth, height);
              
              const numPixels = imageData.data.length / 4;
              for (let i = 0; i < numPixels; i++) {
                const r = imageData.data[i * 4 + 0];
                const g = imageData.data[i * 4 + 1];
                const b = imageData.data[i * 4 + 2];
                if (g > 100 && r < 100) imageData.data[i * 4 + 3] = 0;  // crude green screen
              }
              displayCtx.putImageData(imageData, 0, 0);
              webcamVideoEl.requestVideoFrameCallback(processFrame);
            }
            webcamVideoEl.requestVideoFrameCallback(processFrame);
        }).catch(error => {
          console.error(error);
        });
      }
    </script>
  </body>
</html>
I just released Vidrio, a free app for macOS and Windows to make your screen-sharing awesomely holographic. Vidrio shows your webcam video on your screen, just like a mirror. Then you just share or record your screen with Zoom, QuickTime, or any other app. Vidrio makes your presentations effortlessly engaging, showing your gestures, gazes, and expressions. #1 on Product Hunt. Available for macOS and Windows.

With Vidrio

With generic competitor

More by Jim

Tagged #programming, #web. All content copyright James Fisher 2020. This post is not associated with my employer. Found an error? Edit this page.