How to implement green screen in the browser

You’re making a web app that captures a user’s webcam, your user has a green screen behind them, and you want to “remove the background” from the webcam video in realtime. This post shows one way to do so. First, here’s the live demo, which replaces green pixels with magenta:

The “pipeline” for this demo is:

There are two big deficiencies in this demo, as a result of the naivety of the approach. First, it’s pretty inefficient. For efficiency, everything should happen on the GPU, but this demo does most processing on the CPU. It uses getImageData and putImageData to process frames in JavaScript as ImageData objects. In the next post, I show how to avoid this inefficiency by using a WebGL shader..

The second deficiency here is the naivety of the green screen algorithm. The demo sets a pixel transparent if g > 100 && r < 100. There exist more sophisticated methods to decide whether a pixel should be transparent, or how transparent it should be. There are also algorithms for “color spill reduction”, removing green light reflected from the subject. I’ll also show these in a future post.

Finally, here’s the complete HTML for this example.

<!DOCTYPE html>
<html>
  <body>
    <video id="webcamVideo" style="display: none;"></video>
    <canvas id="displayCanvas" style="background-color: magenta;"></canvas>
    <button onclick="startWebcam(); this.parentElement.removeChild(this)">Start webcam</button>
    <script type="text/javascript">
      function startWebcam() {
        const webcamVideoEl = document.getElementById("webcamVideo");
        const blitCanvas = new OffscreenCanvas(0, 0);  // size dynamically assigned per frame
        const blitCtx = blitCanvas.getContext("2d");
        const displayCanvasEl = document.getElementById("displayCanvas");
        const displayCtx = displayCanvasEl.getContext("2d");
        navigator.mediaDevices.getUserMedia({ video: { facingMode: "user" } }).then(stream => {
            webcamVideoEl.srcObject = stream;
            webcamVideoEl.play();
            function processFrame(now, metadata) {
              // downsample to this width (more sophisticated could dynamically choose size)
              const canvasWidth = 320;

              // use aspect ratio of latest frame
              const height = canvasWidth * metadata.height/metadata.width;

              // note this clears the canvases (at least in Chrome)
              blitCanvas.width = canvasWidth;
              blitCanvas.height = height;
              displayCanvasEl.width = canvasWidth;
              displayCanvasEl.height = height;

              // Downsamples video to canvas size
              blitCtx.drawImage(webcamVideoEl, 0, 0, canvasWidth, height);
              const imageData = blitCtx.getImageData(0, 0, canvasWidth, height);

              const numPixels = imageData.data.length / 4;
              for (let i = 0; i < numPixels; i++) {
                const r = imageData.data[i * 4 + 0];
                const g = imageData.data[i * 4 + 1];
                const b = imageData.data[i * 4 + 2];
                if (g > 100 && r < 100) imageData.data[i * 4 + 3] = 0;  // crude green screen
              }
              displayCtx.putImageData(imageData, 0, 0);
              webcamVideoEl.requestVideoFrameCallback(processFrame);
            }
            webcamVideoEl.requestVideoFrameCallback(processFrame);
        }).catch(error => {
          console.error(error);
        });
      }
    </script>
  </body>
</html>
Tagged #webgl, #greenscreen, #video, #graphics-programming, #javascript, #web, #programming.

Similar posts

More by Jim

Want to build a fantastic product using LLMs? I work at Granola where we're building the future IDE for knowledge work. Come and work with us! Read more or get in touch!

This page copyright James Fisher 2020. Content is not associated with my employer. Found an error? Edit this page.