How to develop a Web AR Facefilter with React and ThreeJS / React Three Fiber in 2021

In this post i'll walk through how we built a small, simple Web AR Facefilter with free resources from Jeeliz and Sketchfab.

7 min read

For the 3 year anniversary of 🧙🏼‍♂️ Level30Wizards we wanted to build something that's related to our work, brand and interests. We finally ended up creating a Snapchat like facefilter for Web AR or Web XR.

You can try the final product here: Facefilter demo

Shameless plug

Hey, are you looking to build a Facefilter for a client or project? We'd love to help.

Send an email!

Intro

Mobile web browsing is advancing fast. We are able to get data from the NFC reader, Storage and also camera. Sadly, as of now, we are not allowed to use the ARKit on iPhones or iPads.

There are a lot of great libraries like 8th Wall or Zappar, but they usually cost a pretty penny.
Luckily, lots of smart people build neural networks that can recognize faces from 2D screens.

WebAR allows you to utilize computer or data visualization without the need of downloading an app, besides your default web browser. Web AR can use both SLAM (Simultaneous Localization and Mapping) and Image Recognition technologies to digitally visualize your content. Since it's still a webpage, we can add a variety of web features and functionality to the AR experience. This opens up loads of opportunities on the web.

Augmented reality will be a valuable addition to a lot of existing web pages. For example, it could help people learn on education sites, or allow potential buyers to visualize objects in their home while shopping.

Goal

The goal is to project a 3D model on top of our face.

Tools

Prototyping disclaimer

This experiment we built, is a complete prototype. We do not recommend you use this in a production environment. This example needs a lot of tweaks before it's viable to use in production.

Code: Introduction

Before we show the camera, we need some time to load the 3D model and calculate some positions. That's why we opted for an introduction screen that add some suspense.

The pretty wavy text is from a previous tutorial! Wavy Text Animation using React Hooks with GSAP v3

1import gsap from 'gsap';
2import React, { useEffect } from 'react';
3
4export const Intro = () => {
5  useEffect(() => {
6    gsap.set('.wavy', { perspective: 400 });
7
8    const sequence = (id, reverse) => {
9      const tl = gsap.timeline({ delay: 0.5 });
10
11      tl.from(`.wavy[data-id="${id}"]`, {
12        duration: 0.5,
13        autoAlpha: 0,
14        ease: 'back',
15        stagger: 1,
16      })
17        .from(`.wavy[data-id="${id}"] [data-letter]`, {
18          duration: 0.5,
19          autoAlpha: 0,
20          scale: 1,
21          y: -30,
22          rotationX: -90,
23          transformOrigin: '0% 50% -50',
24          ease: 'back',
25          stagger: 0.025,
26        })
27        .to(`.wavy[data-id="${id}"] [data-letter]`, {
28          duration: 0.5,
29          delay: 1.25,
30          autoAlpha: 0,
31          scale: 1,
32          y: -30,
33          rotationX: -90,
34          transformOrigin: '0% 50% -50',
35          ease: 'back',
36          stagger: 0.025,
37        })
38        .to(`.wavy[data-id="${id}"]`, {
39          duration: 0.25,
40          autoAlpha: 0,
41          ease: 'back',
42          stagger: 1,
43        });
44
45      return tl;
46    };
47
48    const tl = gsap.timeline();
49
50    tl.add(sequence('one'))
51      .add(sequence('two'))
52      .add(sequence('three'))
53      .to('.intro', {
54        duration: 1,
55        autoAlpha: 0,
56      });
57
58    tl.timeScale(1.2);
59
60    tl.play();
61    return () => {
62      tl.kill();
63    };
64  }, []);
65
66  return (
67    <div className="intro">
68      <AnimatedText id="one" text={'Level30Wizards Presents...'} />
69      <AnimatedText id="two" text={'Are you a true wizard?'} />
70      <AnimatedText id="three" text={"Let's find out!"} />
71      <img
72        src="https://level30wizards.com/images/branded/brand-image-6.png"
73        alt="Dragon"
74        className="dragon"
75      />
76      <a href="https://level30wizards.com" aria-label="level30wizards">
77        <img
78          src="https://level30wizards.com/images/svg/logo.svg"
79          alt="Level30Wizards"
80          className="logo"
81        />
82      </a>
83    </div>
84  );
85};

Wizards hat

To load in a 3D model, we used GLTFJSX.

1npx gltfjsx model.gltf -t

This results in this:

1// @ts-nocheck
2/*
3Auto-generated by: https://github.com/pmndrs/gltfjsx
4author: Paulina (https://sketchfab.com/Byakko)
5license: CC-BY-NC-4.0 (http://creativecommons.org/licenses/by-nc/4.0/)
6source: https://sketchfab.com/3d-models/wizards-hat-68a9fb2dbd8442a5bacf9c0141320308
7title: Wizard's hat
8*/
9
10import { useGLTF } from '@react-three/drei';
11import React, { useRef } from 'react';
12
13export default function Model(props) {
14  const group = useRef();
15  const { nodes, materials } = useGLTF('/models/wizards_hat/scene.gltf');
16  return (
17    <group ref={group} {...props} dispose={null}>
18      <group rotation={[-Math.PI / 2, 0, 0]}>
19        <group rotation={[Math.PI / 2, 0, 0]}>
20          <mesh
21            geometry={nodes.defaultMaterial.geometry}
22            material={nodes.defaultMaterial.material}
23          />
24          <mesh
25            geometry={nodes.defaultMaterial_1.geometry}
26            material={nodes.defaultMaterial_1.material}
27          />
28          <mesh
29            geometry={nodes.defaultMaterial_2.geometry}
30            material={nodes.defaultMaterial_2.material}
31          />
32          <mesh
33            geometry={nodes.defaultMaterial_3.geometry}
34            material={nodes.defaultMaterial_3.material}
35          />
36          <mesh
37            geometry={nodes.defaultMaterial_4.geometry}
38            material={nodes.defaultMaterial_4.material}
39          />
40          <mesh
41            geometry={nodes.defaultMaterial_5.geometry}
42            material={nodes.defaultMaterial_5.material}
43          />
44        </group>
45      </group>
46    </group>
47  );
48}
49
50useGLTF.preload('/models/wizards_hat/scene.gltf');

Web AR Library

We used Jeeliz R3F Demo and combined that with some logic from their Matrix Demo. If you want to, without any shame, copy paste this blogpost to create a working product... You're out of luck. Refer to the demos and tinker yourself, you'll probably learn more.

The resulting code that did most of the work looked a bit like this:

1const FaceFollower = props => {
2  const objRef = useRef();
3  useEffect(() => {
4    const threeObject3D = objRef.current;
5    _faceFollowers[props.faceIndex] = threeObject3D;
6  });
7
8  return (
9    <object3D ref={objRef}>
10      <Suspense fallback={null}>
11        <WizardsHat
12          rotation={[0, -Math.PI, 0]}
13          position={[0, 1.825, 0]}
14          scale={[1.5, 1.5, 1.5]}
15          renderOrder={2}
16        />
17        <Head position={[0, -0.1435, 0]} scale={[1.125, 1, 1.125]} />
18      </Suspense>
19    </object3D>
20  );
21};

We used a model of a human's head and used a trick with colorWrite and renderOrder. We just used some human head from SketchFab, low-poly is fine.

1const hiderMat = new THREE.MeshPhongMaterial({
2  attach: 'material',
3  color: 'hotpink',
4  colorWrite: false,
5  renderOrder: 1,
6});

If we didn't do this, the hat would clip your face when you throw your head back in laughter, because the web ar experience is so amazing.

Snapshot!

It wouldn't be a facefilter if you couldn't take a photo while making a peace-sign and touting your lips like a duck.

For our next trick, I used 3 canvas elements. Yes, we know.

We combine the camera canvas, the canvas containing the "hat" and "head" and the canvas on which we project our logo.

1const faceFilterCanvasRef = useRef(null);
2const canvasRef = useRef(null);
3const pictureCanvasRef = useRef(null);
4const logoCanvasRef = useRef(null);
5
6const snapshot = useCallback(() => {
7  const canvas = pictureCanvasRef.current;
8  canvas.getContext('2d').drawImage(faceFilterCanvasRef.current, 0, 0);
9
10  const img = new Image();
11  img.src = '/images/logo.png';
12
13  img.addEventListener('error', e => {
14    console.error(e);
15  });
16  img.addEventListener('load', e => {
17    logoCanvasRef.current
18      .getContext('2d')
19      .drawImage(
20        img,
21        sizing.width - (144 + 24),
22        sizing.height - (42.38 + 24),
23        144,
24        42.38
25      );
26
27    mergeImages([
28      canvas.toDataURL('image/png'),
29      canvasRef.current.toDataURL('image/png'),
30      logoCanvasRef.current.toDataURL('image/png'),
31    ]).then(b64 => {
32      setDownloadUrl(b64);
33      setShareData(b64);
34    });
35  });
36}, []);

Showing the final image

To download and share your photo, we tried to implement the Native Share API. But, that API required us to save the images, which we didn't like since that's a privacy concern. Please take that into consideration if you build this yourself. Use a token or some way to verify the user looking at the image is from a shared link or the current user.

1<div
2  style={{
3    display: shareData ? 'block' : 'none',
4    position: 'fixed',
5    zIndex: 3,
6    borderRadius: '50%',
7    overflow: 'hidden',
8  }}
9>
10  <img
11    style={{
12      position: 'fixed',
13      zIndex: 4,
14      width: 'auto',
15      height: '100%',
16      top: '50%',
17      left: '50%',
18      transform: 'translate(-50%,-50%)',
19    }}
20    src={shareData}
21    alt="generated"
22  />
23</div>

That's all thanks.

PS.

See an example in action!.
We're working on some magical things at 🧙🏼‍♂️ Level30Wizards, be sure to follow us on socials to keep an eye out for new releases.

Thanks for reading!

We hope you've learned something about augmented reality in web browsers from this article.
If you use this tech, please do so responsibly.