The FaceFilter library allows you to effortlessly create Snapchat-like face filters in your web-app. It is built on top of our face detection neural network. It combines the power of deep learning with the versatility of WebGL/Three.js. To understand this tutorial you only need to have a medium knowledge of JavaScript and a good grasp of Three.js basics. Now let’s create our first filter!

Creating our project folder

First step: clone or download the FaceFilterAPI github repository from https://github.com/jeeliz/jeelizFaceFilter. The demos are located in ./demos/threejs. In this tutorial we build a very simple face filter where we’ll add a Luffy’s hat from One Piece! The cube demo is a very good place to start our project: this filter adds a 3D cube to the detected face, which is fairly close to what we aim to do. Let’s duplicate the “cube” folder, and rename it luffys_hat. N.B.: Running the FaceFilter API requires our assets to be served through a local HTTP server. In the command line from the root of our project folder, launch the httpServer.py script and visit localhost:8455 to check out the demos.

Project Structure

Let’s begin by having a look at how our newly created folder is structured:

Pretty simple, huh? The code that adds our filter is located in demo.js. If you have a look at it, you’ll see that the code is divided into two functions:

  • main(): it handles the FaceFilter API initialization
  • init_threeScene(): it handles the creation of the three.js scene, along with what we place into that scene. This is where our hat will be added.

For now, a cube is positioned at the center of the user’s face. Let’s replace it by our cool hat!

We will do so by following two steps:

  1. Use Blender to export our hat model to JSON. SPOILER ALERT: this will be the more challenging part
  2. Import the JSON file in our project and add it to our scene

Exporting our JSON 3D model from Blender

We use Blender, an open-source 3D computer graphics modelling software. It will allow us to convert our 3D model to JSON, which is a super efficient and compact format. In Blender, open the File tab, click on Import file and then click on the adequate format. You should now see the hat in the middle of your viewport.

We will now need the specific Three.js exporter in order to get our hat in JSON. To this end, we’ll have to download or clone the Three.js repo on Github (https://github.com/mrdoob/three.js/) and change directories to three.js/utils/exporters/blender/addons/. Once there, you should find a folder named io_three. We will compress it in a zip file that we’ll use to install the exporter. Let’s go back to Blender. Firstly, click on the File tab, then on Preferences. Click on the Add-ons tab and then, at the bottom, on Install Add-on from File…. Select your newly zipped folder. The last step is to use the search bar on the top left of the user preferences tab to find the Three add-on in your list, then to activate it by clicking on the checkbox.

N.B.: Click on the Save User Settings to avoid repeating this process each time you want to export a model in JSON. Almost done! We will now export our model using the Three.js exporter (File > Export > Three.js(JSON)). Make sure you export all the textures, and materials in their respective tabs. All set up! Select our project directory, create a models folder and export Luffy’s hat.

Alright, now let’s get back to coding!

Coding our filter

First off, we’ll need to remove the cube from our Three.js scene. Comment the line where the cube is added:

// THREEFACEOBJ3DPIVOTED.add(threeCube);

We create a BufferGeometryLoader and call its .load() method, passing a source and a callback as arguments.

// Create the BufferGeometryLoader for our hat
var loader = new THREE.BufferGeometryLoader()
  // Load our cool hat
  loader.load(
    ‘models/luffys_hat.json’,
    function (geometry, materials) {
    }
  )
}

We then create a material and pass it to a mesh along with the geometry object:

loader.load(
  ‘models/luffys_hat.json’,
  function (geometry, materials) {
    // we create our Hat material
    var mat=new THREE.MeshBasicMaterial({
      // load the texture using a TextureLoader
      map: new THREE.TextureLoader().load( “models/Texture.jpg” )
    });
    // and finally create our mesh
    const hatMesh = new THREE.Mesh(geometry, mat)
  }
)

You can modify the scale, the rotation and position of our hat by using:

loader.load(
  ‘models/luffys_hat.json’,
  function (geometry, materials) {
  // we create our Hat mesh
  var mat=new THREE.MeshBasicMaterial({
    map: new THREE.TextureLoader().load( “models/Texture.jpg” )
  });
  const hatMesh = new THREE.Mesh(geometry, mat)
  // USE THESE METHODS TO MODIFY
  // THE SCALE, ROTATION and POSITION
  hatMesh.scale.multiplyScalar(1.2);
  hatMesh.rotation.set(0, -40, 0);
  hatMesh.position.set(0.0, 0.6, 0.0);
  }
)

Finally, we can add our hat model to our scene:

THREEFACEOBJ3DPIVOTED.add(hatMesh)

And tadaa! You are now wearing your very own Luffy’s hat! How cool is that?? Though this example is basic, you can now use what you learnt to display other types of objects.

Exercises:

  1. try adding another object to the scene: earrings, glasses (visit: https://www.turbosquid.com/ to find free 3D models)
  2. Play around with the position and the scale parameters, and give yourself a goofy style!

In conclusion

Congrats on creating your first filter with Jeeliz’s FaceFilter API!

Through this tutorial, we saw how to process a 3D model with Blender. We also learned how to add a 3D model to our Three.js scene. Pretty cool huh!? Still, our filter isn’t 100% realistic: the shadows are off, and when the user raises their head the hat covers their face. Not to worry! We will be improving our Luffy’s hat filter in an upcoming tutorial. In the next part: we will create a slightly more complex filter involving user interaction and animation! Stay tuned!