Menu

Creating a Snapchat-like face filter 1/2: Creating your first filter

, , , , , , ,

The FaceFilter API allows you to effortlessly create Snapchat-like filters in your web-app. Built on top of our face detection neural network, it combines the power of deep learning with the versatility of WebGL/Three.js.

Prerequisites: for this tutorial you will only need to have good knowledge of Javascript and a good grasp of Three.js basics.

Okay, let’s create our first filter!

Creating our project folder

First step: clone or download the faceFilterAPI repo (https://github.com/jeeliz/jeelizFaceFilter). The demos are located at ./demos/threejs. For this tutorial, we’ll build a very simple filter where we’ll add a Luffy’s hat from One Piece!

The “cube” demo is a very good place to start our project: this filter adds a 3D cube to the detected face, which is fairly close to what we aim to do.
Let’s duplicate the cube folder, and rename it luffys_hat.

N.B.: Running the FaceFilter API requires our assets to be served through a local server. In the command line from the root of our project folder, launch the httpServer.py script and visit localhost:8455 to check out the demos.

Project Structure

Let’s begin by having a look at how our newly created folder is structured:
Pretty simple, huh? The code that adds our filter is located in demo.js. If you have a look at it, you’ll see that the code is divided into two functions:

  • main(): that handles the faceFilter API initialization
  • init_threeScene(): that handles the creation of the three.js scene, along with what we place into that scene. This is where our hat will be added.

For now, a cube is positioned at the center of the user’s face. Let’s replace it by our cool hat!

We will do so by following two steps:

  1. Use Blender to export our hat model to JSON. SPOILER ALERT: this will be the most challenging part
  2. Import the JSON file in our project and add it to our scene

Exporting our JSON 3D model from Blender

We’ll have to use Blender, which is an open-source 3D computer graphics software. It will allow us to convert our 3D model to JSON, which is a super efficient and low sized format.

In Blender, open the File tab, click on import file and then click on the adequate format. You should now see the hat in the middle of your viewport.

We need the specific Three.js exporter in order to get our hat in JSON. To this end, we have to download or clone the Three.js repo on Github (https://github.com/mrdoob/three.js/) and change directories to three.js/utils/exporters/blender/addons/. Once there, you should find a folder named io_three. We will then compress this folder in a zip file that we’ll use to install the exporter.
Let’s go back to Blender. Firstly, click on the File tab, then on Preferences. Click on the Add-ons tab and then, at the bottom, on Install Add-on from File….
Select your newly zipped folder.

The last step is to use the search bar on the top left of the user preferences tab to find the Three add-on in your list, then to activate it by clicking on the checkbox.

N.B.: In order not to have to redo this process each time you’ll want to export a model in JSON, click on the Save User Settings.

Almost done!

We now export our model, using the Three.js exporter (File > Export > Three.js(JSON)). Make sure you export all the textures, and materials in their respective tabs.
All set up! Select our project directory, create a models folder and export Luffy’s hat.

Alright, now let’s get back to coding!

Coding our filter

First off, we’ll need to remove the cube from our Three.js scene. Comment the line where the cube is added:

// THREEFACEOBJ3DPIVOTED.add(threeCube);

We’ll then create a BufferGeometryLoader and call its .load() method, passing a source and a callback as arguments.

// Create the BufferGeometryLoader for our hat
const loader = new THREE.BufferGeometryLoader()
  // Load our cool hat
  loader.load(
    'models/luffys_hat.json',
    function (geometry, materials) {
    }
  )
}

We then create a material and pass it to a mesh along with the geometry object:

loader.load(
  'models/luffys_hat.json',
  function (geometry, materials) {
    // we create our Hat material
    const mat = new THREE.MeshBasicMaterial({
      // load the texture using a TextureLoader
      map: new THREE.TextureLoader().load( “models/Texture.jpg” )
    });
    // and finally create our mesh
    const hatMesh = new THREE.Mesh(geometry, mat)
  }
)

You can modify the scale, the rotation and position of our hat by using:

loader.load(
  'models/luffys_hat.json',
  function (geometry, materials) {
  // we create our Hat mesh
  const mat = new THREE.MeshBasicMaterial({
    map: new THREE.TextureLoader().load( “models/Texture.jpg” )
  });
  const hatMesh = new THREE.Mesh(geometry, mat)
  // USE THESE METHODS TO MODIFY
  // THE SCALE, ROTATION and POSITION
  hatMesh.scale.multiplyScalar(1.2);
  hatMesh.rotation.set(0, -40, 0);
  hatMesh.position.set(0.0, 0.6, 0.0);
  }
)

Finally, we can add our hat model to our scene:

THREEFACEOBJ3DPIVOTED.add(hatMesh);

And tadaa! You are now wearing your very own Luffy’s hat! How cool is that?? Though this example is basic, you can now use what you learnt to display other types of objects.

Exercises:

  1. try adding another object to the scene: earrings, glasses (visit: https://www.turbosquid.com/ to find free 3D models)
  2. Play around with the position and the scale parameters, and give yourself a goofy style!

Sum up

Congrats on creating your first filter with Jeeliz’s FaceFilter API!

Through this tutorial, we saw how to process a 3D model with Blender. We also learned how to add a 3D model to our Three.js scene. Pretty cool huh!? Still, our filter isn’t 100% realistic: the shadows are off, and when the user raises their head the hat covers their face. Not to worry! We will be improving our Luffy’s hat filter in an upcoming tutorial.

In the next part: we will create a slightly more complex filter involving user interaction and animation! Stay tuned!

Related Posts