JeelizAR is a JavaScript library that can detect elements in a video feed. By giving us the position of the detected object, it allows us to add augmented reality (AR) elements. Let’s learn together how to build amazing AR experiences for our users, in the **browser.

Adding the source files to our project

We’ll start off by setting up our index.html, where we’ll build our JeelizAR demo. Let’s add our source file. For this project, we’ll need the JeelizAR script, the JeelizMediaStreamAPIHelper script and the three.js script. The JeelizAR script will let us use all of the JeelizAR features. The JeelizMediaStreamAPIHelper will handle in a simple way getting the video stream from the webcam. And three.js will let us add AR elements and manipulate them. Between our head tags, let’s add:

<script src="./dist/JeelizAR.js"></script>
<script src="https://appstatic.jeeliz.com/helpers/JeelizMediaStreamAPIHelper.js"></script>
<script src="https://appstatic.jeeliz.com/threejs/v97/three.min.js"></script>

Now we’ll set up the basic boilerplate to run the JeelizAR API in our web app.

Setting up the JeelizAR API

The boilerplate to run the JeelizAR API is shown in another tutorial, Getting started with the JeelizAR API. We’ll quickly set it up today, but if you’d like to have a better understanding of is going on here, don’t hesitate to check this first post out.
We’ll start off by adding a canvas tag, where the JeelizAR API will display the video feedback from our webcam.

<canvas id="debugJeeARCanvas" style="width: 800px; height: 600px;"></canvas>

Then, we’ll add the javascript code to run the API between a pair of script tags:


const DOMVIDEO = JeelizMediaStreamAPIHelper.get_videoElement();

JeelizMediaStreamAPIHelper.get(DOMVIDEO, init, () =&gt; {
  alert('Cannot get video bro :(');
}, {
  video: true, // mediaConstraints
  audio: false
});

function init() {
  JEEARAPI.init({
    canvasId: 'debugJeeARCanvas',
    video: DOMVIDEO,
    callbackReady: (errLabel) =&gt; {
      if (errLabel) {
        alert('An error happens bro: ', errLabel);
      } else {
        load_neuralNet();
      }
    }
  });
}

function load_neuralNet() {
  JEEARAPI.set_NN('./basic4.json', (errLabel) =&gt; {
    if (errLabel) {
      console.log('ERROR: cannot load the neural net', errLabel);
    } else {
      iterate();
    }
  });
}

function iterate() {
  const detectState = JEEARAPI.detect(3);
  if (detectState.label) {
    console.log(detectState.label, 'IS DETECTED YEAH !!!');
  }
  window.requestAnimationFrame(iterate);
}

Let’s run our index.html file in a browser. The video feedback is displayed and the detection window is scanning across the canvas. Great ! We can now start building our AR experience by using the detectState object.

The structure of the detectState object

We’ll use the detectState object that we got with the .detect() method of the JEEAR API. This detectState object gives us:

  • a label, that categorizes the object detected
  • the detectScore of the object (between 0 and 1)
  • the yaw, the angle of rotation of the object on the Y axis
  • the distance between the object and the camera
  • the positionScale, which is an array containing the scale of the object (sx, sy) and the position (x, y) of the object

With all this data we should be able to place AR elements that will match the position of the element detected.

Setting up the three.js scene

We’ll use the javascript library three.js to display and manipulate 3D elements in our project. Thus, we need to initialize the library, and create our scene, which is the space in which our 3D objects will exist and be displayed. To do so we’ll create two functions, createScene() and animate(), that will respectively set up the three.js scene and start the animation process.


let camera, scene, renderer;
function createScene() {
  camera = new THREE.PerspectiveCamera( 70, window.innerWidth / 
  window.innerHeight, 0.01, 10 );
  camera.position.z = 1;

  scene = new THREE.Scene();

  renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
  renderer.setSize(800, 600);
  renderer.domElement.style.position = 'fixed';
  renderer.domElement.style.left = '0';
  renderer.domElement.style.top = '0';
  renderer.setClearColor(0xffffff, 0);
  document.body.appendChild(renderer.domElement);

  animate();
}

function animate() {
  requestAnimationFrame(animate);

  renderer.render(scene, camera);

}

Our three.js scene is created. We’ll now need to add our 3D model.

Adding a 3D model

For this tutorial, we’ll simply create a 3D cube, that we’ll display at the position of the detected object. Let’s begin by creating the cube. In the createScene() function:

...
// We build the geometry of the object, its "shape" of sorts
const geometry = new THREE.BoxGeometry(0.2, 0.2, 0.2);
// And we build its material
const material = new THREE.MeshNormalMaterial();
...

With the geometry and the material of our object we can build a mesh, and add it to our scene. In order to be able to manipulate the position of our mesh we’ll declare the variable holding its reference in the global scope.


// We declare our variable `MESH`
let MESH;
function createScene() {
...
  MESH = new THREE.Mesh(geometry, material);
  // and add it to our scene
  scene.add(MESH);
...

Our model is now added ! As you can see below it is displayed in the center of our video feedback. We now want it to be displayed only when an object is detected. Let’s code that.

Displaying our mesh conditionally

We’ll now create a function to display/hide our model, when the target object is detected/not detected. We’ll call this method toggleDisplayMesh(). We’ll also create a variable in the global scope named IS_DETECTED that we’ll use in our toggleDisplayMesh function.


let IS_DISPLAYED = false;
function toggleDisplayMesh(displayMesh) {
  // Return if the mesh has already been showed/hidden
  if (displayMesh === IS_DISPLAYED) return;
  // If we want to display the mesh and if the mesh is not already displayed
  // show it
  if (displayMesh && !IS_DISPLAYED) {
    MESH.visible = true;
    IS_DISPLAYED = true;
    // If we want to hide the mesh and if the mesh is already displayed
    // hide it
   } else if (!displayMesh && IS_DISPLAYED) {
     MESH.visible = false;
     IS_DISPLAYED = false;
   }
}

Then in the iterate function we’ll add:


function iterate() {
  const detectState = window.JEEARAPI.detect(3);
  if (detectState.label) {
    console.log(detectState.label, 'IS DETECTED YEAH !!!');
    // Show mesh
    toggleDisplayMesh(true);
  } else {
    // Hide mesh
    toggleDisplayMesh(false);
  }
  window.requestAnimationFrame(iterate);
}

Great! The last step is to hide by default the mesh, so that it is shown only when a first object is detected. In createScene():


...
const geometry = new THREE.BoxGeometry( 0.2, 0.2, 0.2 );
const material = new THREE.MeshNormalMaterial();

MESH = new THREE.Mesh(geometry, material);
// We hide the MESH by default
MESH.visible = false;
...


As we can see on the capture above, our purple box is displayed when the cup is detected.

Positioning our mesh dynamically

We successfully managed to display conditionally our mesh. Now’s the time to position it, so that it matches the position of the detected object. In that purpose, we’ll use the detectState object.

We’ll create a new function called updateMeshPosition():


function updateMeshPosition(detectState) {
  // We grab the projection
  const A = camera.projectionMatrix.elements[0];
  const B = camera.projectionMatrix.elements[5];
  // The z position of our mesh
  const z = -10;

  // A quick formula and..
  const newX = -detectState.positionScale[0]*z/A;
  const newY = -detectState.positionScale[1]*z/B;

  // ..we update the position of the mesh
  MESH.position.x = newX;
  MESH.position.y = newY;
}

This function needs to be run at each iteration of our render loop, only if an object is detected. Let’s add a call to our function in iterate():


...
if (detectState.label) {
  console.log(detectState.label, 'IS DETECTED YEAH !!!');
  toggleDisplayMesh(true);
  // We call our updateMeshPosition() function
  updateMeshPosition(detectState);
} else {
  toggleDisplayMesh(false);
}
...

Let’s check it out. Our mesh successfully follow our detect object. This is a basic implementation, but virtually anything can be done. Load a custom mesh of a boat, and display it in a detected cup. Load a 2D animation of a man riding a bicycle and add it to a detected bike. Let us know what you managed to build!