Visage Tracker Unity Demo sample project

The VisageTrackerUnityDemo project shows how face tracking is used to put virtual object (glasses) on multiple faces in Unity, how to access the full 3D face model of the face and how to obtain and display face analysis (gender, age and emotion estimation) and face recognition results. Developers may use parts of this project to speed up their own development in Unity using visage|SDK or use this application as a starting point for own projects.

The sample project demonstrates the integration of visage|SDK with Unity game engine and is aimed at developers starting to use VisageTracker, VisageFaceAnalyser and VisageFaceRecogntion functionalities of visage|SDK in Unity game engine and development tool.
The application needs to be built from Unity project which is located in Samples/Android/VisageTrackerUnityDemo. Detailed instructions for building and running the project are available in sections Building the project and Generating the Android application. If you wish to integrate visage|SDK in your own Unity project, instructions can be found here.

The application needs to be built with license key in order to work properly. More information on how to obtain license key and include it in the application can be found in Licensing section of the documentation.


Building the project

The files related to Visage Tracker Unity Demo sample project are located in the Samples/Android/VisageTrackerUnityDemo sub-folder of the visage|SDK folder.

The prerequisites to building and running the sample project are :


Generating the Android application

To generate Android application for this sample follow these steps:

  1. Open VisageTrackerUnityDemo project (Samples/Android/VisageTrackerUnityDemo) in Unity.
  2. Add license key to the application (more information can be found in Licensing section of the documentation).
  3. Generate the Android application by selecting File->Build settings and choosing Android as a target.
  4. Press the Build (or Build and run to directly install the application on your device) button to build the application, a dialog window will appear. In this window, locate and set destination folder for your application.
  5. Transfer the built apk file to your device.

In case you are experiencing issues while building or running the project, please check the following:

Using the application

When a face is found in the camera frame, tracking and face analysis starts. On top of the video frames is the textured 3D face model generated and animated by the tracker and a 3D model of glasses that is transformed with the information about rotation and translation from the tracker. You can switch between the two by pressing the button in the upper right corner. You can also switch between front and back facing camera and pause the application by pressing corresponding buttons.

In the top left corner is a button which enables face recognition feature. After a certain face appears in the frame for the first time, there is an initialization phase where a predefined number of frames is used to collect data for face recognition. After that, the face gets a label in the format Person[N] (Person0, Person1...,Person[N]). All identities collected during the run of the application will additionally be displayed in a list on the left of the screen.

A rectangle element which is displayed from chin down contains face analysis results (age, gender and emotion estimation).

Implementation overview

The project consists of a main scene with the Unity’s GameObjects that provide different functionalities. A GameObject is the main building block of a scene in Unity. It consists of different Components and can parent other GameObjects. Manipulation of GameObjects is done with Scripts. It is recommended for developers to get familiar with Unity basics before continuing. The integration with visage|SDK is done through the VisageTrackerUnityPlugin.

Pre-build step

VisageTrackerUnityDemo includes pre-build step which copies required visage|SDK libraries and data files inside VisageTrackerUnityDemo project. Specifically, libraries are copied to Assets/Plugins/Android and data files to Assets/StreamingAssets/Visage Tracker. This is implemented in script PreBuild.cs located in Assets/Editor and executed before build.

Tracker GameObject

The core of the example is the Tracker GameObject. It consists of different components attached to it. The main component of the Tracker GameObject is the Tracker script component that handles the communication with the tracker (starts it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Tracker script properties of interest are as follows:

Note: Because of the different coordinate systems used by the visage|SDK face tracker and the Unity game engine mirroring around x-axis is applied to the relevant GameObjects. Object in the scene that is also in the Controllable Objects list is the Glasses Model GameObject.

Analyser GameObject

Analyser GameObject is used to obtain and display face analysis results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Analyser GameObject is the Analyser script component that handles the communication with the face analyser (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Analyser script properties of interest are as follows:

Recognition GameObject

Recognition GameObject is used to obtain and display face recognition results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Recognition GameObject is the Recognition script component that handles the communication with the face recognition (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Recognition script properties of interest are as follows:

Cameras

There are two cameras in the scene, one for 3D scene (Main Camera) and other for video display (Video Camera). The main camera "Field of View" is automatically set according to the Tracker focus (camera_focus parameter in configuration file) and input image aspect by the Tracker script. The Video Camera renders objects in the Video layer by using orthographic projection.

Video object

Video GameObject contains Video Plane that is used for displaying video and is automatically scaled to input frame aspect ratio by the VideoController script attached to it.

Glasses object

The rotation and translation information from the tracker are applied to the Glasses object as they are in the Tracker script Controllable Objects list. This enables overlaying virtual objects to the tracked face that transform correctly with it. Other custom objects can be used instead of the glasses object. For a more comprehensive guide on how to achieve occlusion or on how to properly introduce other objects to the scene see AR modeling guide.

Visage Tracker Unity Plugin

The integration of visage|SDK face tracking with Unity is done through a plugin that wraps native code calls and provides functions that can be called from Unity scripts. For more information about the plugin including it's source code, see the VisageTrackerUnityPlugin project and its documentation.

Android Camera Plugin

Accessing and handling Android camera is done through Java plugin. For more information about the plugin including it's source code, see the AndroidCameraPlugin project and its documentation.