VisageTrackerUnity
visage|SDK VisageTrackerUnity example project

The VisageTrackerUnity sample project demonstrates the integration of visage|SDK with Unity game engine and is aimed at developers starting to use face tracking, face analysis and face recognition functionalities of visage|SDK in Unity game engine and development tool.

The sample project shows how face tracking is used to put a virtual object (glasses) on the user's face in Unity, and how to access the full 3D face model of the face. Sample also demonstrates how to use face tracking results to obtain face analysis and face recognition results. Developers may use parts of this project to speed up their own development in Unity using visage|SDK or use this application as a starting point for own projects.
The application needs to be built from Unity project which is located in Samples/Unity/VisageTrackerUnityDemo. Detailed instructions for building and running the project are available in section Building the project. If you wish to integrate visage|SDK in your own Unity project, instructions can be found here.

Building the project

The files related to VisageTrackerUnity sample project are located in the Samples/Unity/VisageTrackerUnityDemo subfolder of the visage|SDK folder.

The application was built and tested with version 2020.3.21f1.

Building the application

To build the application follow these steps:

  1. Open VisageTrackerUnityDemo project (Samples/Unity/VisageTrackerUnityDemo) in Unity.
  1. Add license key to the application (more information can be found in Licensing section of the documentation).
  2. Select and open the Main scene. Please make sure the Analyser.cs script is attached to the Anaylser object in the Main scene.
  3. Select File->Build settings and under Platform choose PC and Mac Standalone and set Windows as target platform. Check if the correct Architecture is set: x86_64.
  4. Press the Build button to generate the project, a dialog window will appear. In this window, locate and set a destination folder to Samples/Unity/VisageTrackerUnityDemo/Build (the Build folder must first be created).

Using the sample application


In order to try the sample application, you need to have a valid license key. More information on how to obtain license key and include it in the application can be found in Licensing section of the documentation.

Tracking will start automatically when a face is found in the camera frame. Available buttons are:

On top of the video frames is a 3D model of glasses or tiger model (depending on which effect is chosen) that is transformed with the information about rotation and translation from the tracker. It blends seamlessly with the background video.

Implementation overview

The project consists of a main scene with the Unity’s GameObjects that provide different functionalities. A GameObject is the main building block of a scene in Unity. It consists of different Components and can parent other GameObjects. Manipulation of GameObjects is done with Scripts. It is recommended for developers to get familiar with Unity basics before continuing. The integration with visage|SDK is done through a plugin (VisageTrackerUnityPlugin) that wraps native code calls to visage|SDK and provides functions that can be called from Unity scripts.

Tracker GameObject

The core of the example is the Tracker GameObject. It consists of different components attached to it. The main component of the Tracker GameObject is the Tracker script component that handles the communication with the tracker (starts it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Tracker script properties of interest are as follows:

Note: Because of the different coordinate systems used by the visage|SDK face tracker and the Unity game engine mirroring around x-axis is applied to the relevant GameObjects.

Objects in the scene that are also in the Controllable Objects list are the Glasses Model GameObjects.

Analyser GameObject

Analyser GameObject is used to obtain and display face analysis results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Analyser GameObject is the Analyser script component that handles the communication with the face analyser (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Analyser script properties of interest are as follows:

Recognition GameObject

Recognition GameObject is used to obtain and display face recognition results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Recognition GameObject is the Recognition script component that handles the communication with the face recognition (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Recognition script properties of interest are as follows:

Cameras

There are two cameras in the scene, one for 3D scene (Main Camera) and other for video display (Video Camera). The main camera "Field of View" is automatically set according to the Tracker focus (camera_focus parameter in configuration file) and input image aspect by the Tracker script. The Video Camera renders objects in the Video layer by using orthographic projection.

Video object

Video GameObject contains Video Plane that is used for displaying video and is automatically scaled to input frame aspect ratio by the VideoController script attached to it. In this project the material used by the Video Plane is tinted green to differentiate it from the textured 3D face model that uses the same texture.

Glasses object

The rotation and translation information from the tracker are applied to the Glasses object as they are in the Tracker script Controllable Objects list. This enables overlaying virtual objects to the tracked face that transform correctly with it. Other custom objects can be used instead of the glasses object. For the correct size and positioning of the glasses objects and also on how to achieve occlusion refer to the Animation & AR modeling guide.

VisageTrackerUnityPlugin

The integration of visage|SDK face tracking with Unity is done through a plugin that wraps native code calls and provides functions that can be called from Unity scripts. The provided prebuilt plugin VisageTrackerUnityPlugin.dll can be used as-is. For more information about the plugin including it's source code, see the VisageTrackerUnityPlugin project and its documentation.