The VisageTrackerUnityDemo sample project demonstrates the integration of visage|SDK with Unity game engine and is aimed at developers starting to use VisageTracker, VisageFaceAnalyser and VisageFaceRecogntion functionalities of visage|SDK in Unity game engine and development tool.
The sample project shows how face tracking is used to put virtual object (glasses) on multiple faces in Unity, how to access the full 3D face model of the face and how to obtain and display face analysis (gender, age and emotion estimation) and face recognition results. Developers may use parts of this project to speed up their own development in Unity using visage|SDK or use this application as a starting point for own projects.
The application needs to be built from Unity project which is located in Samples/iOS/VisageTrackerUnityDemo. Detailed instructions for building and running the project is available in section Generating the Xcode project from Unity. If you wish to integrate visage|SDK in your own Unity project, instructions can be found here.
Prerequisites to running the sample project are:
To generate the Xcode project for this sample follow these steps:
NOTE: Once you link with iOS 10 you must declare access to any user private data types (camera, photos, microphone, etc.). You do this by adding a usage key to your app’s Info.plist together with a purpose string.
When the application starts and if a face is found in the camera frame, tracking from camera starts immediately. Pause/Resume button is placed in the bottom left corner and Switch camera button in the bottom right corner of the screen. On top of the video frames is a 3D model of glasses that is transformed with
the information about rotation and translation from the tracker. It blends seamlessly with the background video.
In the top left corner is a button which enables face recognition feature. After a certain face appears in the frame for the first time, there is an initialization phase where a predefined number of frames is used to collect data for face recognition. After that, the face gets a label in the format Person[N] (Person0, Person1...,Person[N]). All identities collected during the run of the application will additionally be displayed in a list on the left of the screen.
NOTE: Without a valid license an error message will appear and face tracking will not work.
The project consists of a main scene with the Unity’s GameObjects that provide different functionalities. A GameObject is the main building block of a scene in Unity. It consists of different Components and can parent other GameObjects. Manipulation of GameObjects is done with Scripts. It is recommended for developers to get familiar with Unity basics before continuing. The integration with visage|SDK is done through a plugin (VisageTrackerUnityPlugin) that wraps native code calls to visage|SDK and provides functions that can be called from Unity scripts.
Tracker.cs contains the code for device-specific selection of tracking configuration, ensuring optimal tracking performance on different devices by taking into account their performance, camera and screen characteristics. Please read more details about configuration selection in the section Device-specific configuration selection.
The core of the example is the Tracker GameObject. It consists of different components attached to it. The main component of the Tracker GameObject is the Tracker script component that handles the communication with the tracker (starts it and gets results from it). The behavior of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Tracker script properties of interest are as follows:
Analyser GameObject is used to obtain and display face analysis results.
It relies on the functionality of the Tracker object, i.e. Tracker script component.
The GameObject consists of different components attached to it. The main
component of the Analyser GameObject is the Analyser script component that handles the communication with the face analyser
(initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script
code. Other properties can be added as well. The existing Analyser script properties of interest are as follows:
Recognition GameObject is used to obtain and display face recognition results.
It relies on the functionality of the Tracker object, i.e. Tracker script component.
The GameObject consists of different components attached to it. The main
component of the Recognition GameObject is the Recognition script component that handles the communication with the face recognition
(initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script
code. Other properties can be added as well. The existing Recognition script properties of interest are as follows:
There are two cameras in the scene, one for 3D scene (Main Camera) and other for video display (Video Camera). The main camera "Field of View" is automatically set according to the Tracker focus (camera_focus parameter in configuration file) and input image aspect by the Tracker script. The Video Camera renders objects in the Video layer by using orthographic projection.
Video GameObject contains Video Plane that is used for displaying video and is automatically scaled to input frame aspect ratio by the VideoPlaneController script attached to it.
Since the Glasses object is in the Tracker script Controllable Objects list, rotation and translation information from the tracker is applied to it. This enables the overlaying of virtual objects to the tracked face that transform correctly with it. Other custom objects can be used instead of the glasses object. For a more comprehensive guide on how to achieve occlusion or on how to properly introduce other objects to the scene see AR modeling guide.
The integration of visage|SDK face tracking with Unity is done through a plugin that wraps native code calls and provides functions that can be called from Unity scripts. The provided prebuilt plugin VisageTrackerUnityPlugin.a can be used as-is. The project and the source code for the VisageTrackerUnityPlugin plugin are also provided and can be used to implement different functionality if needed. For more information about the plugin including it's source code, see the VisageTrackerUnityPlugin project and its documentation.