Visage Tracker Unity Demo sample project

The VisageTrackerUnityDemo sample project demonstrates the integration of visage|SDK with Unity game engine and is aimed at developers starting to use VisageTracker, VisageFaceAnalyser and VisageFaceRecogntion functionalities of visage|SDK in Unity game engine and development tool.

The sample project shows how face tracking is used to put virtual object (glasses) on multiple faces in Unity, how to access the full 3D face model of the face and how to obtain and display face analysis (gender, age and emotion estimation) and face recognition results. Developers may use parts of this project to speed up their own development in Unity using visage|SDK or use this application as a starting point for own projects.


The application needs to be built with license key in order to work properly. More information on how to obtain license key and include it in the application can be found in Licensing section of the documentation.

Building and running the project


The application needs to be built from Unity project which is located in Samples/iOS/VisageTrackerUnityDemo. Detailed instructions for building and running the project is available in section Generating the Xcode project from Unity. If you wish to integrate visage|SDK in your own Unity project, instructions can be found here.

Prerequisites to running the sample project are:


NOTE: The Visage Tracker Unity plugin for iOS will not work in Unity editor, it will only work on actual device.

Generating the Xcode project from Unity

To generate the Xcode project for this sample follow these steps:

  1. Open VisageTrackerUnityDemo project (Samples/iOS/VisageTrackerUnityDemo) in Unity.

  2. NOTE: Depending on the Unity version that is used, multiple error messages related to project dependencies might appear in the console. Unity offers several possible solutions for this issue and more information can be found here.

  3. Add license key to the application (more information can be found in Licensing section of the documentation)

  4. Select and open the Main scene, located in Project's Assets.
  5. Please make sure the Analyser.cs script is attached to the Anaylser object in the Main scene.

  6. In Unity, select File->Build settings... A new window will appear. Switch the platform to iOS and add the current scene.

  7. Open the player settings by pressing Player Settings... button.

  8. In PlayerSettings please set the Company name to "visagetechnologies", Product name to "visagetrackerunitydemo" and Bundle Identifier to "com.visagetechnologies.visagetrackerunitydemo".

  9. The Plugins/iOS folder contains two frameworks - TFPlugin.framework (for building for device) and TFPlugin-simulator.framework (for simulator). Click on the appropriate framework and check "Add to Embedded Binaries" in the Inspector window on the right-hand side.

  10. In the Plugins/iOS folder, click on libVisageTrackerUnityPlugin. In the Inspector window on the right-hand side, under "Rarely used frameworks" find "Accelerate" and check it.

  11. Press the Build button to generate the project, a new dialog window will appear. In this window, locate Samples/iOS/VisageTrackerUnityDemo. Within it, create a new folder called 'Build' and set it as the build destination. This generates Xcode project in that folder.

  12. NOTE: If Unity reports error during build, please double check which platform is selected in the Build settings.

  13. All the necessary data will be copied from Unity's StreamingAssets folder to the \Data\Raw\ folder of the Xcode project.
    Library libVisageTrackerUnityPlugin.a will however be copied from Unity's Plugins folder to \Libraries\Plugins\ folder of the Xcode project.

  14. Please double check that targeted iOS device is selected as active scheme. Go to Build Settings and set iOS Deployment Target to 11.0 and Enable Bitcode to NO.
    Build the Xcode project.

  15. NOTE: Sometimes Xcode returns link error that it can't locate libiPhone-lib.a. In such cases click on Project->Build Phases-> Link Binary With Libraries, remove the libiPhone-lib.a and add it again. To add a new library click + option, a new window will appear, select Add Other...->Libraries->libiPhone-lib.a.

  16. License key file needs to be copied to Assets/StreamingAssets/Visage Tracker folder in Unity project. After that, licenseString variable should be set to the name of your license key file (XXX-XXX-XXX-XXX-XXX-XXX-XXX-XXX-XXX-XXX-XXX.vlc) in script LicenseString.cs. File will be automatically copied to newly generated Xcode project. For more details please read the section Including the License Key File in your application in the Registration and Licensing page.

  17. NOTE: In case you receive error message that App Transport Security has blocked a cleartext HTTP (http://) resource load in newly generated Xcode project from Unity, please set the NSAllowsArbitraryLoads key to YES under NSAppTransportSecurity dictionary in your .plist file

NOTE: Once you link with iOS 10 you must declare access to any user private data types (camera, photos, microphone, etc.). You do this by adding a usage key to your app’s Info.plist together with a purpose string.

Using the sample app

When the application starts and if a face is found in the camera frame, tracking from camera starts immediately. Pause/Resume button is placed in the bottom left corner and Switch camera button in the bottom right corner of the screen. On top of the video frames is a 3D model of glasses that is transformed with the information about rotation and translation from the tracker. It blends seamlessly with the background video.

In the top left corner is a button which enables face recognition feature. After a certain face appears in the frame for the first time, there is an initialization phase where a predefined number of frames is used to collect data for face recognition. After that, the face gets a label in the format Person[N] (Person0, Person1...,Person[N]). All identities collected during the run of the application will additionally be displayed in a list on the left of the screen.

NOTE: Without a valid license an error message will appear and face tracking will not work.

Implementation overview

The project consists of a main scene with the Unity’s GameObjects that provide different functionalities. A GameObject is the main building block of a scene in Unity. It consists of different Components and can parent other GameObjects. Manipulation of GameObjects is done with Scripts. It is recommended for developers to get familiar with Unity basics before continuing. The integration with visage|SDK is done through a plugin (VisageTrackerUnityPlugin) that wraps native code calls to visage|SDK and provides functions that can be called from Unity scripts.

Configuration selection

Tracker.cs contains the code for device-specific selection of tracking configuration, ensuring optimal tracking performance on different devices by taking into account their performance, camera and screen characteristics. Please read more details about configuration selection in the section Device-specific configuration selection.

Tracker GameObject

The core of the example is the Tracker GameObject. It consists of different components attached to it. The main component of the Tracker GameObject is the Tracker script component that handles the communication with the tracker (starts it and gets results from it). The behavior of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Tracker script properties of interest are as follows:

NOTE: Because of the different coordinate systems used by the visage|SDK face tracker and the Unity game engine mirroring around x-axis is applied to the relevant GameObjects.

Object in the scene that is also in the Controllable Objects list is the GlassesModel GameObject.

Analyser GameObject

Analyser GameObject is used to obtain and display face analysis results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Analyser GameObject is the Analyser script component that handles the communication with the face analyser (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Analyser script properties of interest are as follows:

Recognition GameObject

Recognition GameObject is used to obtain and display face recognition results. It relies on the functionality of the Tracker object, i.e. Tracker script component. The GameObject consists of different components attached to it. The main component of the Recognition GameObject is the Recognition script component that handles the communication with the face recognition (initializes it and gets results from it). The behaviour of the script can be modified by changing its properties and the script code. Other properties can be added as well. The existing Recognition script properties of interest are as follows:

Cameras

There are two cameras in the scene, one for 3D scene (Main Camera) and other for video display (Video Camera). The main camera "Field of View" is automatically set according to the Tracker focus (camera_focus parameter in configuration file) and input image aspect by the Tracker script. The Video Camera renders objects in the Video layer by using orthographic projection.

Video object

Video GameObject contains Video Plane that is used for displaying video and is automatically scaled to input frame aspect ratio by the VideoPlaneController script attached to it.

Glasses object

Since the Glasses object is in the Tracker script Controllable Objects list, rotation and translation information from the tracker is applied to it. This enables the overlaying of virtual objects to the tracked face that transform correctly with it. Other custom objects can be used instead of the glasses object. For a more comprehensive guide on how to achieve occlusion or on how to properly introduce other objects to the scene see AR modeling guide.

VisageTrackerUnityPlugin

The integration of visage|SDK face tracking with Unity is done through a plugin that wraps native code calls and provides functions that can be called from Unity scripts. The provided prebuilt plugin VisageTrackerUnityPlugin.a can be used as-is. The project and the source code for the VisageTrackerUnityPlugin plugin are also provided and can be used to implement different functionality if needed. For more information about the plugin including it's source code, see the VisageTrackerUnityPlugin project and its documentation.