visageSDK

Public Attributes | List of all members
VisageSDK::FaceData Struct Reference

Face data structure, used as container for all face tracking and detection results. More...

`#include <FaceData.h>`

## Public Attributes
float | trackingQuality
| Tracking quality level. More...
float | frameRate
| The frame rate of the tracker, in frames per second, measured over last 10 frames. More...
long | timeStamp
| Timestamp of the current video frame. More...
float | faceTranslation [3]
| Translation of the head from the camera. More...
float | faceRotation [3]
| Rotation of the head. More...
float | faceRotationApparent [3]
| Rotation of the head from the camera viewpoint This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face. More...
float | gazeDirection [2]
| Gaze direction. More...
float | gazeDirectionGlobal [3]
| Global gaze direction, taking into account both head pose and eye rotation. More...
float | eyeClosure [2]
| Discrete eye closure value. More...
float | irisRadius [2]
| Iris radius values in px. More...
int | shapeUnitCount
| Number of facial Shape Units. More...
float * | shapeUnits
| List of current values for facial Shape Units, one value for each shape unit. More...
int | actionUnitCount
| Number of facial Action Units. More...
int * | actionUnitsUsed
| Used facial Action Units. More...
float * | actionUnits
| List of current values for facial Action Units, one value for each Action Unit. More...
const char ** | actionUnitsNames
| List of facial Action Units names. More...
FDP * | featurePoints3D
| Facial feature points (global 3D coordinates). More...
FDP * | featurePoints3DRelative
| Facial feature points (3D coordinates relative to the face origin, placed at the center between eyes). More...
FDP * | featurePoints2D
| Facial feature points (2D coordinates). More...
int | faceModelVertexCount
| Number of vertices in the 3D face model. More...
float * | faceModelVertices
| List of vertex coordinates of the 3D face model. More...
float * | faceModelVerticesProjected
| List of projected (image space) vertex coordinates of the 3D face model. More...
int | faceModelTriangleCount
| Number of triangles in the 3D face model. More...
int * | faceModelTriangles
| Triangles list for the 3D face model. More...
float * | faceModelTextureCoords
| Texture coordinates for the 3D face model. More...
float * | faceModelTextureCoordsStatic
| Static texture coordinates of the mesh from the configuration file parameter mesh_fitting_model. More...
int | faceScale
| Scale of facial bounding box expressed in pixels. More...
VsRect | faceBoundingBox
| Face bounding box. More...
float | cameraFocus
| Focal distance of the camera, as configured in the tracker/detector configuration file. More...
ScreenSpaceGazeData | gazeData
| Structure holding screen space gaze position and quality for current frame. More...
float | gazeQuality
| The session level gaze tracking quality in online mode. More...

## Detailed Description

Face data structure, used as container for all face tracking and detection results.

This structure is passed as parameter to the VisageTracker::track() or VisageFeaturesDetector::detectFacialFeatures() method. Any of these methods copies latest tracking or detection results into it.

When filling the structure with data some members are filled while some are left undefined depending on tracking/detection status.

## Obtaining tracking data

The tracker returns these main classes of data:

- 3D head pose
- facial expression
- gaze direction
- eye closure
- facial feature points
- full 3D face model, textured
- screen space gaze position (if calibrated)

The tracker status is the return value of the VisageTracker::track() functions. The following table describes the possible states of the tracker, and lists active member variables (those that are filled with data) for each status when 3D face fitting is enabled.

TRACKER STATUS | DESCRIPTION | ACTIVE VARIABLES
TRACK_STAT_OFF | Tracker is not active, i.e. it has not yet been started, or it has been stopped. | N/A
TRACK_STAT_OK | Tracker is tracking normally. | frameRate, cameraFocus, faceScale, faceBoundingBox, faceTranslation, faceRotation, faceRotationApparent, actionUnitCount, actionUnitsUsed, actionUnits, actionUnitsNames, featurePoints3D, featurePoints3DRelative, featurePoints2D, faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles, faceModelTextureCoords
TRACK_STAT_RECOVERING | Tracker has lost the face and is attempting to recover and continue tracking. If it can not recover within the time defined by the parameter recovery_timeout in the tracker configuration file, the tracker will fully re-initialize (i.e. it will assume that a new user may be present). | frameRate, cameraFocus
TRACK_STAT_INIT | Tracker is initializing. The tracker enters this state immediately when it is started, or when it has lost the face and failed to recover (see TRACK_STAT_RECOVERING above). The initialization process is configurable through a number of parameters in the tracker configuration file. | frameRate, cameraFocus

The following table describes the possible states of the tracker, and lists active member variables (those that are filled with data) for each status when 3D face fitting is disabled.

TRACKER STATUS | DESCRIPTION | ACTIVE VARIABLES
TRACK_STAT_OFF | Tracker is not active, i.e. it has not yet been started, or it has been stopped. | N/A
TRACK_STAT_OK | Tracker is tracking normally. | frameRate, faceScale, faceBoundingBox, featurePoints2D
TRACK_STAT_RECOVERING | Tracker has lost the face and is attempting to recover and continue tracking. If it can not recover within the time defined by the parameter recovery_timeout in the tracker configuration file, the tracker will fully re-initialize (i.e. it will assume that a new user may be present). | frameRate
TRACK_STAT_INIT | Tracker is initializing. The tracker enters this state immediately when it is started, or when it has lost the face and failed to recover (see TRACK_STAT_RECOVERING above). The initialization process is configurable through a number of parameters in the tracker configuration file. | frameRate

### Smoothing

The tracker can apply a smoothing filter to tracking results to reduce the inevitable tracking noise. Smoothing factors are adjusted separately for different parts of the face. The smoothing settings in the supplied tracker configurations are adjusted conservatively to avoid delay in tracking response, yet provide reasonable smoothing. For further details please see the smoothing_factors parameter array in the VisageTracker Configuration Manual.

## Obtaining detection data

The detector returns these main classes of data for each detected face:

- 3D head pose
- gaze direction
- eye closure
- facial feature points
- full 3D face model, textured.

Detection result is returned from VisageFeaturesDetector::detectFacialFeatures() function. The following table describes possible output from the detector and the list of active variables (those that are filled with data). All other variables are left undefined.

DETECTION RESULT | DESCRIPTION | ACTIVE VARIABLES
0 | Detector did not find any faces in the image | N/A
N > 0 | Detector detected N faces in the image. | For first N FaceData objects in the array: cameraFocus, faceTranslation, faceRotation, faceRotationApparent, featurePoints3D, featurePoints3DRelative, featurePoints2D, faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles, faceModelTextureCoords For other FaceData objects in the array: N/A

## Returned data

The following sections give an overview of main classes of data that may be returned in FaceData by the tracker or the detector, and pointers to specific data members.

### 3D head pose

The 3D head pose consists of head translation and rotation. Absolute pose of the head with respect to the camera is available.

The following member variables return the head pose:

- faceTranslation
- faceRotation
- faceRotationApparent

Both face tracker and face detector return the 3D head pose.

### Facial expression

Facial expression is available in a form of Action Units (AUs), which describe the basic motions of the face, such as mouth opening, eyebrow raising etc.

AUs are fully configurable in the tracker configuration files. For the full list of default AUs, and for other information about the use and configuration of AUs, please refer to the Tracker Configuration Manual.

Please note that the AUs are similar, but not identical to the FACS Action Units.

The following member variables return Action Units data:

- actionUnitCount
- actionUnitsUsed
- actionUnits
- actionUnitsNames

Only face tracker returns the facial expression; face detector leaves these variables undefined. Furthermore, it is returned only if the au_fitting_model parameter is defined in the configuration file. An example of such file is Facial Features Tracker.cfg. Please see the VisageTracker Configuration Manual for further details.

### Gaze direction and eye closure

Gaze direction is available in local coordinate system of the person's face or global coordinate system of the camera. Eye closure is available as binary information (OPEN/CLOSED).

The following member variables return gaze direction and eye closure:

- gazeDirection
- gazeDirectionGlobal
- eyeClosure

Both face tracker and face detector return gaze direction and eye closure.

### Facial feature points

2D or 3D coordinates of facial feature points, including the ones defined by the MPEG-4 FBA standard and some additional points are available.

3D coordinates are available in global coordinate system or relative to the origin of the face (i.e. the point in the center between the eyes in the input image).

Facial features are available through the following member variables:

- featurePoints3D
- featurePoints3DRelative
- featurePoints2D

Both face tracker and face detector return facial feature points.

### 3D face model

The 3D face model is fitted in 3D to the face in the current image/video frame. The model is a single textured 3D triangle mesh. The texture of the model is the current image/video frame. This means that, when the model is drawn using the correct perspective it exactly recreates the facial part of the image. The correct perspective is defined by camera focal length (cameraFocus), width and height of the input image or the video frame, model rotation (faceRotation) and translation (faceTranslation).

The 3D face model is fully configurable and can even be replaced by a custom model; it can also be disabled for performance reasons if not required. Please see the VisageTracker Configuration Manual for further details. The default model is illustrated in the following image:

3D face model

There are multiple potential uses for the face model. Some ideas include, but are not limited to:

- Draw textured model to achieve face paint or mask effect.
- Draw the 3D face model into the Z buffer to achieve correct occlusion of virtual objects by the head in AR applications.
- Use texture coordinates to cut out the face from the image.
- Draw the 3D face model from a different perspective than the one in the actual video.
- Insert the 3D face model into another video or 3D scene.

Note that the vertices of the face model may not always exactly correspond to the facial feature points obtained from tracking/detection (featurePoints3D). For applications where the precise positioning of the facial feature points is recommended (e.g. virtual make-up), it is important to use the featurePoints3D and not the face model.

The 3D face model is contained in the following members:

- faceModelVertexCount
- faceModelVertices
- faceModelTriangleCount
- faceModelTriangles
- faceModelTextureCoords

Both face tracker and face detector return the 3D face model, if the mesh_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details).

### Screen space gaze position

Screen space gaze position is available if the tracker was provided with calibration repository and screen space gaze estimator is working in real time mode. Otherwise tracker returns default screen space gaze data. Default gaze position is center of screen. Default estimator state is off (ScreenSpaceGazeData::inState == 0). Please refer to VisageGazeTracker documentation for instructions on usage of screen space gaze estimator.

Screen space gaze position is contained in member gazeData. The session level gaze tracking quality is contained in member gazeQuality.

Only face tracker returns screen space gaze position.

Definition at line 259 of file FaceData.h.

## Member Data Documentation

## ◆ actionUnitCount

int VisageSDK::FaceData::actionUnitCount

Number of facial Action Units.

This variable is set only while tracker is tracking (TRACK_STAT_OK) and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details). Face detector leaves this variable undefined.

Number of action units that are defined for current face model.

See also
  actionUnits, actionUnitsUsed, actionUnitsNames

Definition at line 485 of file FaceData.h.

## ◆ actionUnits

float* VisageSDK::FaceData::actionUnits

List of current values for facial Action Units, one value for each Action Unit.

Action Units are the basic motions of the face, such as mouth opening, eyebrow raising etc. This variable is set only while tracker is tracking (TRACK_STAT_OK) and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details). Face detector leaves this variable undefined.

The Action Units used by the tracker are defined in the 3D face model file specified by au_fitting_model parameter in the configuration file (see the VisageTracker Configuration Manual for details). Furthermore, the tracker configuration file defines the names of Action Units and these names can be accessed through actionUnitsNames. Please refer to section or VisageTracker Configuration Manual for full list of Action Units.

See also
  actionUnitsUsed, actionUnitCount, actionUnitsNames

Definition at line 510 of file FaceData.h.

## ◆ actionUnitsNames

const char** VisageSDK::FaceData::actionUnitsNames

List of facial Action Units names.

This variable is set only while tracker is tracking (TRACK_STAT_OK) and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details). Face detector leaves this variable undefined.

See also
  actionUnitsUsed, actionUnitCount, actionUnits

Definition at line 520 of file FaceData.h.

## ◆ actionUnitsUsed

int* VisageSDK::FaceData::actionUnitsUsed

Used facial Action Units.

This variable is set only while tracker is tracking (TRACK_STAT_OK) and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details). Face detector leaves this variable undefined.

List of values, one for each action unit, to determine if specific action unit is actually used in the current tracker configuration. Values are as follows: 1 if action unit is used, 0 if action unit is not used.

See also
  actionUnits, actionUnitCount, actionUnitsNames

Definition at line 496 of file FaceData.h.

## ◆ cameraFocus

float VisageSDK::FaceData::cameraFocus

Focal distance of the camera, as configured in the tracker/detector configuration file.

This variable is set while tracker is running (any status other than TRACK_STAT_OFF), or if the detector has detected a face.

Focal length of a pinhole camera model used as approximation for the camera used to capture the video in which tracking is performed. The value is defined as distance from the camera (pinhole) to an imaginary projection plane where the smaller dimension of the projection plane is defined as 2, and the other dimension is defined by the input image aspect ratio. Thus, for example, for a landscape input image with aspect ratio of 1.33 the imaginary projection plane has height 2 and width 2.66.

This value is used for 3D scene set-up and accurate interpretation of tracking data.

Corresponding FoV (field of view) can be calculated as follows:

fov = 2 * atan( size / (2*cameraFocus) ), where size is 2 if width is larger than height and 2*height/width otherwise.

This member corresponds to the camera_focus parameter in the tracker/detector configuration file.

Definition at line 763 of file FaceData.h.

## ◆ eyeClosure

float VisageSDK::FaceData::eyeClosure[2]

Discrete eye closure value.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

Index 0 represents closure of left eye. Index 1 represents closure of right eye. Value of 1 represents open eye. Value of 0 represents closed eye.

Definition at line 442 of file FaceData.h.

## ◆ faceBoundingBox

VsRect VisageSDK::FaceData::faceBoundingBox

Face bounding box.

The bounding box is rectangular determined by the x and y coordinates of the upper-left corner and the width and height of the rectangle. Values are expressed in pixels.

Definition at line 745 of file FaceData.h.

## ◆ faceModelTextureCoords

float* VisageSDK::FaceData::faceModelTextureCoords

Texture coordinates for the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

A pair of u, v coordinates for each vertex. When FaceData is obtained from the tracker, the texture image is the current video frame. When FaceData is obtained from detector, the texture image is the input image of the detector.

See also
  faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles

Definition at line 726 of file FaceData.h.

## ◆ faceModelTextureCoordsStatic

float* VisageSDK::FaceData::faceModelTextureCoordsStatic

Static texture coordinates of the mesh from the configuration file parameter mesh_fitting_model.

They can be used to apply textures based on the texture template for the unwrapped mesh to the face model. The texture template for these coordinates is provided in jk_300_textureTemplate.png This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

A pair of u, v coordinates for each vertex.

See also
  faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles

Definition at line 736 of file FaceData.h.

## ◆ faceModelTriangleCount

int VisageSDK::FaceData::faceModelTriangleCount

Number of triangles in the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

See also
  faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangles, faceModelTextureCoords

Definition at line 707 of file FaceData.h.

## ◆ faceModelTriangles

int* VisageSDK::FaceData::faceModelTriangles

Triangles list for the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK), or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

Each triangle is described by three indices into the list of vertices faceModelVertices (counter-clockwise convention is used for normals direction).

See also
  faceModelVertexCount, faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTextureCoords

Definition at line 716 of file FaceData.h.

## ◆ faceModelVertexCount

int VisageSDK::FaceData::faceModelVertexCount

Number of vertices in the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

See also
  faceModelVertices, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles, faceModelTextureCoords

Definition at line 668 of file FaceData.h.

## ◆ faceModelVertices

float* VisageSDK::FaceData::faceModelVertices

List of vertex coordinates of the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

The format of the list is x, y, z coordinate for each vertex.

The coordinates are in the local coordinate system of the face, with the origin (0,0,0) placed at the center between the eyes. The x-axis points laterally towards the side of the face, y-axis points up and z-axis points into the face - see illustration below.

Coordinate system

To transform the coordinates into the coordinate system of the camera, use faceTranslation and faceRotation.

If the value set for the camera focal length in the tracker/detector configuration file corresponds to the real camera used, the scale of the coordinates shall be in meters; otherwise the scale is not known.

See also
  faceModelVertexCount, faceModelVerticesProjected, faceModelTriangleCount, faceModelTriangles, faceModelTextureCoords

Definition at line 689 of file FaceData.h.

## ◆ faceModelVerticesProjected

float* VisageSDK::FaceData::faceModelVerticesProjected

List of projected (image space) vertex coordinates of the 3D face model.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face, providing that the mesh_fitting_model parameter is set in the configuration file (see VisageTracker Configuration Manual for details).

The format of the list is x, y coordinate for each vertex. The 2D coordinates are normalised to image size so that the lower left corner of the image has coordinates 0,0 and upper right corner 1,1.

See also
  faceModelVertexCount, faceModelVertices, faceModelTriangleCount, faceModelTriangles,faceModelTextureCoords

Definition at line 700 of file FaceData.h.

## ◆ faceRotation

float VisageSDK::FaceData::faceRotation[3]

Rotation of the head.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

This is the estimated rotation of the head, in radians. Rotation is expressed with three values determining the rotations around the three axes x, y and z, in radians. This means that the values represent the pitch, yaw and roll of the head, respectively. The zero rotation (values 0, 0, 0) corresponds to the face looking straight ahead along the camera axis. Positive values for pitch correspond to head turning down. Positive values for yaw correspond to head turning right in the input image. Positive values for roll correspond to head rolling to the left in the input image, see illustration below. The values are in radians.

Note: The order to properly apply these rotations is y-x-z.

Rotations

See also
  faceTranslation

Definition at line 369 of file FaceData.h.

## ◆ faceRotationApparent

float VisageSDK::FaceData::faceRotationApparent[3]

Rotation of the head from the camera viewpoint This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

This is the estimated apparent rotation of the head, in radians. Rotation is expressed with three values determining the rotations around the three axes x, y and z, in radians. This means that the values represent the pitch, yaw and roll of the head, respectively. The zero apparent rotation (values 0, 0, 0) corresponds to the face looking straight into the camera i.e. a frontal face. Positive values for pitch correspond to head turning down. Positive values for yaw correspond to head turning right in the input image. Positive values for roll correspond to head rolling to the left in the input image. The values are in radians.

Note: The order to properly apply these rotations is y-x-z.

Definition at line 386 of file FaceData.h.

## ◆ faceScale

int VisageSDK::FaceData::faceScale

Scale of facial bounding box expressed in pixels.

Definition at line 739 of file FaceData.h.

## ◆ faceTranslation

float VisageSDK::FaceData::faceTranslation[3]

Translation of the head from the camera.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

Translation is expressed with three coordinates x, y, z. The coordinate system is such that when looking towards the camera, the direction of x is to the left, y iz up, and z points towards the viewer - see illustration below. The global origin (0,0,0) is placed at the camera. The reference point on the head is in the center between the eyes.

Coordinate system

If the value set for the camera focal length in the tracker/detector configuration file corresponds to the real camera used, the returned coordinates shall be in meters; otherwise the scale of the translation values is not known, but the relative values are still correct (i.e. moving towards the camera results in smaller values of z coordinate).

Aligning 3D objects with the face

The translation, rotation and the camera focus value together form the 3D coordinate system of the head in its current position and they can be used to align 3D rendered objects with the head for AR or similar applications. This example code shows how to do this using OpenGL.

The relative facial feature coordinates (featurePoints3DRelative) can then be used to align rendered 3D objects to the specific features of the face, like putting virtual eyeglasses on the eyes. Samples projects demonstrate how to do this, including full source code.

See also
  faceRotation

Definition at line 320 of file FaceData.h.

## ◆ featurePoints2D

FDP* VisageSDK::FaceData::featurePoints2D

Facial feature points (2D coordinates).

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

The 2D feature point coordinates are normalised to image size so that the lower left corner of the image has coordinates 0,0 and upper right corner 1,1.

The feature points are identified according to the MPEG-4 standard (with extension for additional points), so each feature point is identified by its group and index. For example, the tip of the chin belongs to group 2 and its index is 1, so this point is identified as point 2.1. The identification of all feature points is illustrated in the following image:

Visible/Physical contour

Certain feature points, like the ones on the tongue and teeth, can not be reliably detected so they are not returned and their coordinates are always set to zero. These points are: 6.1, 6.2, 6.3, 6.4, 9.8, 9.9, 9.10, 9.11, 11.4, 11.5, 11.6.

Several other points are estimated, rather than accurately detected, due to their specific locations. These points are: 2.10, 2.11, 2.12, 2.13, 2.14, 5.1, 5.2, 5.3, 5.4, 7.1, 9.1, 9.2, 9.6, 9.7, 9.12, 9.13, 9.14, 11.1, 11.2, 11.3, 12.1.

Ears' points - group 10 (points 10.1 - 10.24) can be either set to zero, accurately detected or estimated:

- zero:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 NOT provided
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 NOT provided
- detected:
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 provided
- estimated:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 provided

Face contour - group 13 and group 15. Face contour is available in two versions: the visible contour (points 13.1 - 13.17) and the physical contour (points 15.1 - 15.17). For more details regarding face contour please refer to the documentation of FDP class.

Nose contour - group 14, points: 14.21, 14.22, 14.23, 14.24, 14.25.

The resulting feature point coordinates are returned in form of an FDP object. This is a container class used for storage of MPEG-4 feature points. It provides functions to access each feature point by its group and index and to read its coordinates. Note that FDP stores 3D points and in the case of 2D feature points only the x and y coordinates of each point are used.

See also
  featurePoints3D, featurePoints3DRelative, FDP

Definition at line 660 of file FaceData.h.

## ◆ featurePoints3D

FDP* VisageSDK::FaceData::featurePoints3D

Facial feature points (global 3D coordinates).

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

The coordinate system is such that when looking towards the camera, the direction of x is to the left, y iz up, and z points towards the viewer. The global origin (0,0,0) is placed at the camera, see illustration.

Coordinate system

If the value set for the camera focal length in the tracker/detector configuration file corresponds to the real camera used, the returned coordinates shall be in meters; otherwise the scale is not known, but the relative values are still correct (i.e. moving towards the camera results in smaller values of z coordinate).

The feature points are identified according to the MPEG-4 standard (with extension for additional points), so each feature point is identified by its group and index. For example, the tip of the chin belongs to group 2 and its index is 1, so this point is identified as point 2.1. The identification of all feature points is illustrated in the following image:

Visible/Physical contour

Certain feature points, like the ones on the tongue and teeth, can not be reliably detected so they are not returned and their coordinates are always set to zero. These points are: 6.1, 6.2, 6.3, 6.4, 9.8, 9.9, 9.10, 9.11, 11.4, 11.5, 11.6.

Several other points are estimated, rather than accurately detected, due to their specific locations. These points are: 2.10, 2.11, 2.12, 2.13, 2.14, 5.1, 5.2, 5.3, 5.4, 7.1, 9.1, 9.2, 9.6, 9.7, 9.12, 9.13, 9.14, 11.1, 11.2, 11.3, 12.1.

Ears' points - group 10 (points 10.1 - 10.24) can be either set to zero, accurately detected or estimated:

- zero:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 NOT provided
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 NOT provided
- detected:
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 provided
- estimated:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 provided

Face contour - group 13 and group 15. Face contour is available in two versions: the visible contour (points 13.1 - 13.17) and the physical contour (points 15.1 - 15.17). For more details regarding face contour please refer to the documentation of FDP class.

Nose contour - group 14, points: 14.21, 14.22, 14.23, 14.24, 14.25.

The resulting feature point coordinates are returned in form of an FDP object. This is a container class used for storage of MPEG-4 feature points. It provides functions to access each feature point by its group and index and to read its coordinates.

See also
  featurePoints3DRelative, featurePoints2D, FDP

Definition at line 571 of file FaceData.h.

## ◆ featurePoints3DRelative

FDP* VisageSDK::FaceData::featurePoints3DRelative

Facial feature points (3D coordinates relative to the face origin, placed at the center between eyes).

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

The coordinates are in the local coordinate system of the face, with the origin (0,0,0) placed at the center between the eyes. The x-axis points laterally towards the side of the face, y-xis points up and z-axis points into the face - see illustration below.

Coordinate system

The feature points are identified according to the MPEG-4 standard (with extension for additional points), so each feature point is identified by its group and index. For example, the tip of the chin belongs to group 2 and its index is 1, so this point is identified as point 2.1. The identification of all feature points is illustrated in the following image:

Visible/Physical contour

Certain feature points, like the ones on the tongue and teeth, can not be reliably detected so they are not returned and their coordinates are always set to zero. These points are: 6.1, 6.2, 6.3, 6.4, 9.8, 9.9, 9.10, 9.11, 11.4, 11.5, 11.6.

Several other points are estimated, rather than accurately detected, due to their specific locations. These points are: 2.10, 2.11, 2.12, 2.13, 2.14, 5.1, 5.2, 5.3, 5.4, 7.1, 9.1, 9.2, 9.6, 9.7, 9.12, 9.13, 9.14, 11.1, 11.2, 11.3, 12.1.

Ears' points - group 10 (points 10.1 - 10.24) can be either set to zero, accurately detected or estimated:

- zero:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 NOT provided
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 NOT provided
- detected:
  - refine_ears configuration parameter turned on, 3D model with ears vertices and points mapping file for group 10 provided
- estimated:
  - refine_ears configuration parameter turned off, 3D model with ears vertices and points mapping file for group 10 provided

Face contour - group 13 and group 15. Face contour is available in two versions: the visible contour (points 13.1 - 13.17) and the physical contour (points 15.1 - 15.17). For more details regarding face contour please refer to the documentation of FDP class.

Nose contour - group 14, points: 14.21, 14.22, 14.23, 14.24, 14.25.

The resulting feature point coordinates are returned in form of an FDP object. This is a container class used for storage of MPEG-4 feature points. It provides functions to access each feature point by its group and index and to read its coordinates.

See also
  featurePoints3D featurePoints2D, FDP

Definition at line 618 of file FaceData.h.

## ◆ frameRate

float VisageSDK::FaceData::frameRate

The frame rate of the tracker, in frames per second, measured over last 10 frames.

This variable is set while tracker is running., i.e. while tracking status is not TRACK_STAT_OFF. Face detector leaves this variable undefined.

Definition at line 282 of file FaceData.h.

## ◆ gazeData

ScreenSpaceGazeData VisageSDK::FaceData::gazeData

Structure holding screen space gaze position and quality for current frame.

This variable is set only while tracker is tracking (TRACK_STAT_OK). Face detector leaves this variable undefined.

Position values are dependent on estimator state. Please refer to VisageGazeTracker and ScreenSpaceGazeData documentation for more details.

Definition at line 777 of file FaceData.h.

## ◆ gazeDirection

float VisageSDK::FaceData::gazeDirection[2]

Gaze direction.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

This is the current estimated gaze direction relative to the person's head. Direction is expressed with two values x and y, in radians. Values (0, 0) correspond to person looking straight. X is the horizontal rotation with positive values corresponding to person looking to his/her left. Y is the vertical rotation with positive values corresponding to person looking down.

See also
  gazeDirectionGlobal

Definition at line 398 of file FaceData.h.

## ◆ gazeDirectionGlobal

float VisageSDK::FaceData::gazeDirectionGlobal[3]

Global gaze direction, taking into account both head pose and eye rotation.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

This is the current estimated gaze direction relative to the camera axis. Direction is expressed with three values determining the rotations around the three axes x, y and z, i.e. pitch, yaw and roll. Values (0, 0, 0) correspond to the gaze direction parallel to the camera axis. Positive values for pitch correspond to gaze turning down. Positive values for yaw correspond to gaze turning right in the input image. Positive values for roll correspond to face rolling to the left in the input image, see illustration below.

The values are in radians.

Rotations

The global gaze direction can be combined with eye locations to determine the line(s) of sight in the real-world coordinate system with the origin at the camera. To get eye positions use featurePoints3D and FDP::getFP() function, e.g.:

FeaturePoint *left_eye_fp = const_cast<FeaturePoint*>( &featurePoints3D->getFP(3,5) );
FeaturePoint *right_eye_fp = const_cast<FeaturePoint*>( &featurePoints3D->getFP(3,6) );
float left_eye_pos[3], right_eye_pos[3];
left_eye_pos[0] = left_eye_fp->pos[0]; // x
left_eye_pos[1] = left_eye_fp->pos[1]; // y
left_eye_pos[2] = left_eye_fp->pos[2]; // z
right_eye_pos[0] = right_eye_fp->pos[0]; // x
right_eye_pos[1] = right_eye_fp->pos[1]; // y
right_eye_pos[2] = right_eye_fp->pos[2]; // z

See also
  gazeDirection, featurePoints3D

Definition at line 434 of file FaceData.h.

## ◆ gazeQuality

float VisageSDK::FaceData::gazeQuality

The session level gaze tracking quality in online mode.

Quality is returned as a value from 0 to 1, where 0 is the worst and 1 is the best quality. The quality is 0 also when the gaze tracking is off or calibrating.

Definition at line 783 of file FaceData.h.

## ◆ irisRadius

float VisageSDK::FaceData::irisRadius[2]

Iris radius values in px.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

The value with index 0 represents the iris radius of the left eye. The value with index 1 represents the iris radius of the right eye. If iris is not detected, the value is set to -1.

Definition at line 450 of file FaceData.h.

## ◆ shapeUnitCount

int VisageSDK::FaceData::shapeUnitCount

Number of facial Shape Units.

This variable is set while tracker is running., i.e. while tracking status is not TRACK_STAT_OFF or if the detector has detected a face and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details).

Number of shape units that are defined for current face model.

See also
  shapeUnits

Definition at line 460 of file FaceData.h.

## ◆ shapeUnits

float* VisageSDK::FaceData::shapeUnits

List of current values for facial Shape Units, one value for each shape unit.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face and only if au_fitting_model parameter in the configuration file is set (see VisageTracker Configuration Manual for details)..

Shape units can be described as static parameters of the face that are specific for each individual (e.g. shape of the nose).

The shape units used by the tracker and detector are defined in the 3D face model file, specified by the au_fitting_model in the configuration file (see the VisageTracker Configuration Manual for details).

See also
  shapeUnitCount

Definition at line 474 of file FaceData.h.

## ◆ timeStamp

long VisageSDK::FaceData::timeStamp

Timestamp of the current video frame.

This variable is set while tracker is running., i.e. while tracking status is not TRACK_STAT_OFF. Face detector leaves this variable undefined.

It returns the value passed to timeStamp argument in VisageTracker::track() method if it is different than -1, otherwise it returns time, in milliseconds, measured from the moment when tracking started.

Definition at line 291 of file FaceData.h.

## ◆ trackingQuality

float VisageSDK::FaceData::trackingQuality

Tracking quality level.

This variable is set only while tracker is tracking (TRACK_STAT_OK) or if the detector has detected a face.

Estimated tracking quality level for the current frame. The value is between 0 and 1.

Definition at line 275 of file FaceData.h.

The documentation for this struct was generated from the following file:

- FaceData.h