visageSDK

Public Member Functions | List of all members
VisageSDK::VisageTracker Class Reference

VisageTracker is a face tracker capable of tracking the head pose, facial features and gaze for multiple faces in video coming from a video file, camera or other sources. More...

`#include <VisageTracker.h>`

Inheritance diagram for VisageSDK::VisageTracker:

## Public Member Functions
| VisageTracker (const char *trackerConfigFile)
| Constructor. More...
virtual | ~VisageTracker ()
| Destructor.
virtual int * | track (int frameWidth, int frameHeight, const char *p_imageData, FaceData *facedata, int format=VISAGE_FRAMEGRABBER_FMT_RGB, int origin=VISAGE_FRAMEGRABBER_ORIGIN_TL, int widthStep=0, long timeStamp=-1, int maxFaces=1)
| Performs face tracking (one or more faces) in the given image and returns tracking results and status. More...
void | setTrackerConfiguration (const char *trackerConfigFile, bool au_fitting_disabled=false, bool mesh_fitting_disabled=false)
| Sets tracking configuration. More...
void | setIPD (float value)
| Sets the inter pupillary distance. More...
float | getIPD ()
| Returns the current inter pupillary distance (IPD) setting. More...
void | stop ()
void | reset ()
| Reset tracking. More...
const char * | getSDKVersion ()
| Returns visageSDK version and revision number.
VisageConfiguration | getTrackerConfiguration ()
| Returns tracking configuration. More...
void | setTrackerConfiguration (VisageConfiguration &configuration)
| Sets tracking configuration. More...

## Detailed Description

VisageTracker is a face tracker capable of tracking the head pose, facial features and gaze for multiple faces in video coming from a video file, camera or other sources.

Frames (images) need to be passed sequentially to the VisageTracker::track() method, which immediately returns results for the given frame.

The tracker offers the following outputs as a part of FaceData object:

- 3D head pose,
- facial expression,
- gaze information,
- eye closure,
- iris radius,
- facial feature points,
- full 3D face model, textured.

The tracker requires a set of data and configuration files, available in Samples/data folder. For the list of required data files please consult VisageTracker() constructor description or simply copy the complete contents of this folder into your application's working folder.

### Configuring VisageTracker

The tracker is fully configurable through comprehensive tracker configuration files and VisageConfiguration class allowing to customize the tracker in terms of performance, quality and other options. The configuration files are intended to be used for tracker initialization while the VisageConfiguration class allows the specific configuration parameter to change in runtime.

visage|SDK contains optimal configurations for common uses such as head tracking, facial features tracking and ears tracking.

The VisageTracker Configuration Manual (later in text referred to as VTCM) provides the list of available configurations and full detail on all available configuration options.

Specific configuration parameters are used to enable features such as:

- ear tracking,
- pupil refinement,
- landmarks refinement,
  - smoothing filter

#### Ear tracking

Ear tracking includes tracking of additional 24 points (12 points per ear). Detailed illustration of the points' location can be found in the description of the FaceData::featurePoints2D member. Ears' feature points are part of the group 10 (10.1 - 10.24). Tracking the ears' points require the 3D model with defined ears vertices, as well as corresponding points mapping file that includes definition for group 10. visage|SDK containes examples of such model files within Samples/data/vft/fm folder: jk_300_wEars.wfm and jk_300_wEars.fdp.

For the list of model's vertices and triangles see chapter 2.3.2.1 The jk_300_wEars of VTCM.

A set of three configuration parameters is used to configure ear tracking:

- refine_ears,
- mesh_fitting_model and mesh_fitting_fdp if fine 3D mesh is enabled, otherwise pose_fitting_model and pose_fitting_fdp
- smoothing_factors 'ears' group (smoothing_factors[7])

visage|SDK provides optimal configuration for ear tracking. Please consult Facial Features Tracker - With Ears.cfg configuration file, located in Samples/data, for more details on the set configuration values.

#### Pupil refinement

Pupil refinement improves the precision of pupil center-point detection and provides iris radius information.

See process_eyes in chapter 2.1. Configuration parameters of VTCM.

#### Landmarks refinement

Landmarks refinement improves tracking accuracy and robustness and minimizes tracking jitter at the cost of reduced tracking speed (performance).

See refine_landmarks in chapter 2.1. Configuration parameters of VTCM.

#### Smoothing filter

The tracker can apply a smoothing filter to tracking results to reduce the inevitable tracking noise. Smoothing factors are adjusted separately for different parts of the face. The smoothing settings in the supplied tracker configurations are adjusted conservatively to achieve optimal balance between smoothing and delay in tracking response for a general use case.

See smoothing_factors in chapter 2.1. Configuration parameters of VTCM.

### Tracking data files

Features are determined by their own set of data files contained within Samples/data folder. In addition to the set values ​​of the configuration parameters, it is also necessary to ensure that the data of each feature is available. Default configuration files set the path to 3D models and data files within Samples/data folder as follows:

- vft folder that contains tracking algorithm and features specific files. NOTE: The subfolder structure and names must be retained.

TRACKING ALGORITHM DATA FILES
ff/ ff.vino.* fa/ d1qy.vino.* aux_file.bin fm/ *.fdp *.wfm

The list of tracking features available to use:

TRACKER FEATURES | REQUIRED DATA FILES
Ear tracking | er/ efa.lbf efc.lbf
Pupil refinement | pr/ pr.vino.*
Landmarks refinement | fa/ d2.vino.*

NOTE: .fdp and .wfm files correspond to the 3D model (configuration parameters *_fitting_model, *_fitting_fdp).

For further details on 3D models used in tracking and configuration parameters see VisageTracker Configuration Manual.

Definition at line 260 of file VisageTracker.h.

## Constructor & Destructor Documentation

## ◆ VisageTracker()

VisageSDK::VisageTracker::VisageTracker | ( | const char * | trackerConfigFile | ) |

Constructor.

Initializes tracker with specified configuration file.

Configuration file is used to modify tracker behavior and set paths to the required data files. Default configuration files are provided in the Samples/data folder.

Parameters

  trackerConfigFile | the name of the tracker configuration file (.cfg), relative to the current working directory, e.g. "Samples/data/Facial Features Tracker.cfg", considering visageSDK as a working directory.

## Member Function Documentation

## ◆ getIPD()

float VisageSDK::VisageTracker::getIPD | ( |  | ) |

Returns the current inter pupillary distance (IPD) setting.

IPD setting is used by the tracker to estimate the distance of the face from the camera. See setIPD() for further details.

Returns
  current setting of inter pupillary distance (IPD) in meters.

See also
  setIPD()

## ◆ getTrackerConfiguration()

VisageConfiguration VisageSDK::VisageTracker::getTrackerConfiguration | ( |  | ) |

Returns tracking configuration.

Returns
  VisageConfiguration object with the values currently used by tracker.

## ◆ reset()

void VisageSDK::VisageTracker::reset | ( |  | ) |

Reset tracking.

Resets the tracker. Tracker will reinitialise.

## ◆ setIPD()

void VisageSDK::VisageTracker::setIPD | ( | float | value | ) |

Sets the inter pupillary distance.

Inter pupillary distance (IPD) is used by the tracker to estimate the distance of the face from the camera. By default, IPD is set to 0.065 (65 millimeters) which is considered average. If the actual IPD of the tracked person is known, this function can be used to set this IPD. As a result, the calculated distance from the camera will be accurate (as long as the camera focal lenght is also set correctly). This is important for applications that require accurate distance. For example, in Augmented Reality applications objects such as virtual eyeglasses can be rendered at appropriate distance and will thus appear in the image with real-life scale.

IMPORTANT NOTE: In case of multitracking, same IPD will be set for all tracked faces!

Parameters

  value | the inter pupillary distance (IPD) in meters.

See also
  getIPD()

## ◆ setTrackerConfiguration() [1/2]

void VisageSDK::VisageTracker::setTrackerConfiguration | ( | const char * | trackerConfigFile,
|  | bool | au_fitting_disabled = `false`,
|  | bool | mesh_fitting_disabled = `false`
| ) |  |

Sets tracking configuration.

The tracker configuration file name and other configuration parameters are set and will be used for the next tracking session (i.e. when track() is called). Default configuration files (.cfg) are provided in Samples/data folder. Please refer to the VisageTracker Configuration Manual for further details on using the configuration files and all configurable options.

Parameters

  trackerConfigFile | the name of the tracker configuration file.
  au_fitting_disabled | disables the use of the 3D model used to estimate action units (au_fitting_model configuration parameter). If set to true, estimation of action units shall not be performed, and action units related data in the returned FaceData structure will not be available (FaceData::ActionUnits etc.). Disabling will result in a small performance gain.
  mesh_fitting_disabled | disables the use of the fine 3D mesh (mesh_fitting_model configuration parameter). If set to true, the 3D mesh shall not be fitted and the related information shall not be available in the returned FaceData structure (FaceData::FaceModelVertices etc.). Disabling will result in a small performance gain.

## ◆ setTrackerConfiguration() [2/2]

void VisageSDK::VisageTracker::setTrackerConfiguration | ( | VisageConfiguration & | configuration | ) |

Sets tracking configuration.

The tracker configuration object is set and will be used for the next tracking session (i.e. when track() is called).

Parameters

  configuration | object obtained by calling getTrackerConfiguration() function.

## ◆ stop()

void VisageSDK::VisageTracker::stop | ( |  | ) |

Deprecated:
  Stops the tracker.

## ◆ track()

virtual int* VisageSDK::VisageTracker::track | ( | int | frameWidth,
|  | int | frameHeight,
|  | const char * | p_imageData,
|  | FaceData * | facedata,
|  | int | format = `VISAGE_FRAMEGRABBER_FMT_RGB`,
|  | int | origin = `VISAGE_FRAMEGRABBER_ORIGIN_TL`,
|  | int | widthStep = `0`,
|  | long | timeStamp = `-1`,
|  | int | maxFaces = `1`
| ) |  |
| virtual

Performs face tracking (one or more faces) in the given image and returns tracking results and status.

This function should be called repeatedly on a series of images in order to perform continuous tracking.

If the tracker needs to be initialized, this will be done automatically before tracking is performed on the given image. Initialization means loading the tracker configuration file, required data files and allocating various data buffers to the given image size. This operation may take several seconds. This happens in the following cases:

- In the first frame (first call to track() function).
- When frameWidth or frameHeight are changed, i.e. when they are different from the ones used in the last call to track() function.
- If setTrackerConfiguration() function was called after the last call to track() function.
- When maxFaces is changed, i.e. when it its different from the one used in the last call to track() function.

The tracker results are returned in an array of FaceData objects, one FaceData object for each tracked face. Contents of each FaceData element depend on the corresponding tracker status (tracker statuses are returned as return value from the function).

The tracking of multiple faces is performed in parallel ( using OpenMP ) and performance (speed) may vary depending on the number of CPU cores, the number of faces in the current image and the value of maxFaces argument.

Parameters

  frameWidth | Width of the frame
  frameHeight | Height of the frame
  p_imageData | Pointer to image pixel data; size of the array must correspond to frameWidth and frameHeight
  facedata | Array of FaceData instances that will receive the tracking results. No tracking results will be returned if NULL pointer is passed. On first call of this function, the memory for the required member variables of the passed array of FaceData objects will be allocated and initialized automatically. The FacaData array must have the size equal to maxFaces argument.
  format | Format of input images passed in p_imageData. It can not change during tracking. Format can be one of the following: VISAGE_FRAMEGRABBER_FMT_RGB: each pixel of the image is represented by three bytes representing red, green and blue channels, respectively. VISAGE_FRAMEGRABBER_FMT_BGR: each pixel of the image is represented by three bytes representing blue, green and red channels, respectively. VISAGE_FRAMEGRABBER_FMT_RGBA: each pixel of the image is represented by four bytes representing red, green, blue and alpha (ignored) channels, respectively. VISAGE_FRAMEGRABBER_FMT_BGRA: each pixel of the image is represented by four bytes representing blue, green, red and alpha (ignored) channels, respectively. VISAGE_FRAMEGRABBER_FMT_LUMINANCE: each pixel of the image is represented by one byte representing the luminance (gray level) of the image.
  origin | No longer used, therefore, passed value will not have an effect on this function. However, the parameter is left to avoid API changes.
  widthStep | Width of the image data buffer, in bytes.
  timeStamp | The timestamp of the the input frame in milliseconds. The passed value will be returned with the tracking data for that frame (FaceData::timeStamp). Alternatively, the value of -1 can be passed, in which case the tracker will return time, in milliseconds, measured from the moment when tracking started.
  maxFaces | The maximum number of faces that will be tracked (tracker currently supports up to 20 faces).

Returns
  array of tracking statuses for each of the tracked faces (TRACK_STAT_OFF, TRACK_STAT_OK, TRACK_STAT_RECOVERING and TRACK_STAT_INIT, see FaceData for more details).

See also
  FaceData

The documentation for this class was generated from the following file:

- VisageTracker.h