VisageTrackerUnityPlugin
docs_VisageTrackerUnityPlugin.cs
1 using System;
2 using System.Collections.Generic;
3 using System.Linq;
4 using System.Text;
5 
6 
7 /// <summary>
8 /// Provides high-level functions for using visage|SDK face tracking capabilities in Unity game engine.
9 ///Additionally, provides high-level functions for camera management such as _openCamera(), _closeCamera() and _getCameraInfo().
10 ///
11 ///It implements high-level wrapper functions around key functionalities from visage|SDK that enable usage of visage|SDK face tracking capabilities in Unity C# scripts.
12 ///
13 ///VisageTracker is a face tracker capable of tracking the head pose and facial features for multiple faces in video coming from a video file, camera or other sources.
14 ///Maximum number of faces that can be tracked is limited to 20 within native plugin library with MAX_FACES parameter.
15 ///Parametar faceIndex with value between 0 and MAX_FACES is passed to the functions within native plugin library to obtain the data for a particular tracked face.
16 ///All functions that return the data obtained from the tracker, internally check the tracker status as well as the value of the parameter faceIndex.
17 ///
18 ///The tracker is fully configurable through comprehensive tracker configuration files.visage|SDK contains optimal configurations for common uses such as head tracking and facial features tracking.Please refer to the VisageTracker Configuration Manual for the list of available configurations.
19 ///The VisageTracker Configuration Manual provides full detail on all available configuration options, allowing to customize the tracker in terms of performance, quality, and other options.
20 ///
21 ///High-level functions offer the following outputs:
22 /// - 3D head pose
23 /// - facial expression
24 /// - gaze direction information
25 /// - eye closure
26 /// - facial feature points
27 /// - full 3D face model, textured
28 ///
29 ///The tracker can apply a smoothing filter to tracking results to reduce the inevitable tracking noise. Smoothing factors are adjusted separately for different parts of the face.The smoothing settings in the supplied tracker configurations are adjusted conservatively to avoid delay in tracking response, yet provide reasonable smoothing. For further details please see the smoothing_factors parameter array in the VisageTracker Configuration Manual.
30 ///
31 ///Frames are fetched natively using VideoInput library by calling _grabFrame() function and are applied to the Unity-created texture using OpenGL in _setFrameData() function.
32 ///Function _grabFrame() needs to be called before each sequential call to the _track() function to obtain a new frame.
33 ///
34 ///Afterwords, high-level function such as _getFeaturePoints2D() are called to obtain tracker results for the given frame.
35 ///
36 ///The tracker requires a set of data and configuration files, available in Samples/data folder.
37 ///For most Unity applications usage of Application.streamingAssetsPath folder is encouraged.
38 ///
39 /// </summary>
40 ///
41 
42 /**
43  * \defgroup VisageTrackerUnityPlugin VisageTrackerUnityPlugin
44  * @{
45  */
46 
47 /** Initializes the license.
48 *
49 * It is recommended to initialize the license in the Awake () function within the unity script.
50 *
51 * @param license Relative path from the working directory to the directory that contains license file(s) (e.g. Application.streamingAssetsPath + "/" + licenseFileFolder).
52 */
53 void _initializeLicense(string license);
54 
55 /** Updates the pixel data in Unity with data from tracker.
56 *
57 * Before each call of the _setFrameData(), _grabFrame() function needs to be called to obtain a new frame.
58 
59 * Note that the VisageTrackerUnityPlugin requires the RBGA texture format and assumes that the width and heigh of the texture will be set to the nearest heigher power of 2 from the width and height of the image.
60 *
61 * \code
62 * if (texture == null)
63 * {
64 * // create texture with dimensions that are dependent on image dimensions as the nearest heigher power of number 2
65 * texture = new Texture2D(TexWidth, TexHeight, TextureFormat.RGBA32, false);
66 *
67 * // "pin" the pixel array in memory, so we can pass direct pointer to it's data to the plugin,
68 * // without costly marshaling of array of structures.
69 * texturePixels = ((Texture2D)texture).GetPixels32(0);
70 * texturePixelsHandle = GCHandle.Alloc(texturePixels, GCHandleType.Pinned);
71 * }
72 *
73 * // on every frame
74 * // send memory address of textures' pixel data to VisageTrackerUnityPlugin
75 * VisageTrackerNative._setFrameData(texturePixelsHandle.AddrOfPinnedObject());
76 * \endcode
77 *
78 * @param texIDPointer Pointer to the pixel data obtained from tracker to be used in Unity.
79  */
80 void _setFrameData(IntPtr texIDPointer);
81 
82 /** Grabs current frame from the camera.
83 *
84 * Frame from the camera is obtained using VideoInput library.
85 * This function needs to be called before each call of the _track() function.
86 */
87 void _grabFrame();
88 
89 /** Initializes camera using native VideoInput library with given orientation, camera width and height and deviceID.
90  * Device orientation parameter can have the following 4 values:
91  * - 0: Portrait
92  * - 1: LandscapeRight
93  * - 2: PortraitUpsideDown
94  * - 3: LandscapeLeft
95 *
96 * @param orientation Physical orientation of device.
97 * @param deviceID ID of the camera device from Windows system. For the first camera device value is 0, for the second 1 an so on.
98 * @param width Width of the frame.
99 * @param height Height of the frame.
100 */
101 void _openCamera(int orientation, int deviceID, int width, int height);
102 
103 /** Closes camera and clears memory allocated by the camera API.
104 */
105 void _closeCamera();
106 
107 /** Initialises the tracker with configuration file and a desired number of tracked faces. Number of tracked faces is limited within native plugin library with MAX_FACES parameter to 20.
108 *
109 * Note that in order for license to be registered, _initializeLicense() method has to be called before.
110 *
111 * @param config Relative path from the working directory to the directory that contains tracker configuration file, including configuration file name (e.g. Application.streamingAssetsPath + "/" + ConfigFileEditor).
112 * @param numFaces The maximum number of faces that will be tracked (<= 20).
113 */
114 void _initTracker(string config, int numFaces);
115 
116 /** Releases memory allocated by the tracker in the _initTracker() function.
117 *
118 * This function is called after the tracker ceases to be used or before the reinitialization of the tracker.
119 */
120 void _releaseTracker();
121 
122 /** Performs face tracking on current frame.
123 *
124 * Before each call of the _track() function, _grabFrame() function needs to be called to obtain the new frame.
125 */
126 void _track();
127 
128 /** Returns tracker status.
129 *
130 *
131 * An example of use:
132 * \code
133 *
134 * // initialization of TrackerStatus array
135 * public int[] TrackerStatus = new int[MAX_FACES];
136 *
137 * // call of the _getTrackerStatus() function
138 * VisageTrackerNative._getTrackerStatus(TrackerStatus);
139 *
140 * // an example of applying the obtained values
141 * if (TrackerStatus[0] != (int)TrackStatus.Off)
142 * {
143 * ....
144 * }
145 * \endcode
146 *
147 *
148 * @param tStatus Array that will be filled with tracking statuses for each of the tracked faces.
149 */
150 void _getTrackerStatus(int[] tStatus);
151 
152 /** Estimated tracking quality for the current frame for the given faceIndex.
153 *
154 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
155 * @return Estimated tracking quality level for the current frame. The value is between 0 and 1.
156 */
157 float _getTrackingQuality(int faceIndex);
158 
159 /** Returns the frame rate of the tracker, in frames per second, measured over last 10 frames.
160 *
161 */
162 float _getFrameRate();
163 
164 /** Returns timestamp of the current video frame, in milliseconds, measured from the moment when tracking started.
165 *
166 */
167 int _getTimeStamp();
168 
169 /** Returns size of face bounding box in pixels for the given faceIndex.
170 *
171 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
172 */
173 int _getFaceScale(int faceIndex);
174 
175 /** Returns the face bounding box for the given faceIndex.
176 
177 * Parameter <i>rectangle</i> is a VsRect structure whose members represent x and y coordinates of the upper-left corner and the width and height of the box.
178 * All members are expressed in pixels.
179 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will fill <i>rectangle</i> members with -1 values.
180 * @param rectangle Structure where face bounding box's coordinates and size will be stored.
181 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
182 */
183 void _getFaceBoundingBox(VsRect* rectangle, int faceIndex);
184 
185 /**
186 * Returns the current head translation in meters for the given faceIndex.
187 *
188 * Parameter <i>translation</i> is 3-dimensional array which will be filled x, y and z head translation coordinates
189 * for the particular face given with <i>faceIndex</i> parameter.
190 * The coordinate system is such that when looking towards the camera, the direction of x is to the left, y iz up, and z points towards the viewer.
191 * The global origin (0,0,0) is placed at the camera. The reference point on the head is in the center between the eyes.
192 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will fill translation array with -10000 values.
193 * @param translation 3D array in which the head translation values will be stored.
194 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
195 */
196 void _getHeadTranslation(float* translation, int faceIndex);
197 
198 /**
199 * Returns the current head rotation in radians for the given faceIndex.
200 *
201 * Parameter <i>rotation</i> is 3-dimensional array which will be filled x, y and z head rotation coordinates
202 * for the particular face given with <i>faceIndex</i> parameter.
203 *
204 * This is the estimated rotation of the head, in radians. Rotation is expressed with three values determining the rotations around the three axes x, y and z, in radians.
205 * This means that the values represent the pitch, yaw and roll of the head, respectively. The zero rotation (values 0, 0, 0) corresponds to the face looking straight ahead along the camera axis.
206 * Positive values for pitch correspond to head turning down. Positive values for yaw correspond to head turning right in the input image.
207 * Positive values for roll correspond to head rolling to the left in the input image. The values are in radians.
208 *
209 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will fill rotation array with 0 values.
210 * @param rotation 3D array in which the head rotation values will be stored.
211 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
212 */
213 void _getHeadRotation(float* rotation, int faceIndex);
214 
215 /**
216 * Returns the current apparent head rotation in radians for the given faceIndex.
217 *
218 * Parameter <i>rotation</i> is 3-dimensional array which will be filled x, y and z head rotation coordinates
219 * for the particular face given with <i>faceIndex</i> parameter.
220 *
221 * This is the estimated apparent rotation of the head, in radians. Rotation is expressed with three values determining the rotations around the three axes x, y and z, in radians.
222 * This means that the values represent the pitch, yaw and roll of the head, respectively. The zero rotation (values 0, 0, 0) corresponds to the face looking straight into the camera i.e. a frontal face.
223 * Positive values for pitch correspond to head turning down. Positive values for yaw correspond to head turning right in the input image.
224 * Positive values for roll correspond to head rolling to the left in the input image. The values are in radians.
225 *
226 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will fill rotation array with 0 values.
227 * @param rotationApparent 3D array in which the apparent head rotation values will be stored.
228 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
229 */
230 void _getHeadRotationApparent(float* rotationApparent, int faceIndex);
231 
232 /** Returns number of vertices in the 3D face model.
233 *
234 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will be set to 0.
235 * @return Number of vertices.
236 */
238 
239 /**
240 * Returns number of triangles in the 3D face model.
241 * This variable is set only while tracker is tracking (TRACK_STAT_OK), otherwise it will be set to 0.
242 * @return Number of triangles.
243 */
245 
246 /**
247 * Returns triangles coordinates of the 3D face model.
248 *
249 * Each triangle is described by three indices into the list of vertices.
250 *
251 * Function returns triangles coordinates in the triangleNumber*3-dimensional <i>triangles</i> array, where triangleNumber is value
252 * obtained from _getFaceModelTriangleCount() function, for the particular faceIndex.
253 * @param triangles Array in which values of triangle coordinates will be stored.
254 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
255 * <br/>
256 */
257 void _getFaceModelTriangles(int* triangles, int faceIndex);
258 
259 /**
260 * Returns vertex coordinates of the 3D face model.
261 *
262 * The coordinates are in the local coordinate system of the face, with the origin (0,0,0) placed at the center between the eyes.
263 * The x-axis points laterally towards the side of the face, y-axis points up and z-axis points into the face.
264 *
265 * Function returns vertices position in the vertexNumber*3-dimensional <i>vertices</i> array, where vertexNumber is value
266 * obtained from _getFaceModelVertexCount() function, for the particular faceIndex.
267 *
268 * An example of use:
269 * \code
270 *
271 * //define MaxVertices value that will be larger than the real number of vertices
272 * public const int MaxVertices = 1024;
273 * private float[] vertices = new float[MaxVertices * 3];
274 *
275 * //call of _getFaceModelVertices() function
276 * VisageTrackerNative._getFaceModelVertices(vertices, faceIndex);
277 *
278 * \endcode
279 * @param vertices Array in which the values of vertex coordinates will be stored.
280 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
281 */
282 void _getFaceModelVertices(float* vertices, int faceIndex);
283 
284 /**
285 * Returns projected (image space) vertex coordinates.
286 *
287 * The 2D coordinates are normalised to image size so that the lower left corner of the image has coordinates 0,0 and upper right corner 1,1.
288 *
289 * Function returns vertices position in the vertexNumber*2-dimensional <i>verticesProjected</i> array, where vertexNumber is value
290 * obtained from _getFaceModelVertexCount() function, for the particular faceIndex.
291 *
292 * An example of use:
293 * \code
294 *
295 * //define MaxVertices value that will be larger than the real number of vertices
296 * public const int MaxVertices = 1024;
297 * private float[] verticesProjected = new float[MaxVertices * 2];
298 *
299 * //call of _getFaceModelVerticesProjected() function
300 * VisageTrackerNative._getFaceModelVerticesProjected(verticesProjected, faceIndex);
301 *
302 * \endcode
303 *
304 * @param verticesProjected Array in which the values of projected vertex coordinates will be stored.
305 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
306 */
307 void _getFaceModelVerticesProjected(float* verticesProjected, int faceIndex);
308 
309 /**
310 * Returns texture coordinates.
311 * Texture coordinates for the 3D face model.
312 * A pair of u, v coordinates for each vertex.
313 *
314 * Function returns texture coordinates in the vertexNumber*2-dimensional <i>texCoord</i> array, where vertexNumber is value
315 * obtained from _getFaceModelVertices() function, for the particular faceIndex.
316 *
317 * An example of use:
318 * \code
319 *
320 * //define MaxVertices value that will be larger than the real number of vertices
321 * public const int MaxVertices = 1024;
322 * private float[] texCoords = new float[MaxVertices * 2];
323 *
324 * //call of _getFaceModelTextureCoords() function
325 * VisageTrackerNative._getFaceModelTextureCoords(texCoord, faceIndex);
326 *
327 * \endcode
328 * @param texCoord Array in which values of texture coordinates will be stored.
329 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
330 * <br/>
331 */
332 void _getFaceModelTextureCoords(float* texCoord, int faceIndex);
333 
334 /**
335 * Index of the first feature point group.
336 * @return 2
337 * <br/>
338 */
340 
341 
342 /**
343 * Index of the last feature point group.
344 * @return 15
345 * <br/>
346 */
348 
349 
350 /**
351 * Get the size of all feature point groups.
352 * @param groupSizeArray Array in which the size of each group will be stored.
353 * @param length Length of groupSizeArray array.
354 * <br/>
355 */
356 void _getGroupSizes(int* groupSizeArray, int length);
357 
358 /** Returns camera info.
359 *
360 * This function should be called after _openCamera() and _track() functions are preformed, otherwise it will join -1 value to parametar focus and 0 to parameters width and height.
361 *
362 * @param focus Parameter to which focal distance of the camera will be assigned.
363 * @param width Parameter to which width of the frame will be assigned.
364 * @param height Parameter to which height of the frame will be assigned.
365  */
366 void _getCameraInfo(out float focus, out int width, out int height);
367 
368 
369 /** Returns the action unit count.
370 *
371 * @return Action unit count
372  */
373 int _getActionUnitCount();
374 
375 /** Returns the name of the action unit with the specified index.
376 *
377 * @param auIndex Index of the action unit for the given faceIndex.
378 * @return The name of the action unit at the given index.
379 */
380 string[] _getActionUnitName(int auIndex);
381 
382 /** Returns true if the action unit is used.
383 *
384 * @param auIndex Index of the action unit.
385 * @return True if action unit is used, false otherwise.
386 */
387 bool _getActionUnitUsed(int auIndex);
388 
389 /** Returns all action unit values.
390 *
391 *
392 * An example of use:
393 * \code
394 *
395 * int AU_COUNT = VisageTrackerNative._getActionUnitCount();
396 *
397 * float[] AUvalues = new float[AU_COUNT];
398 *
399 * // au values for the first tracked face
400 * VisageTrackerNative._getActionUnitValues(AUvalues, 0);
401 *
402 * \endcode
403 *
404 *
405 * @param values Array that will be filled with action unit values.
406 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
407 */
408 void _getActionUnitValues(float[] values, int faceIndex);
409 
410 /** Returns the gaze direction for the given faceIndex.
411 *
412 * Direction is expressed with two values x and y, in radians. Values (0, 0) correspond to person looking straight. X is the horizontal rotation with positive values corresponding to person looking to his/her left. Y is the vertical rotation with positive values corresponding to person looking down.
413 *
414 * An example of use:
415 * \code
416 *
417 * float[] direction = new float[2];
418 *
419 * for (int faceIndex = 0; faceIndex < MAX_FACES; faceIndex++)
420 * {
421 * VisageTrackerNative._getGazeDirection(direction, faceIndex);
422 * }
423 * \endcode
424 *
425 * Note that gaze direction is influenced with process_eyes parameter from tracker configuration file.
426 *
427 * @param direction Array that will be filled with direction expressed with two values x and y.
428 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
429 * @return True on success, false on failure that will occur if faceIndex is greater than MAX_FACES or TrackerStatus for the given faceIndex is different than OK.
430 */
431 bool _getGazeDirection(float[] direction, int faceIndex);
432 
433 /** Returns the global gaze direction for the given faceIndex.
434 *
435 * Global gaze direction is the current estimated gaze direction relative to the camera axis. Direction is expressed with three values determining the rotations around the three axes x, y and z, i.e. pitch, yaw and roll.
436 *
437 * An example of use:
438 * \code
439 *
440 * float[] direction = new float[3];
441 *
442 * for (int faceIndex = 0; faceIndex < MAX_FACES; faceIndex++)
443 * {
444 * VisageTrackerNative._getGazeDirectionGlobal(direction, faceIndex);
445 * }
446 * \endcode
447 *
448 * Note that global gaze direction influenced with process_eyes parameter from tracker configuration file.
449 *
450 * @param direction Array that will be filled with direction expressed with three values determining the rotations around the three axes x, y and z.
451 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
452 * @return True on success, false on failure that will occur if faceIndex is greater than MAX_FACES or TrackerStatus for the given faceIndex is different than OK.
453 */
454 bool _getGazeDirectionGlobal(float[] direction, int faceIndex);
455 
456 /** Returns eye closure value for the given faceIndex.
457 *
458 * Discrete eye closure value is expressed with two values: index 0 represents closure of left eye. Index 1 represents closure of right eye. Value of 1 represents open eye. Value of 0 represents closed eye.
459 *
460 * An example of use:
461 * \code
462 *
463 * float[] closure = new float[2];
464 *
465 * // eye closure value for the first tracked face
466 * VisageTrackerNative._getEyeClosure(closure, 0);
467 *
468 * \endcode
469 *
470 *
471 * @param closure Array that will be filled with eye closure values: index 0 represents closure of left eye. Index 1 represents closure of right eye. Value of 1 represents open eye. Value of 0 represents closed eye.
472 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
473 * @return True on success, false on failure that will occur if faceIndex is greater than MAX_FACES or TrackerStatus for the given faceIndex is different than OK.
474 */
475 bool _getEyeClosure(float[] closure, int faceIndex);
476 
477 /** Returns iris radius values in px for the given faceIndex.
478 *
479 * This variable is set only while tracker is tracking (TRACK_STAT_OK).
480 *
481 * The value with index 0 represents the iris radius of the left eye. The value with index 1 represents the iris radius of the right eye. If iris is not detected, the value is set to -1.
482 *
483 * An example of use:
484 * \code
485 *
486 * float[] radius = new float[2];
487 *
488 * // iris radius values for the first tracked face
489 * VisageTrackerNative._getIrisRadius(radius, 0);
490 *
491 * \endcode
492 *
493 *
494 * @param radius Array that will be filled with iris radius values: index 0 represents iris radius of the left eye. Index 1 represents the iris radius of the right eye.
495 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
496 * @return True on success, false on failure that will occur if faceIndex is greater than MAX_FACES or TrackerStatus for the given faceIndex is different than OK.
497 */
498 bool _getIrisRadius(float[] radius, int faceIndex);
499 
500 /**
501 * Returns for all feature points the position in normalized 2D screen coordinates, indicator whether the point is defined, indicator whether the point is detected or estimated
502 * and feature point quality for the given faceIndex.
503 *
504 * The 2D feature point coordinates are normalised to image size so that the lower left corner of the image has coordinates 0,0 and upper right corner 1,1.
505 *
506 * For example, first feature point is 2.1, therefore first five values in featurePointArray array will be filled with the following values:
507 * \code
508 *
509 * featurePointArray = [x_(2.1), y_(2.1), defined_(2.1), detected_(2.1), quality_(2.1), ...]
510 *
511 * \endcode
512 * An example of use in Unity:
513 * \code
514 *
515 * //array which will be filled with values
516 * public float[] featurePointArray = new float[2000];
517 *
518 * //call of the _getAllFeaturePoints2D() function
519 * VisageTrackerNative._getAllFeaturePoints2D(featurePointArray, featurePointArray.Length, faceIndex);
520 *
521 * \endcode
522 *
523 * @param featurePointArray array in which the position in normalized 2D screen coordinates, indication whether the point is defined and detected and
524 * feature point quality will be stored.
525 * @param length length of featurePointArray.
526 * @param faceIndex value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
527 *
528 */
529 void _getAllFeaturePoints2D(float* featurePointArray, int length, int faceIndex);
530 
531 /**
532 * Returns for all feature points the global 3D position in meters, indicator whether the point is defined, indicator whether the point is detected or estimated
533 * and feature point quality for the given faceIndex.
534 *
535 * The coordinate system is such that when looking towards the camera, the direction of x is to the left, y iz up, and z points towards the viewer. The global origin (0,0,0) is placed at the camera.
536 *
537 * For example, first feature point is 2.1, therefore first six values in featurePointArray array will be filled with the following values:
538 * \code
539 *
540 * featurePointArray = [x_(2.1), y_(2.1), z_(2.1), defined_(2.1), detected_(2.1), quality_(2.1), ...]
541 *
542 * \endcode
543 * An example of use in Unity:
544 * \code
545 *
546 * //array which will be filled with values
547 * public float[] featurePointArray = new float[2000];
548 *
549 * //call of the _getAllFeaturePoints3D() function
550 * VisageTrackerNative._getAllFeaturePoints3D(featurePointArray, featurePointArray.Length, faceIndex);
551 *
552 * \endcode
553 *
554 * <b>NOTE:</b> quality information is returned exclusively within _getAllFeaturePoints2D() function. Therefore, this function will always return -1 for the quality of each point.
555 *
556 * @param featurePointArray array in which the position in x, y and z coordinates, indication whether the point is defined and detected and
557 * feature point quality will be stored.
558 * @param length length of featurePointArray.
559 * @param faceIndex value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
560 *
561 */
562 void _getAllFeaturePoints3D(float* featurePointArray, int length, int faceIndex);
563 
564 /**
565 * Returns for all feature points the position in 3D coordinates relative to the face origin, placed at the center between eyes in meters, indicator whether the point is defined, indicator whether the point is detected or estimated
566 * and feature point quality for the given faceIndex.
567 *
568 * Returned coordinates are in the local coordinate system of the face, with the origin (0,0,0) placed at the center between the eyes. The x-axis points laterally towards the side of the face, y-xis points up and z-axis points into the face.
569 *
570 * For example, first feature point is 2.1, therefore first six values in featurePointArray array will be filled with the following values:
571 * \code
572 *
573 * featurePointArray = [x_(2.1), y_(2.1), z_(2.1), defined_(2.1), detected_(2.1), quality_(2.1), ...]
574 *
575 * \endcode
576 *
577 * An example of use in Unity:
578 * \code
579 *
580 * //array which will be filled with values
581 * public float[] featurePointArray = new float[2000];
582 *
583 * //call of the _getAllFeaturePoints3DRelative() function
584 * VisageTrackerNative._getAllFeaturePoints3DRelative(featurePointArray, featurePointArray.Length, faceIndex);
585 *
586 * \endcode
587 *
588 * <b>NOTE:</b> quality information is returned exclusively within _getAllFeaturePoints2D() function. Therefore, this function will always return -1 for the quality of each point.
589 *
590 * @param featurePointArray array in which the position in x, y and z coordinates, indication whether the point is defined and detected and
591 * feature point quality will be stored.
592 * @param length length of featurePointArray.
593 * @param faceIndex value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
594 *
595 */
596 void _getAllFeaturePoints3DRelative(float* featurePointArray, int length, int faceIndex);
597 
598 /**
599 * Returns the feature points position in normalized 2D screen coordinates, indicator whether the point is defined, indicator whether the point is detected or estimated
600 * and feature point quality for the given faceIndex.
601 *
602 * The 2D feature point coordinates are normalised to image size so that the lower left corner of the image has coordinates 0,0 and upper right corner 1,1.
603 *
604 * Function returns position for N feature points in the N*2-dimensional <i>positions</i> array, an indication of whether the point is defined as 0 and 1 in the N-dimensional <i>defined</i> array and detected as 0 and 1 in the N-dimensional <i>detected</i> array.
605 * The quality of each point as value between 0 and 1 is returned in <i>quality</i> array.
606 * For each point, its group and index are specifically listed in the <i>groups</i> and <i>indices</i> arrays.
607 *
608 * An example of use:
609 * \code
610 * // an example for points 2.1, 8.3 and 8.4
611 * const int N;
612 * int[] groups = new int[N] {2, 8, 8};
613 * int[] indices = new int[N]{ 1, 3, 4 };
614 * float[] positions2D = new float[2*N];
615 * int[] defined = new int[N];
616 * int[] detected = new int[N];
617 * float[] quality = new float[N];
618 *
619 * for (int faceIndex = 0; faceIndex < MAX_FACES; faceIndex++)
620 * {
621 * VisageTrackerNative._getFeaturePoints2D(N, groups, indices, positions3D, defined, detected, quality, faceIndex);
622 * }
623 *
624 * \endcode
625 *
626 * @param N number of feature points.
627 * @param groups N-dimensional array of the groups of the feature points.
628 * @param indices N-dimensional array of the indices of the feature points within the groups.
629 * @param positions N*2-dimensional array filled with resulting facial feature points positions.
630 * @param defined N-dimensional array filled for every feature point with a value that indicates whether the point is defined. Value 1 is assigned to defined points, and 0 to undefined points.
631 * @param detected N-dimensional array filled for every feature point with a value that indicates whether the point is detected. Value 1 is assigned to detected points, and 0 to undetected points.
632 * @param quality N-dimensional array filled for every feature point with a value from 0 to 1, where 0 is the worst and 1 is the best quality.
633 * @param faceIndex value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
634 */
635 void _getFeaturePoints2D(int N, int* groups, int* indices, float* positions, int* defined, int* detected, float* quality, int faceIndex);
636 
637 /**
638 * Returns the global 3D feature point position in meters, indicator whether the point is defined, indicator whether the point is detected or estimated
639 * and feature point quality for the given faceIndex.
640 *
641 * The coordinate system is such that when looking towards the camera, the direction of x is to the left, y iz up, and z points towards the viewer. The global origin (0,0,0) is placed at the camera.
642 *
643 * Function returns position for N feature points in the N*3-dimensional <i>positions</i> array, an indication of whether the point is defined as 0 and 1 in the N-dimensional <i>defined</i> array and detected as 0 and 1 in the N-dimensional <i>detected</i> array.
644 * The quality of each point as value between 0 and 1 is returned in <i>quality</i> array.
645 * For each point, its group and index are specifically listed in the <i>groups</i> and <i>indices</i> arrays.
646 *
647 * An example of use:
648 * \code
649 * // an example for points 2.1, 8.3 and 8.4
650 * const int N;
651 * int[] groups = new int[N] {2, 8, 8};
652 * int[] indices = new int[N]{ 1, 3, 4 };
653 * float[] positions3D = new float[3*N];
654 * int[] defined = new int[N];
655 * int[] detected = new int[N];
656 * float[] quality = new float[N];
657 *
658 * for (int faceIndex = 0; faceIndex < MAX_FACES; faceIndex++)
659 * {
660 * VisageTrackerNative._getFeaturePoints3D(N, groups, indices, positions3D, defined, detected, quality, faceIndex);
661 * }
662 *
663 * \endcode
664 *
665 * <b>NOTE:</b> quality information is returned exclusively within _getFeaturePoints2D() function. Therefore, this function will always return -1 for the quality of each point.
666 *
667 * @param N number of feature points.
668 * @param groups N-dimensional array of the groups of the feature points.
669 * @param indices N-dimensional array of the indices of the feature points within the groups.
670 * @param positions N*3-dimensional array filled with resulting facial feature points positions.
671 * @param defined N-dimensional array filled for every feature point with a value that indicates whether the point is defined. Value 1 is assigned to defined points, and 0 to undefined points.
672 * @param detected N-dimensional array filled for every feature point with a value that indicates whether the point is detected. Value 1 is assigned to detected points, and 0 to undetected points.
673 * @param quality N-dimensional array filled for every feature point with a value from 0 to 1, where 0 is the worst and 1 is the best quality.
674 * @param faceIndex value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
675 */
676 void _getFeaturePoints3D(int N, int* groups, int* indices, float* positions, int* defined, int* detected, float* quality, int faceIndex);
677 
678 /**
679 * Returns the 3D coordinates relative to the face origin, placed at the center between eyes in meters, indicator whether the point is defined, indicator whether the point is detected or estimated
680 * and feature point quality for the given faceIndex.
681 *
682 * Returned coordinates are in the local coordinate system of the face, with the origin (0,0,0) placed at the center between the eyes. The x-axis points laterally towards the side of the face, y-xis points up and z-axis points into the face.
683 *
684 * Function returns position for N feature points in the N*3-dimensional <i>positions</i> array, an indication of whether the point is defined as 0 and 1 in the N-dimensional <i>defined</i> array and detected as 0 and 1 in the N-dimensional <i>detected</i> array.
685 * The quality of each point as value between 0 and 1 is returned in <i>quality</i> array.
686 * For each point, its group and index are specifically listed in the <i>groups</i> and <i>indices</i> arrays.
687 *
688 * An example of use:
689 * \code
690 *
691 * // an example for points 2.1, 8.3 and 8.4
692 * const int N;
693 * int[] groups = new int[N] {2, 8, 8};
694 * int[] indices = new int[N]{ 1, 3, 4 };
695 * float[] positions = new float[3*N];
696 * int[] defined = new int[N];
697 * int[] detected = new int[N];
698 * float[] quality = new float[N];
699 *
700 *
701 * for (int faceIndex = 0; faceIndex < MAX_FACES; faceIndex++)
702 * {
703 * VisageTrackerNative._getFeaturePoints3DRelative(N, groups, indices, positions, defined, detected, quality, faceIndex);
704 * }
705 *
706 * \endcode
707 *
708 * <b>NOTE:</b> quality information is returned exclusively within _getFeaturePoints2D() function. Therefore, this function will always return -1 for the quality of each point.
709 *
710 * @param N Number of feature points.
711 * @param groups N-dimensional array of the groups of the feature points.
712 * @param indices N-dimensional array of the indices of the feature points within the groups.
713 * @param positions N*3-dimensional array filled with resulting facial feature points positions.
714 * @param defined N-dimensional array filled for every feature point with a value that indicates whether the point is defined. Value 1 is assigned to defined points, and 0 to undefined points.
715 * @param detected N-dimensional array filled for every feature point with a value that indicates whether the point is detected. Value 1 is assigned to detected points, and 0 to undetected points.
716 * @param quality N-dimensional array filled for every feature point with a value from 0 to 1, where 0 is the worst and 1 is the best quality.
717 * @param faceIndex Value between 0 and MAX_FACES-1, used to access the data of a particular tracked face.
718 */
719 void _getFeaturePoints3DRelative(int N, int* groups, int* indices, float* positions, int* defined, int* detected, float* quality, int faceIndex);
720 
721 /** Sets tracking configuration.
722 *
723 * The tracker configuration file name and other configuration parameters are set and will be used for the next tracking session (i.e. when _track() is called). Default configuration files (.cfg) are provided in Samples/data folder.
724 * Please refer to the VisageTracker Configuration Manual for further details on using the configuration files and all configurable options.
725 *
726 *
727 * @param trackerConfigFile the name of the tracker configuration file.
728 * @param au_fitting_disabled disables the use of the 3D model used to estimate action units (au_fitting_model configuration parameter). If set to true, estimation of action units shall not be performed, and action units related data will not be available. Disabling will result in a small performance gain.
729 * @param mesh_fitting_disabled disables the use of the fine 3D mesh (mesh_fitting_model configuration parameter). If set to true, the 3D mesh shall not be fitted and the related information shall not be available. Disabling will result in a small performance gain.
730 */
731 void _setTrackerConfiguration(string trackerConfigFile, bool au_fitting_disabled, bool mesh_fitting_disabled);
732 
733 /** Sets the inter pupillary distance.
734 *
735 * Inter pupillary distance (IPD) is used by the tracker to estimate the distance of the face from the camera. By default, IPD is set to 0.065 (65 millimeters) which is considered average.
736 * If the actual IPD of the tracked person is known, this function can be used to set this IPD. As a result, the calculated distance from the camera will be accurate (as long as the camera focal lenght is also set correctly). This is important for applications that require accurate distance. For example, in Augmented Reality applications objects such as virtual eyeglasses can be rendered at appropriate distance and will thus appear in the image with real-life scale.
737 *
738 * IMPORTANT NOTE: In case of multitracking, same IPD will be set for all tracked faces!
739 *
740 * @param IPDvalue the inter pupillary distance (IPD) in meters.
741 */
742 void _setIPD(float IPDvalue);
743 
744 /** Returns the current inter pupillary distance (IPD) setting.
745  *
746  * IPD setting is used by the tracker to estimate the distance of the face from the camera.
747 *
748 *
749 * @return Current setting of inter pupillary distance (IPD) in meters.
750 */
751 float _getIPD();
752 
753 /** @} */
754 
void _getAllFeaturePoints3DRelative(float *featurePointArray, int length, int faceIndex)
Returns for all feature points the position in 3D coordinates relative to the face origin...
int _getFP_END_GROUP_INDEX()
Index of the last feature point group.
void _closeCamera()
Closes camera and clears memory allocated by the camera API.
void _getFaceBoundingBox(VsRect *rectangle, int faceIndex)
Returns the face bounding box for the given faceIndex.
void _getFaceModelTextureCoords(float *texCoord, int faceIndex)
Returns texture coordinates.
void _initializeLicense(string license)
Initializes the license.
void _getFaceModelVertices(float *vertices, int faceIndex)
Returns vertex coordinates of the 3D face model.
void _setFrameData(IntPtr texIDPointer)
Updates the pixel data in Unity with data from tracker.
void _track()
Performs face tracking on current frame.
void _openCamera(int orientation, int deviceID, int width, int height)
Initializes camera using native VideoInput library with given orientation, camera width and height an...
bool _getActionUnitUsed(int auIndex)
Returns true if the action unit is used.
int _getFaceScale(int faceIndex)
Returns size of face bounding box in pixels for the given faceIndex.
void _initTracker(string config, int numFaces)
Initialises the tracker with configuration file and a desired number of tracked faces.
void _getFaceModelVerticesProjected(float *verticesProjected, int faceIndex)
Returns projected (image space) vertex coordinates.
bool _getGazeDirectionGlobal(float[] direction, int faceIndex)
Returns the global gaze direction for the given faceIndex.
void _getHeadRotation(float *rotation, int faceIndex)
Returns the current head rotation in radians for the given faceIndex.
int _getFP_START_GROUP_INDEX()
Index of the first feature point group.
float _getTrackingQuality(int faceIndex)
Estimated tracking quality for the current frame for the given faceIndex.
float _getFrameRate()
Returns the frame rate of the tracker, in frames per second, measured over last 10 frames...
void _releaseTracker()
Releases memory allocated by the tracker in the _initTracker() function.
void _getHeadRotationApparent(float *rotationApparent, int faceIndex)
Returns the current apparent head rotation in radians for the given faceIndex.
void _getCameraInfo(out float focus, out int width, out int height)
Returns camera info.
bool _getGazeDirection(float[] direction, int faceIndex)
Returns the gaze direction for the given faceIndex.
float _getIPD()
Returns the current inter pupillary distance (IPD) setting.
void _getAllFeaturePoints2D(float *featurePointArray, int length, int faceIndex)
Returns for all feature points the position in normalized 2D screen coordinates, indicator whether th...
void _getAllFeaturePoints3D(float *featurePointArray, int length, int faceIndex)
Returns for all feature points the global 3D position in meters, indicator whether the point is defin...
void _setTrackerConfiguration(string trackerConfigFile, bool au_fitting_disabled, bool mesh_fitting_disabled)
Sets tracking configuration.
void _getTrackerStatus(int[] tStatus)
Returns tracker status.
void _getHeadTranslation(float *translation, int faceIndex)
Returns the current head translation in meters for the given faceIndex.
void _setIPD(float IPDvalue)
Sets the inter pupillary distance.
void _getFaceModelTriangles(int *triangles, int faceIndex)
Returns triangles coordinates of the 3D face model.
void _getFeaturePoints3D(int N, int *groups, int *indices, float *positions, int *defined, int *detected, float *quality, int faceIndex)
Returns the global 3D feature point position in meters, indicator whether the point is defined...
int _getFaceModelVertexCount()
Returns number of vertices in the 3D face model.
bool _getIrisRadius(float[] radius, int faceIndex)
Returns iris radius values in px for the given faceIndex.
void _grabFrame()
Grabs current frame from the camera.
int _getTimeStamp()
Returns timestamp of the current video frame, in milliseconds, measured from the moment when tracking...
string[] _getActionUnitName(int auIndex)
Returns the name of the action unit with the specified index.
int _getActionUnitCount()
Returns the action unit count.
int _getFaceModelTriangleCount()
Returns number of triangles in the 3D face model.
void _getGroupSizes(int *groupSizeArray, int length)
Get the size of all feature point groups.
void _getFeaturePoints2D(int N, int *groups, int *indices, float *positions, int *defined, int *detected, float *quality, int faceIndex)
Returns the feature points position in normalized 2D screen coordinates, indicator whether the point ...
bool _getEyeClosure(float[] closure, int faceIndex)
Returns eye closure value for the given faceIndex.
void _getFeaturePoints3DRelative(int N, int *groups, int *indices, float *positions, int *defined, int *detected, float *quality, int faceIndex)
Returns the 3D coordinates relative to the face origin, placed at the center between eyes in meters...
void _getActionUnitValues(float[] values, int faceIndex)
Returns all action unit values.