Lie to me: reading hidden emotions

Microexpressions magnified at different levels Microexpressions magnified at different levels

IEEE researchers have studied micro-expressions: rapid, involuntary facial expressions which reveal emotions that people do not intend to show. This is of extreme importance in forensic science and psychotherapy, and we all remember the TV show Lie to me, which focused on these micro-expressions (MEs). Unlike the pompous TV shows, real-life analysis of spontaneous MEs is very challenging due to their short duration and low intensity. Therefore, automatic ME analysis includes two different tasks: ME spotting and ME recognition. Previous studies have focused on posed rather than spontaneous videos for ME spotting, and the performance of previous studies was low for ME spotting. The novel approach consists of a new method for spotting spontaneous MEs by exploiting feature-difference contrast, and an advanced ME-recognition framework is introduced as well, along with the first automatic ME-analysis system, that spots and recognizes MEs from spontaneous video data.

MEs are very short (1/3 to 1/25 seconds), and the intensities of involved muscle movements are subtle. The phenomenon was first discovered by Haggard and Isaacs in 1966, who called these the micromomentary facial expressions. In 1969 Ekman and Friesen also reported finding MEs while examining a video of a psychiatric patient, trying to find a possible trait of her suicide tendency. Even though the patient seemed happy throughout the film, there was a fleeting look of anguish lasting two frames when the tape was examined in slow motion. The other famous usage is for lie detection, where a face could tell a different story than the statement.

Although finding MEs is very difficult for human beings, it seems that we can do some improvement in computer vision techniques. Facial-expressions analysis has received a great performance with an accuracy of above 90% on FE databases, but studies of MEs see a different challenge: it is difficult to gather a large number of spontaneous ME samples, since they occur only under certain conditions. Other challenges include their short duration and low intensity level.

ME recognition framework diagram

ME recognition framework diagram

For the problem of spontaneous ME analysis there are two major problems that have been mentioned: ME spotting and ME recognition. In natural circumstances MEs may occur along with other more prominent motion such as head movements and eye blinks, which makes the spotting and recognizing MEs very challenging. IEEE researchers have proposed a new method for spotting spontaneous MEs, which is based on feature-difference contrast and peak detection, that does not require training. The consists of four steps: 1. three facial landmark points (inner eye corners and a nasal spine point) are detected in the first frame and tracked through the video, and each face image is divided into a block structure based on the tracked landmark positions; 2. appearance-based features are calculated for each block; 3. the dissimilarity of features for each block of sequential frames within a defined time interval is calculated using the chi-squared distance; 4. thresholding and peak detection are applied to spot rapid facial movements from the video.

Also, they have developed an advanced framework for ME recognition, where they have employed Eulerian video magnification method to counter the low intensity of MEs, and different feature descriptors (Local Binary Pattern, Histograms of Oriented Gradients, Histograms of Image Gradient Orientation) are comprehensively investigated as well.