FLIR thermal imaging cameras allow -machines to read human emotions.  


Will robots ever be able to interact with humans? This may sound like a futuristic question, but scientists today are actually researching how machines can understand human emotions. A very promising technology to make this work, is thermal imaging.


Machines have pervaded our daily lives, and so we need to interact with them all the time. Over the centuries, the way we interact has dramatically changed since the machines have evolved from pure mechanical tools to complex robots endowed with humanoid capabilities. Robots have already started to leave factories and begin to enter our schools, workplaces, and homes.

Of course, it makes most sense that people should be able to interact with robots in ways that are comfortable and natural to them. A common key challenge of robots or Artificial Agents (AA) is to set up a contingent interaction. This means that Artificial Agents should not only react to human actions, but that they should react in ways that are congruent with the emotional and psychophysiological state of the human user or interlocutor. The latter aspect is especially relevant for so-called social robots: robots devoted primarily to interact with human interlocutors. Take museum tour-guide robots and robots that interact with the elderly, for example.


Reading human emotions

To endow artificial agents with the capability of reading and interpreting human psychophysiological and emotional states represents a major issue in the field of human-machine interaction. Typically, monitoring psychophysiological and emotional states is performed through the measurements of several autonomic nervous system (ANS) parameters, like skin conductance response, hand palm temperature, heart beat and/or breath rate modulations, peripheral vascular tone and facial expression. 


Conventional technology for monitoring ANS activity usually includes contact sensors or devices. These technologies are of course invasive by nature, and therefore, they can provide biased estimations, because the compliant participation of the individual is required. Thermal imaging on the other hand is increasingly regarded as a potential solution for recording thermal signatures of ANS activity non-invasively. Thermal imaging, allows the contact-less and non-invasive recording of skin temperatures through the measurement of the spontaneous body thermal heat. As the face is usually exposed to social communication and interaction, thermal imaging for psychophysiology is performed on the subject’s face.


Merla 1   Merla 2

In an experimental set-up, a FLIR SC660 camera installed on the vehicle windscreen is monitoring driver behavior.


Screenshot from the driver monitoring processing suite. Visible and thermal videos of the driver are simultaneously displayed together with a set of user-defined physiological parameters.


In fact, it is possible to see a wide range of psychophysiological signals in the thermal signature of a person. Breathing activity can be monitored with thermal imaging as environmental air flows via the nostrils to the lungs and back. High thermal values indicate expiration, while low values indicate inspiration/inhalation. Thermal imaging also allows the computation of the cardiac pulse through the spectral analysis of the thermal signature of the superficial vessels’ blood flow pulsation. Cutaneous (skin-related) blood perfusion rate and facial perspiration activity are other examples of phenomena that can be detected, recorded and quantified by means of thermal imaging.

Emotions may have a thermal signature or may be characterized by activity of the autonomic nervous system, which in turn provides a thermal imprint through which it can be detected. In fact, thermal imaging has been indicated as a potential tool to build up an atlas of the thermal expression of emotional states, given the use of proper classification algorithms. Startling, distress, fear, sexual arousal, and even deception, all have specific thermal signatures that can be detected with thermal images cameras. One study for example suggested that temperature and cutaneous blood flow monitoring of the periorbital vessel during interrogation provides 87.2% accuracy in detecting deceptive individuals.


Assistive robots using thermal imaging

The benefit of thermal imaging for psychophysiology applications, is that it can obtain physiological information noninvasively and ecologically, which means without interfering with the spontaneous activity of a person. In that respect, thermal imaging could open the way for assistive robots, for example, for elder people or for monitoring the regular breathing activity of newborns. Automatic agents devoted to the control of environmental conditions, for example within a car or a house, could work via thermal-based monitoring of vital signs of the human user, so the system can adapt to the user.


Although still theoretical, thermal imaging could be particularly effective for affective robots and automatic agents designed for improving and personalizing learning or treatment strategies on the basis of the measured user’s psychophysiological feedback.


Merla 3

Emotional sweating and sudomotor response. The delivery of emotional pressure (see section 3.2) or stress stimulation (on the right) changes the rest (on the left) temperature distribution. The spotted dark signature is associated with the activity of the sweating glands. (Adapted from Merla, 2007a).



Thermal imaging for driver monitoring

In a study by Arcangelo Merla and others of the Infrared Imaging Lab, ITAB (Chieti, Italy) the possibility of real-time driver monitoring by means of thermal imaging was demonstrated. One of the most important requirements to increase the safety in both private and public motor vehicles transportation is to monitor non-invasively the driver’s vigilance, attention and state.


The most sophisticated technologies so far developed use multi-sensor approaches, involving electro­encephalo­graphy, contact thermistors, near infrared or visible band cameras (especially for eye blinking monitoring). Unfortunately, apart from the cameras, all the proposed approaches require to use contact sensors, which may distract the driver and interfere with the complete and safe control of the vehicle. That is why several studies have been performed only on driving simulator suites and not directly onboard of an actual vehicle.


Merla 4

Facial thermal imprints of a mother-child dyad and nose tip temperature synchronization during distressing situation (Adapted from Ebisch, 2012).


Parameters variations, such as breathing, pulse and eye blinking rate are very effective descriptors of the physical state of individuals, the onset of drowsiness or loss of vigilance. In Merla’s study, thermal imaging proved to be an efficient solution for recording almost all of these relevant vital signs.


Advanced FLIR research camera

In the study, a FLIR SC660 thermal imaging camera was used to monitor the breathing activity and rate, the blinking rate and the heart beat rate variation. The temperature fluctuations in facial regions of sympathetic interests could be monitored as well, which provided additional information, such as maintaining or loss of attention by measuring the forehead temperature, responsiveness to hazard conditions by monitoring periorbital response, and arousal by evaluating perioral and nasal temperature.


An autofocusing FLIR SC660 camera (640x480 FPA; NETD @ 30°C: 0.06 K) was installed on the vehicle windscreen, pointing at the driver’s face. The thermal video was acquired at a full frame rate of 15 fps. For reference purposes, vital information (heart beat rate, breathing rate, galvanic skin response, palm temperature) was acquired by means of an ADInstruments PowerLab data acquisition system. Both the driver and the road were also filmed by means of a high-resolution video camera.

The facial regions of interest (ROI) included the orbital, periorbital, perioral, nose tip, chin and forehead areas. For each ROI, mean, standard deviation, minimum and maximum of the temperature distribution were computed. Eye blinking rate was estimated by means of the variation of the mean temperature in orbital region, as open and close eyes present a completely different temperature distribution. The proposed methodology is completely noninvasive, passive, and suitable for continuous monitoring on board, without interfering with the driving activity and without restrictions for the driver.

Merla 5

Thermal representation for extraction of ROIs along with a vascular representation of the major vessels affecting the subcutaneous temperature of the face (Berkovitz, Kirsch, Moxham, Alusi, & Cheesman, 2013).


“The head of the driver executes complex movements for controlling the traffic requirements and the vehicle,” says Arcangelo Merla. “In this approach, we used a simple but effective tracking method able to properly follow the ROIs when the head of the driver moved without leaving the frontal plane or rotating the head over a limited angle of rotation.”

“Thermal imaging is an extremely promising solution to the problem of contactless, noninvasive, ubiquitous monitoring of driver’s physical state. It can also be performed during nighttime, when the driver’s vigilance, attention and state are more likely to be impaired.”