It’s hard to believe portable video cameras have been available since the 1980s. Sony and Panasonic were crucial players in the race toward market share with VHS technology winning the first major market battle. Of course digital technology is now favoured by manufacturers and developers alike, and so the market continues twisting and turning depending on the speed and saturation which can be gained by software and hardware developers.
As we step through our own R&D environment, it’s interesting to consider the significant differences in software and hardware available in the automotive landscape and equally important to ensure we can adapt to the specific market forces in both the consumer and industrial markets.
PERCLOS limitations
At Optalert, we have made our intentions fairly clear. For industrial environments, where driver privacy is an issue and where the driver is often moving in and around the cabin or train carriage, Optalert’s glasses, with their sampling rate of 500 times a second and ability to be always tracking the eye, independent of head position, will always provide the most practical solution. However, given the rapid adoption of driver-facing video systems in the broader consumer market, it is important our technology can run effectively on existing in-vehicle video systems.
Many existing fatigue systems available in the market use PERCLOS; the percentage of eyelid closure over the pupil over time. While Optalert’s Johns Drowsiness Scale (JDS) measures the amplitude and velocity of blinks (AVRs), PERCLOS reflects slow eyelid closures or “droops” rather than blinks.
Optalert Founding Director Dr Murray Johns noted the limitations of the PERCLOS measurement many years ago. While it recognises drowsiness in its late stages, he believed that was far too late. If an operator is already asleep while they are travelling at high speed, they can do a great deal of harm in the seconds before they are awoken. It was these limitations which led him to the significant research required in developing the JDS; named after him.
There have also been significant research papers written about the limitations of PERCLOS. Jackson et al authored The utility of automated measures of ocular metrics for detecting driver drowsiness during extended wakefulness (November 2015). In it they compared PERCLOS and Optalert’s JDS and concluded “Comprehensive algorithms inclusive of ocular parameters (i.e. the JDS) may be a better indicator of performance impairment following sleep loss.”
Measuring with the JDS
The most significant difference when measuring with the JDS is the opportunity to identify the risk of drowsiness before it occurs. It is predicting the likelihood a person will become drowsy, although it is impossible to determine exactly when a person will fall asleep. It measures the risk of performance failure which may occur even before the driver has a micro sleep, and interestingly, a micro sleep can occur while the person has their eyes wide open. It is therefore imperative a measurement like the JDS is used to ensure the person remains safe and is alerted before they are in danger.
Optalert’s JDS is the most reliable method on the market and Optalert has produced the only systems currently capable of measuring the dangerous state of drowsiness and provide the only solutions with scientific validation. For consumer automotive applications however, the prevalence of camera technology has led us to embark upon a video certification program to determine systems capable of running Optalert’s patented JDS. The first system certified was the Smart Eye system which has come out of Europe.
The increasing video camera market
Two years ago it was reported there were more than 245 million video surveillance cameras installed across the globe and this number has been expected to rise 10-fold. Action cameras like the GoPro® have become ubiquitous for part-time sport lovers and thrill seekers, while Facebook, Google and Apple have been scrambling to own the first facial recognition software and now gaze and emotion detection software.
Video in the automotive space
Denso, is one company looking at driver assistance video camera options. It is claimed they are even looking at a fatigue system which would be connected to the vehicle’s air conditioner. If the system detects the driver is fatigued, it would provide a blast of cold air to wake them up. We would have to assume though, this is still only detecting drowsiness in its late stages, and therefore may be less successful than the JDS used by Optalert, unless they have found a way to measure the risk of drowsiness before the driver becomes drowsy.
HARMAN have been generating 65% of their revenue from their automotive systems and unveiled new driver monitoring earlier this year. They claim to be able to “capture the driver’s pupil dilation and their proprietary software algorithm analyses the pupil reflex using advanced filtering and signal processing”. Samsung obviously has a firm belief in their software as they are in the process of acquiring the company.
Delphi-Mobileye are developing a low-cost self-driving solution and now have Intel along for the ride.
Optalert’s current focus
As we continue down the path of video camera certification, Optalert will be testing a number of low-cost camera systems. One of the more recent systems evaluated was RealSense, for which we purchased a developer kit including camera for under $150.
Testing RealSense
For this testing, it’s important to note we used their standard image processing library, which has no optimisation for measuring eye closure. But, even with this off-the-shelf product, we were able to obtain good correlation for a number of drowsy and alert sets between the JDS measured from Optalert’s eagle glasses and the RealSense video system.
We observed some cases where the image processing failed to track the eye reopening accurately, which led to a less accurate measurement of drowsiness (see picture below). It should be reiterated that the RealSense software tracks over 80 facial landmarks and has not been optimised for eye opening measurements.
This is a limitation on the image processing software algorithm, which when fixed, will lead to even better results.
The best 5 results
We showed the Optalert algorithm can measure drowsiness of subjects even on low cost hardware and software platforms. All that is required is 50fps and an accurate tracking of the eye opening.
Future focus
The ability of a car to detect the early signs of driver drowsiness in a quantified method opens up a range of new features as they could then delay or prevent the driver from reaching a state where they may have a “performance failure” due to drowsiness. This would not just be through giving warnings to the driver, but also interacting with other car systems, such as environment control, GPS waypoints and all the way through to computer verbal interaction with the driver, to keep their brain engaged. This becomes even more important with semi-autonomous vehicles where the drivers’ brain is under loaded even further, yet needs to be alert to take control at any moment.
Optalert is looking to work with more companies in the in-vehicle camera and driver image processing space and they can submit certification requests here.
