From mid-2024, all new models of passenger vehicles in Europe require a driver monitoring system (DMS) that can detect drowsiness. Engineering teams across the automotive industry have scrambled to learn about the science of drowsiness and develop systems that can detect it in drivers. Each of these systems then needs to be validated and shown to accurately detect drowsiness. One recurring problem we see in many teams is erroneous test protocols.
Within Optalert, we are frequently astounded that so many tests are not oriented around the only ground truth that matters: driver impairment.
This article begins with a table listing common mistakes to watch out for when assessing a given test protocol. It then outlines in more detail how to devise a protocol that avoids these traps, in line with decades of meticulous research from sleep science laboratories.
Longer-term we believe the industry should agree upon a standardised test protocol. This will allow systems to be compared in terms of performance. Until a broader consensus is reached, adhere to the principles outlined below to ensure a system is measuring what it should.
Our recommendations below provide a clear overview of how to thoroughly administer such a validation test, drawing from preeminent research in this field.
To measure a system’s ability to detect impairment, subjects must be…impaired. This involves a considerable period of extended wakefulness. In general, after 17 hours of extended wakefulness relative risk begins increasing in psychomotor vigilance tests. Driving tasks usually only reveal an increase in relative risk after 24 hours of extended wakefulness, although it varies across subjects. High impact research in the field has tended to keep subjects awake for 30 hours or even 32-34 hours before driving tasks.
If a well-rested person is kept up for a few hours after their bedtime, their ability to drive will likely be unimpaired or only very slightly impaired. They cannot self-assess with KSS or any other subjective measure in this regard. The only relevant ground truth is an increase in performance failures.
One dead giveaway of a test involving unimpaired subjects is if there is no safety driver in the passenger seat. If this is the case, the test is either egregiously unethical or subjects are not truly impaired.
It is critical that a test ensures sleep deprived subjects are indeed kept awake. Carefully controlled research verifies that the impaired participants did not sleep in their extended wakeful period. Methods for verification include wearable accelerometer technology or direct observation by an invigilator. A sleep diary is less reliable because it can be falsified.
Two types of testing are employed for calibrating, testing, or validating a drowsiness detection system: laboratory tests and on-track validation. Ordinarily a team runs many more laboratory tests due to the lower cost and effort involved in their administration.
The Johns Test of Vigilance (JTV) is the psychomotor vigilance task that correlates most closely with performance failures in driving tasks. Over two decades ago, Optalert’s founder Dr. Murray Johns found that the impairment caused by drowsiness presents far more often as simply not responding to a stimulus as opposed to a gradual increase in response time. Consequently, performance failure in the JTV is defined as non-response within 2,000 milliseconds.
Any laboratory test attempting to simulate driving should adopt this definition of performance failure, as it has been extensively validated across decades of research. It is also important to note that extremely rapid responses should be excluded as they are considered anticipatory. The exact timing depends on the complexity of the task, but usually sits somewhere between 50 and 150 milliseconds.
When conducting on-track validation of the drowsiness detection within a DMS, numerous definitions of performance failure could be adopted. Practical options include:
Poor examples of performance failures include:
The table shows an example of a protocol, although as noted there are a range of performance failures that could be selected that accurately map to impairment in real-world driving tasks.
Regardless of which performance failure is selected, the DMS must detect impairment before it occurs. This shows it would have prevented the performance failure in real-world conditions.
In addition, the driver must not be roused throughout the drive by either the safety co-driver or any devices in-vehicle. The DMS must not sound any alerts that would wake the driver.
We encourage automotive OEMs and tier 1s to contact us for more information on how to rigorously validate the drowsiness detection within a DMS.
We are eager to support the industry to employ sound science and ensure we are doing all we can to keep drivers and other road users safe.