How Smart Are Commercial Sleep-Tracking Devices and Apps?

In tandem with a study that demonstrates that online information about common sleep disorders are largely inaccurate and unreliable based on a review of the first 200 sites presented in the three most popular search engines, studies have also reviewed sleep apps and discovered that they largely mirror the warnings about content and marketing inaccuracy. Search engines and apps may be very useful when the information is contextualized by a well-established and responsible source. Unfortunately, that is not the case for much that is communicated and marketed for sleep-specific apps.

The graphic on the smartphone screen came off of an advertisement for a modern device that measures sleep. Unfortunately, the chart shows that the person entered sleep through REM first. If that were the case, the person would be suspected of either narcolepsy or severe sleep deprivation. Would you know?
A better way to utilize technology as a powerful tool is to understand the context and limitations of its application. When actual metrics are combined with actual knowledge, the value of the technology increases exponentially. More importantly, undue harm is not done by misleading users about the nature of the metrics performed.
Smart devices and apps are increasingly utilized for acquiring actual information about our sleep, nutrition and fitness habits. That is a wonderful trend that should continue. Nonetheless, contextualization is still required. It is often the case that acquisition data and acquiring methodologies are not explained or are inaccurate.

Consumer Sleep Technologies

a prolific gift... when properly understood
Consumer sleep technologies (CSTs) are widespread applications and devices— sometimes considered 'smart devices', personal health devices and apps— that purport to measure and improve sleep quality.  The American Academy of Sleep Medicine (AASM) has developed a Position Statement on the role of CST's for clinical sleep that is also contextually informative for our purposes. AASM is the leading professional clinical society dedicated to promotion of sleep health by fostering responsible patient-centered care through research, advocacy, and evidence-based practice standards. The Position Statement was published in the Journal of Clinical Sleep Medicine in May of 2018 (Vol. 14, No. 5).

According to AASM, consumer sleep technology (CST) utilization continues to increase as sleep apps remain among the most popular apps downloaded for Apple and Android devices. Sleep specialists, according to AASM, are increasingly asked to affirm or provide feedback regarding the utility and accuracy of sleep apps and devices, including wearables. An increasing number of apps and devices claim to track and define sleep-related metrics and/or improve sleep quality— and even screen for sleep disorders. As evidenced by a couple of fairly comprehensive looks at many of the most popular apps (Bhat et al and Peta et al), the problem is that not only do minimal validation data exist regarding the ability of CSTs to accurately perform these functions, but many of the data acquisition methodologies and functional algorithms inherently cannot measure standard sleep stage metrics (such as brain EEG, for example), while alternative metric algorithms do not even attempt to correlate with EEG standard sleep stage metrics, but instead, remain proprietary in both acquisition methodology and the terminology utilized to report "sleep depth"— (as if implying actual sleep staging without reporting the difference).
The problem, therefore, is not so much that these various devices acquire different metrics— some of which may in fact be extremely useful even to sleep specialists in certain situations, it is the simple fact that THEY ARE NOT CONTEXTUALIZED.

To be fair, of course, many of the devices are not being sold as medical devices or medical apps, although some are categorized as 'medical' within the app store (while other CSTs are self-described as health & fitness or lifestyle & entertainment devices that are not subject to United States FDA oversight). The question really becomes one regarding the viability of the data for authentic contexts such as their potential use to correlate with sleep logs for circadian rhythm characterization or optimization.

AASM's Position Statement has been crafted to provide a strategic outline for future evaluation of CST's, not to dismiss them. Actually, ubiquitous smart technologies are considered useful tools when provided in tandem with evidence-based references and contextualizing information. As stated, the acquisition methodology of most smart phone apps and wearables addressing sleep utilize non EEG technology, in particular actigraphy, which measures body movements by means of incorporated accelerometers. They then interpret the movement pattern as sleep or wake with non-standardized proprietary algorithms.

Despite these warnings, actigraphy monitoring (a similar technology to many devices and apps) has been demonstrated to be very useful when combined with sleep logs and intervention methods (including responsible education) for the differential diagnoses of insomnia types and/or circadian rhythm phase shift characterization. The salient point is that this must be communicated and contextualized.
In addition to the journal commentaries discussed, another Review Article from October 2018 entitled Smartphone Applications to Support Sleep Self-Management: Review and Evaluation, published in the Journal of Clinical Sleep Medicine (JCSM) identified 2,431 apps purported to assist sleep that were then narrowed down to 73 which met the inclusion criteria to study because they attempted to do more than provide relaxing music or an alarm. Out of the 73 only 33 actually collected data automatically through embedded sensors, and only 3 have been evaluated for clinical validity comparing the parameters reported by the apps to the EEG component of a sleep study. The validation studies have shown that the sleep parameters by the apps poorly correlate with sleep staging from EEGs and failed to accurately reflect actual sleep stages. Apps that purport to measure sleep architecture and depth, therefore, are misleading. Some apps collect daily sleep duration estimates, but also other information including snoring or sleep sounds, daily moods, diet, and physical activity (requiring the phone to be place in close proximity to the bed). It is well-established that sleep logs do not align with physiological sleep. Nonetheless, if the function of sleep apps were to appropriately contextualize an individual's general pattern, they could still be valuable devices to assist and inform individuals about their sleep patterns. They could even red-flag circadian rhythm issues such as the difference between weekday and weeknight sleep, the tendency toward later (or earlier) sleep, and specific cognitive associations that could be logged. To date, apps collectively do not do a very good job at raising awareness or promoting healthy individualized sleep optimization.

Journal Commentary

A commentary in the Journal of Clinical Sleep Medicine (Journal of Clinical Sleep Medicine, Vol. 11, No. 7, 2015) on an oft-referenced study (Bhat et al.; "Is There a Clinical Role For Smartphone Sleep Apps? Comparison of Sleep Cycle Detection by a Smartphone Application to Polysomnography"; J Clin SleepMed 2015;11(7):709–715)) states the following:

"Bhat et al... take an important step toward bringing the conversation about consumer sleep apps into the realm of independent validation studies... Anyone familiar with movement-based staging algorithms used in actigraphy monitoring will find no surprise that the phone-near-pillow app cannot distinguish sleep stages... In contrast, and perhaps surprisingly, the sleep versus wake discrimination was reasonable and in fact quite similar to that reported for wrist actigraphy: ~90% sensitive and ~50% specific for sleep. Like wrist actigraphy, the app overestimated sleep, probably because quiet wakefulness contains little movement. Given the lack of distinction between sleep sub-stages, it is not surprising that the smart-alarm function was ineffective in this report".

Lack of Context

Peta et al, (Rate My Sleep: Examining the Information, Function, and Basis in Empirical Evidence Within Sleep Applications for Mobile Devices, Journal of Clinical Sleep Medicine, Vol. 13, No. 11, 2017) reported that many, if not most of the most popular apps with the highest user rating did not provide evidence-based sources for their information content. They go on to state that:

"Sleep apps might influence sleep behavior in several ways. For example, sleep apps may provide users with a helpful tool to manage their own sleep, and self-recorded data provided by apps may prompt those experiencing a sleep problem to seek professional assessment as  findings from this study suggest that people may “self-diagnose” and worry about having a “sleep disorder” on the basis of information which may not be evidence-based. However, app users may also be exposed to their smartphones more frequently before bedtime, a habit known to inhibit the onset of sleep due to artificial light exposure." (See also Circadian Rhythm and Sleep Preparation/Hygiene).

Context Matters.