Every day, billions of people around the world spend hours using laptops, smartphones and other devices. While these technologies are incredibly useful, many systems impose emotional and cognitive costs. Motivated by an industry that incentivises user growth, these technologies employ design mechanisms—such as never-ending newsfeeds, eye-catching content and constant notification delivery—to maintain user attention. In recent years, people have voiced concerns that many of such services exploit vulnerabilities in human psychology. These techniques make it difficult for users to focus on their current task and avoid the constant influx of notifications and habitual check-ins. It may then come as no surprise that compulsive internet use is linked with a loss of agency and decreased psychological wellbeing.
Digital wellbeing has emerged as a term to describe the extent to which a person perceives their technology use to be aligned with their long-term goals. Researchers in this space have developed digital wellbeing tools, using techniques like self-monitoring screen time and removing or blocking access to distracting functionality. Despite these efforts, the relationship between screen time and wellbeing measures is still unknown. The key ingredient missing from such tools is understanding how one's emotional state relates to technology use. Emotions are complex and dynamic processes that can have a range of beneficial and detrimental long-term health consequences. For example, happiness correlates with greater longevity, while stress can increase susceptibility to infection and illness. As technology adoption increases—the time spent online has doubled in the last decade—there is a growing need for a system that helps people understand which digital activities have the most influence over emotional state.
In this article, I present Koan, an emotion prediction ring-wearable that helps users understand the relationship between their digital behaviours and emotions. It is intended to motivate better self-management of digital behaviours by overlaying technology usage statistics with affective data and making evidence-based recommendations to the user.
Wearables
Recent innovations in wearable technologies have made it possible to collect high-resolution data in a non- invasive way. The wearable market is flooded with devices that provide support and advice relating to physical activity, physiological monitoring, and to a lesser extent, emotional monitoring. Large firms like Amazon, Microsoft and Google (Fitbit), along with newcomers Upmood and Moodmetric, have begun developing emotion recognition and stress management tools.
Physiological and emotional states are measured via various signals, including heart rate (HR), heart rate variability (HRV), body temperature, respiration rate, and electrodermal activity (EDA). EDA refers to slight changes in the skin's electrical activity such as sweating, blood pressure changes, and hair follicle elevation. Physiologists have described it as the most effective correlate for determining emotional arousal.
Predicting Emotions
As wearables become more accurate, the goal of forecasting emotions becomes more of a reality. Researchers have used deep learning methods such as long short-term memory (LSTM) neural networks to predict stress, mood and physical health with striking accuracy. Taylor and colleagues trained LSTM models using today's physiological and behavioural data to forecast tomorrow's wellbeing (good-poor stress and mood) with 78-82% accuracy. Umematsu and colleagues used LSTM neural networks to forecast tomorrow's high-low stress using seven days of time-series multimodal data. In a follow up to these studies, researchers have predicted tomorrow's affective state on a scale of 0-100 using physiological data gathered between 10 am and 5 pm the previous day.
Koan
Koan is an emotion prediction system composed of a ring-wearable and companion smartphone and desktop applications. The system is aimed at anyone who spends prolonged periods using technology. However, it is especially helpful for people who experience cognitive failures and poor subjective wellbeing from excessive technology use.
Koan is intended to improve digital wellbeing by drawing attention to the relationship between emotions and device usage. Affective data is overlayed with technology use in a self-monitoring dashboard (pictured below). Koan will highlight trends between digital behaviours and emotions, helping users uncover harmful patterns of behaviour. Based on these insights, the system will make evidence-based recommendations. For example, a user who is frequently annoyed during moments when they receive several email notifications, perhaps while trying to concentrate on something important, may benefit from limiting email checking. Studies show that checking email less often can reduce stress, increase productivity, and improve general wellbeing. Most importantly, Koan enables users to monitor whether the recommended change in behaviour improves their emotional state, motivating them to stick with the new behaviour or try something else.
The system creates opportunities to improve digital wellbeing by delivering just-in-time adaptive interventions. These nudges are tailored to the immediate and changing needs of the user. For instance, if Koan predicts a fast-approaching moment of boredom, it will recommend taking a break (see example below). Research suggests that going for a walk can improve mood, boost productivity, and enhance creativity. Again, users are encouraged to monitor whether the behaviour change influences their emotions.
The ring comprises multiple sensors that measure high-resolution physiological data (see ring schematic pictured below). EDA, measured as skin conductance and skin temperature, is used to determine sympathetic nervous activity. A 3-axis accelerometer tracks step counts and stillness, both of which correlate to emotional arousal. Blood volume pulse is extracted via a photoplethysmography sensor from which HR and HRV can be derived.
A miniature touch-enabled display located on the ring allows users to view the information most important to them. Data is displayed in a duo-tone minimalist design to maximise visual acuity. Previous work notes that miniature displays present several interaction challenges for wearable devices. Consequently, the touch interaction is constrained to a single swipe across the screen that cycles between data options. Micro-animations signal this action to first-time users; the data will bounce slightly from left to right, prompting a swipe. Alternatively, users can adjust the display settings via the smartphone and desktop applications.
Koan integrates with popular consumer devices (e.g. phones, laptops, desktops, smart TVs and smart speakers) to track technology usage. The data is displayed to users in a self-monitoring dashboard, showing them when and for how long they used each device or application.
The system also collects social activity (e.g. messaging and phone call frequency) and mobility data. Recent studies suggest the level of social support in one's life has a strong connection to positive emotions and that affective state can be accurately inferred from a user's mobility patterns. Physiological, social, and mobility data are fed into a LSTM neural network which predicts current and future affective states (pictured below). While the model needs only a few hours of data capture to make a prediction, accuracy improves as more data is collected.
Persona and Context of Use
Charlotte is a 32-year old stockbroker living in New York City. She juggles a busy schedule, constantly moving between meetings, researching the financial markets, and pitching to new clients. Recently, she's been feeling down in the dumps and is struggling to understand why. She is interested in tracking her emotions to see what can be done to improve how she feels.
The day is coming to an end, and Charlotte is figuring out tomorrow's schedule (see storyboard blow, a). She closes her calendar and opens the Koan desktop app (b). The system predicts she will likely feel annoyed, apprehensive and a little sad tomorrow morning. A recommendation beside this insight reads: "You usually spend the first 15 minutes of the day in bed, checking the news and scrolling through Instagram. Why not try reading a book or going for a walk instead?" The following morning she reluctantly reaches for her Kindle instead of her phone (c). A few hours later, Charlotte notices a spring in her step on her way to work (d). Curious about whether her emotional state was different than usual this morning, she checks the Koan smartphone app (e). Sure enough, joy was spiking all morning, even after she had finished reading. Several days later, after a few more mornings like this, Koan notifies Charlotte that her emotional state has significantly improved since she stopped checking the news and social media first thing in the morning (f). This information reinforces the positive behaviour, motivating Charlotte to stick with her new morning habit.
Measuring Success
Koan's efficacy could be evaluated in a between-subjects study assessing (i) whether the system affects psychological wellbeing and (ii) if users learn anything from the experience.
- Baseline phase (1 week): Participants are instructed to use technology as they usually would.
- Experimental phase (2 weeks): Participants are randomly assigned to either continue using technology as normal or to use Koan.
Throughout both phases, subjects would self-report measures of psychological wellbeing. Wellbeing measures could include mood, positive and negative affect (i.e. happiness), stress, anxiety, inattention and social connectedness.
At the end of Week 3, participants would be interviewed to understand (i) whether Koan functioned as intended, (ii) what participants learnt from the intervention and (iii) why they thought the intervention was a success or failure.
In this article, I presented a novel emotion prediction wearable that liberates users from the forces of intelligent, persuasive design. While current approaches to improving digital wellbeing focus on reducing screentime, Koan helps users distinguish between healthy and harmful technology use. The system combines high-resolution multimodal data with deep learning methods to forecast emotions and enable better self-management of digital behaviours. Ultimately, Koan acts as an exoskeleton for the mind, which puts user values, not impulses, first.
