Koan—a teaching in the practice of Zen Buddhism to invoke the exploration of the inner workings of the mind.
Every day we spend hours on end using smartphones, laptops and other devices. While these technologies are immensely useful, many apps impose significant cognitive and emotional costs. For example, motivated by an industry that incentivises user growth, social networking sites use design mechanisms such as never-ending newsfeeds, eye-catching content and constant notification delivery to maintain user attention.
In recent years, technologists have voiced concerns that these designs exploit vulnerabilities in human psychology. Research shows they make it difficult for users to focus on their current tasks and avoid the constant influx of notifications and habitual check-ins. It's no secret that social networks are heavily incentivised to maintain your attention. More user attention means more opportunities to serve adverts, collect data and make money.
In response, designers have innovated digital wellbeing tools that use techniques like self-monitoring screen time and removing or blocking access to distracting functionality. While useful, these apps are not as effective as they claim to be. The data suggests they fail to influence a long term change in behaviour once the monitoring ends.
Researchers agree that focusing on screentime is too narrow. It's common for digital wellbeing apps to ask users to set oversimplified goals (e.g. 30 minutes of Facebook use per day) when what the user actually wants is to spend less time mindlessly scrolling the newsfeed but more time connecting with people and finding interesting communities.
Wearables and Emotion Prediction
So, how might we help people develop a more nuanced view of their digital behaviours? One solution is to have them self-report their emotional state as they use different apps and features throughout the day. This is useful data, but research shows that people generally have difficulty identifying their emotions. Another solution is to detect these emotions automatically and highlight trends between emotional data and device usage.
Recent advances in wearable tech mean that it's now possible to collect high-resolution data in a non-invasive way. The wearable market is flooded with devices that provide support and advice relating to physical activity and physiological monitoring. Recently, however, we're starting to see large firms like Amazon, Microsoft and Google release emotion recognition and stress management features.
At the same time, as wearable sensors become more accurate, it's becoming easier to predict a user's emotional state. Researchers at the MIT Media Lab have used deep learning methods such as long short-term memory (LSTM) neural networks to predict stress, mood and physical health with striking accuracy. One study forecasted the user's emotional state the following day using only physiological data gathered between 10 am and 5 pm the previous day.
Koan
Based on this research, I designed Koan, an emotion-prediction ring wearable that highlights trends between digital behaviours and emotions, helping users uncover harmful patterns of behaviour. Koan integrates with popular consumer devices like phones, laptops, desktops, smart TVs and smart speakers to track technology usage. Usage data is overlayed with affective data in a self-monitoring dashboard, showing users how long they used a device or app and how they felt throughout. In addition, the system makes evidence-based recommendations to the user based on these insights.
For example, someone who's frequently annoyed during moments when they receive several email notifications, perhaps while trying to concentrate on something important, may benefit from limiting email checking. Studies show that checking email less often can reduce stress, increase productivity and improve general wellbeing. Most importantly, Koan enables users to monitor whether the recommended change in behaviour improves their emotional state, motivating them to stick with the new behaviour or try something else.
Adaptive Interventions
Koan also creates opportunities to improve digital wellbeing by delivering just-in-time adaptive interventions. These nudges are tailored to the immediate and changing needs of the user. For instance, if Koan predicts a fast-approaching moment of boredom, it will recommend taking a break. Research suggests that going for a walk can improve mood, boost productivity and enhance creativity. Again, users are encouraged to monitor whether the behaviour change influences their emotions.
Microsensors
The ring comprises multiple sensors that measure high-resolution physiological data. Electrodermal activity, measured as skin conductance and skin temperature, determines sympathetic nervous state. A 3-axis accelerometer tracks step counts and stillness, both of which correlate to emotional arousal. Blood volume pulse is extracted via a photoplethysmography sensor from which heart rate and heart rate variability can be derived.
Machine Learning Algorithm
To improve the emotion-prediction accuracy, users can opt to share social activity (e.g. messaging and phone call frequency) and mobility data. Previous efforts to predict emotions show that the level of social support in one's life strongly correlates to positive emotions and that one's affective state can be accurately inferred from a user's mobility patterns. Physiological, social and mobility data are fed into a LSTM neural network that predicts the user's current and future emotional state.
So, that's one solution to liberating users from the forces of intelligent, persuasive design. While current approaches to improving digital wellbeing focus on reducing overall screentime, Koan helps users distinguish between healthy and harmful technology use. The system combines high-resolution multimodal data with deep learning methods to predict user emotions and enable better self-management of digital behaviours. Ultimately, Koan acts as an exoskeleton for the mind that puts user values, not impulses, first.
* This article summarises a project I worked on during my master's in Human-Computer Interaction at University College London. You can read the full paper here.
