Multimodal Fusion of Behavioral Signals for Real Time Stress and Focus Detection in Mobile Gaming

Abstract

Mobile gaming reaches hundreds of millions of users worldwide, yet sustained gameplay sessions are associated with rising stress levels and declining attentional focus. Existing affect detection systems depend on physiological sensors such as EEG headsets, electrodermal activity monitors, and photoplethysmography devices that are expensive, obtrusive, and poorly suited to everyday smartphone use. This paper introduces MindPlay, a lightweight multimodal machine learning framework that detects stress and focus as independent binary classification tasks in mobile gamers using only the device front facing camera and in game interaction logs. The system combines Convolutional Neural Network facial emotion embeddings with XGBoost classifiers trained on gameplay behavioural data preprocessed by Fast Independent Component Analysis. The CNN module is pre trained on FER2013 (35,887 images) and fine tuned on a multi source corpus spanning 127 unique participants. Stress is labelled using the Self Assessment Manikin arousal dimension: sessions with a SAM score equal to or greater than 5 on a 9 point scale are classified as high stress. A 128-dimensional CNN embedding concatenated with four FastICA transformed behavioural components forms a 132-dimensional feature vector. On the combined corpus the stress task achieved 89.4% accuracy and ROC AUC of 0.941; the focus task reached 87.8% accuracy and F1 of 87.2%. The full pipeline runs at 12fps with end to end latency below 85ms on mid range Android hardware, requiring no additional sensors.