Attention with variational autoencoder enabled one-dimensional convolutional memory network for emotion recognition using EEG and speech signals

Emotion recognition is an emerging task that concentrates on recognizing the person’s diverse emotional states. However, several previous attempts have been made to acknowledge the person’s emotions and effectively classify them with better performance. However, several limitations have occurred during the detection of human emotions, such as high computational power consumption, minimal recognition accuracy, and diverse modality recognition.

Therefore, the research proposes the Self-modular Attention with Variational Autoencoder-enabled one-dimensional Convolutional Neural Network and Long Short-Term Memory Network (SMAVA-1DCNN-LSTM) that overcomes the previous approach’s limitations and provides robust detection results with minimal computational complexity. In the research, emotions are recognized by internal physiological signals (EEG) and voice tone. During an evaluation, the significant features of speech and EEG signals are extracted through multiple feature extraction mechanisms that boost the classification accuracy and training process by minimizing overfitting problems.

Specifically, the Self-modular Attention (SMA) mechanism aids in fusing the modality-specific informative features to concentrate on important parts of the input multimodal data. Experimental results show that the SMAVA-1DCNN-LSTM method demonstrates exceptional performance for emotion recognition, reporting accuracy of 97.70%, F1-score of 97.79%, sensitivity of 97.06%, MCC of 0.954, precision of 98.54%, and specificity of 98.34% for 90% of training with the EAV dataset. The author would like to express my very great appreciation to the co-authors of this manuscript for their valuable and constructive suggestions during the planning and development of this research work.

Department of Electronics and Telecommunication Engineering, Dr. Babasaheb Ambedkar Technological University, Lonere, Maharashtra, 402103, India The authors declare no competing interests. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Kothari, N.S., More, S.A. & Jadhav, N.S. Attention with variational autoencoder enabled one-dimensional convolutional memory network for emotion recognition using EEG and speech signals.