EEG-controlled meal assistance robot with camera-based automatic mouth position tracking and mouth open detection

A Meal Assistance Robot is an assistive device that is used to aid individuals who cannot independently direct food to their mouths for consuming. For individuals who undergo loss of upper limb functions due to amputations, spinal cord injuries or cerebral palsy, self-feeding can be impossible, and to assist such individuals in regaining their independence meal assistance robots have been introduced. In this paper we propose a meal assistance robot that is controlled using user intentions based on Electroencephalography (EEG) signals while incorporating camera-based automatic mouth position tracking and mouth open detection systems. In the proposed system, users select any solid food item that they desire to consume from three different containers by looking at corresponding flickering LED matrices. User intentions are identified through EEG signals using a Steady State Visual Evoked Potentials (SSVEP) based intention detection method. Initial motion commands for scooping food from the containers are generated and sent to the meal assistance robot from this first stage. At the second stage, a camera-based mouth position tracking method is proposed for automatically detecting the user's mouth position and thereby moving the spoon or end-effector of the meal assistance robot towards the mouth of the user. This method is capable of automatically tracking the mouth position of users irrespective of their individual body differences and seating positions. A mouth open/closed recognition method is implemented at the final stage in order to feed food to the users when they desire consumption, indicated by the opening/closing of their mouth. A set of experiments were carried out with healthy subjects to validate the proposed system and results are here presented.