Sergio Pulido - Aprendiendo con Emma

About

Aprendiendo con Emma is an innovative educational game designed to empower children with ASD (Autism Spectrum Disorder) by enhancing their emotional awareness and recognition abilities. In this engaging experience, players collaborate with Emma, a virtual guide, to explore the world of emotions through interactive activities. Aprendiendo con Emma is a powerful tool for helping children better understand and express complex emotions.

Info

Role: Designer/Solo developer

Team size: 1

State: Completed

Time frame: 1 year

Introduction

Aprendiendo con Emma invites you to embark on an enriching journey within the realm of educational gaming. This unique Windows-based game was meticulously crafted as part of an innovative project during my master's degree in electronic engineering.

Tailored for the special needs of children with Autism Spectrum Disorder (ASD), Aprendiendo con Emma serves as an empathetic companion in the world of emotional intelligence. Our goal is to foster a deeper understanding and expression of complex emotions.

Step into the virtual world with Emma, your virtual guide, as she demystifies the intricate language of emotions. Throughout the game, you'll embark on a journey that involves detecting emotions in various situations, expressing emotions yourself (with guidance and feedback), and learning to connect emotions with their corresponding facial expressions.

Aprendiendo con Emma is more than a game; it's a powerful tool designed to nurture emotional growth in children with ASD, providing them with essential life skills. Join us on this educational adventure and discover the world of emotions through the eyes of Emma.

What I learned...

Aprendiendo con Emma was a unique project that exemplified the power of collaboration across different disciplines. In this endeavor, the worlds of electronic engineering and psychology merged to create an innovative game. This game was designed to assist children with ASD in recognizing and expressing emotions. The utilization of computer vision and other technologies in this context underscored the importance of leveraging diverse expertise for a common goal. This project taught me the value of multi-disciplinary teamwork and the endless possibilities it can unlock.

Design

Aprendiendo con Emma presented a distinctive design challenge, where the primary focus transcended entertainment and delved into education. This multidisciplinary project, led by experts in electronic engineering and psychology, aimed to assist children with ASD in recognizing and expressing emotions.

Traditional game development primarily centers on entertainment, but Aprendiendo con Emma's core objective was educational, with a specific focus on aiding children with ASD.

Harmonizing the elements of childhood engagement with the educational significance of recognizing and expressing emotions became a fundamental challenge. Striking this balance was essential to ensure the game's effectiveness.

Effective communication of complex technical concepts, typically associated with game development, to a diverse team of experts, including psychologists, was vital for successful collaboration and comprehension.

Establishing clear objectives, defining goals, and adhering to predefined timelines facilitated the creation of a design that seamlessly integrated education and entertainment, garnering satisfaction from all team members.

Dynamic Libraries

This project opened the door to dynamic libraries (DLLs) that played a crucial role in incorporating the OpenCV library for advanced image processing into the game. Bridging OpenCV and our C# game demanded coding vital computer vision functions in a dedicated C++ environment.

The need to incorporate OpenCV for image processing presented a unique challenge, as the library was predominantly designed for C++ and Python environments.

To integrate OpenCV into our C#-based game, I crafted a comprehensive suite of computer vision functions within a controlled C++ environment.

Thorough testing ensured the reliability and functionality of these functions before they were separated into a distinct project that included only the specific OpenCV packages required for face detection and facial expression analysis. Extraneous packages were excluded, optimizing resource usage.

Once this dedicated project was exported as a dynamic library, it was seamlessly integrated into Unity, providing our game with the necessary OpenCV-based functions to elevate its computer vision capabilities.

Recognition of Facial Expressions

A crucial aspect of Aprendiendo con Emma was recognizing facial expressions, which involved several key algorithms and techniques.

Image enhancement using histogram equalization to improve image quality.

Face detection with OpenCV's Single Shot Detector (SSD) algorithm.

Kazemi's face landmark detection for locating 68 key facial landmarks.

Computation of facial features, such as eyebrow angles, lip thickness, eye height, forehead gradient, using the 68 landmarks.

Utilization of a pre-trained Random Forest machine learning algorithm to recognize and classify users' facial expressions based on computed features.

Detection of faces through SSD.

Identification of landmarks through Kazemi's algorithm.

Some of the features used for facial expression recognition.

Results

Aprendiendo con Emma brought many satisfactory results, both in terms of the results of the project and to my growth as game developer and software developer in general.

Project Results

Experimental Protocol: Designed an experimental protocol based on psychological and neurodevelopmental aspects of ASD, featuring a progressive difficulty level focused on imitation and stimulation.
Subject Improvement: Psychological tests revealed significant improvements in the expression and recognition of emotions through facial expressions for most participants.
Published Papers:
- One paper detailing the facial expression recognition algorithm has already been published:
  - Pulido, S.D., Palacios, N., Ballen, M.P., et al. Ensemble of Machine Learning Models for Improved Facial Emotion Recognition. In: 2021 IEEE URUCON. Montevideo, Uruguay (2021).
- Another paper, detailing the methodology and results of the project, is currently in the process of being published.

Personal Results

Game Development Expertise: The creation of Aprendiendo con Emma marked a significant milestone in my journey as a game developer. It provided me with hands-on experience in developing a Windows-based educational game, expanding my skill set and knowledge in game development.
Dynamic Libraries (DLLs): Aprendiendo con Emma was my introduction to dynamic libraries (DLLs). This experience was invaluable in integrating the OpenCV library into the game, enabling advanced image processing capabilities. It required me to bridge the gap between C++ and Python-centric OpenCV, used in a C# game, by coding essential computer vision functions in a dedicated C++ environment.
Enhanced Multidisciplinary Collaboration: Working within a multidisciplinary team alongside psychologists, I honed my ability to communicate complex technical concepts to non-technical team members. Finding a harmonious balance between entertainment and educational content was a challenge I successfully navigated.
Project Management: Managing an educational game project for children with ASD involved setting clear goals, establishing time limits, and prioritizing objectives. This experience provided me with essential project management skills.

Roulette mini-game in Aprendiendo con Emma.

Facial expression mini-game in Aprendiendo con Emma.