Pianoverse: Immersive Piano Learning with Wearable Haptic Feedback Glove

UX DESIGN | RESEARCH | PROTOTYPING 

How can we improve music learning experience in Virtual Reality?

The project aims to revolutionize piano learning by introducing a VR piano with a haptic glove add-on. This virtual experience simplifies the traditional challenges of learning piano, offering marked notes/chords on a virtual keyboard.


The haptic glove enhances the learning process by guiding finger movements based on note sequences, facilitating muscle memory for chords/songs. Targeted at beginners with limited resources, the system provides a cost-effective and space-efficient alternative to traditional pianos and lessons.

OVERVIEW

Demo

We decided to work on Piano learning in VR

The project aims to revolutionize piano learning by introducing a VR piano with a haptic glove add-on. This virtual experience simplifies the traditional challenges of learning piano, offering marked notes/chords on a virtual keyboard.

Ideation and Brainstorming

Wearables

Muscle stimulation

Muscle memory

Guidance

Visual Cues

Haptic Feedback

Wearables

Error correction

Ease of learning

Final Idea

THE PRODUCT

The Virtual Reality Piano Tutor with EMS Feedback is an innovative educational technology project designed to revolutionize piano learning experiences. The primary goal is to create an immersive virtual reality environment where users can learn to play the piano with real-time feedback provided through an electric muscle stimulator (EMS) integrated into a specialized glove,

Wearable EMS(Electric muscle simulation) feedback gloves for Piano learning in VR

The gloves are meant to hold the fingers back when the EMS is activated

Steps to restrain fingers:


A servo motor is used to rotate a piece of string tied around the servo horn


The string is connected to a ring that can be fastened around the user’s finger


This restricts the user’s range of motion, so that they can’t bring the targeted finger all the way down

Implementation

STAGE 1

Low-fi prototype to visualise the idea

STAGE 2

Testing different individual parts of the idea

VR Environment Setup:

Established a virtual reality (VR) environment using Meta Quest 2, incorporating precise hand tracking for interactive experiences with virtual objects.

Integrated a pre-made Piano prefab into the VR space, customized it for individual pressable keys, and synchronized it with the headset's hand tracking capabilities.


Wearable Device Prototype - EMS Placement:

Explored and tested optimal locations for Electric Muscle Stimulation (EMS) pads on the hand and arm to target specific fingers and nerves.

Experimented with using EMS to induce arm movements, enhancing sensory feedback for users regarding hand positioning during piano play.


Sound Integration for Virtual Piano:

Edited and added sound to the Piano prefab, ensuring that each key produces a distinct sound upon interaction with virtual hands/fingers.

Enabled seamless integration between the virtual piano and the Meta Quest 2's built-in hand tracking system.


Communication Integration in Progress:

Reached a stage where both the VR headset and the Wearable Prototype functioned independently, yet efforts were ongoing to establish communication between the two systems.

Working towards seamless integration to enable real-time coordination between the virtual piano simulation and the EMS feedback provided by the wearable device.

STAGE 3

Final Prototyping

Control Mechanism:

Integrated the wearable device with a VR headset using Wi-Fi and ESP32, allowing real-time communication for finger control. Created a Peer to Peer network, enabling the VR headset to send instructions to the ESP32, controlling both servo motors and EMS.


VR Practice Mode:

Implemented a "Practice" mode in the VR scene, displaying a falling key visualization for users to follow the note sequence. Visual feedback included turning keys green for correct presses and red for incorrect ones.


Communication Protocol:

Established communication between VR headset and ESP32 using a web socket server. Unity's Native WebSockets library facilitated message exchange, sending note sequences from the VR scene to the wearable device.


Sequenced Learning:

Utilized Unity to assign specific piano keys for the note sequence. The VR headset sent the expected note names to the ESP32, guiding the user through the sequence. After completing the sequence, a stop command was sent, and the scene reset.


Arduino Code Logic:

On the Arduino side, the code stored the most recent note value and activated the play function. The play function triggered EMS at intensity level 1 and adjusted the servos, following the mapped finger sequence. A stop command reset servos and set EMS back to level 0, concluding the learning sequence.


This project successfully merged VR technology with a wearable device, providing an immersive and interactive platform to teach piano muscle memory through a combination of servo motor control and EMS feedback.

Fin.

Team

My Role

Team of 5 students

I collaborated with my team to effectively implement the prototype. My significant contributions were primarily focused on hardware prototyping and the design of the haptic glove.

Timeline

Aug’23 - Dec’23

Interactive Technologies in HCI

Course

Demo

We decided to work on Piano learning in VR

The project aims to revolutionize piano learning by introducing a VR piano with a haptic glove add-on. This virtual experience simplifies the traditional challenges of learning piano, offering marked notes/chords on a virtual keyboard.

Ideation and Brainstorming

Muscle memory

Guidance

Visual Cues

Wearables

Haptic Feedback

Error correction

Ease of learning

Muscle stimulation

Final Idea

THE PRODUCT

Wearable EMS(Electric muscle simulation) feedback gloves for Piano learning in VR

The Virtual Reality Piano Tutor with EMS Feedback is an innovative educational technology project designed to revolutionize piano learning experiences. The primary goal is to create an immersive virtual reality environment where users can learn to play the piano with real-time feedback provided through an electric muscle stimulator (EMS) integrated into a specialized glove,

The gloves are meant to hold the fingers back when the EMS is activated

Steps to restrain fingers:


A servo motor is used to rotate a piece of string tied around the servo horn


The string is connected to a ring that can be fastened around the user’s finger


This restricts the user’s range of motion, so that they can’t bring the targeted finger all the way down

Implementation

STAGE 1

Low-fi prototype to visualise the idea

STAGE 2

Testing different individual parts of the idea

VR Environment Setup:

Established a virtual reality (VR) environment using Meta Quest 2, incorporating precise hand tracking for interactive experiences with virtual objects.

Integrated a pre-made Piano prefab into the VR space, customized it for individual pressable keys, and synchronized it with the headset's hand tracking capabilities.


Wearable Device Prototype - EMS Placement:

Explored and tested optimal locations for Electric Muscle Stimulation (EMS) pads on the hand and arm to target specific fingers and nerves.

Experimented with using EMS to induce arm movements, enhancing sensory feedback for users regarding hand positioning during piano play.


Sound Integration for Virtual Piano:

Edited and added sound to the Piano prefab, ensuring that each key produces a distinct sound upon interaction with virtual hands/fingers.

Enabled seamless integration between the virtual piano and the Meta Quest 2's built-in hand tracking system.


Communication Integration in Progress:

Reached a stage where both the VR headset and the Wearable Prototype functioned independently, yet efforts were ongoing to establish communication between the two systems.

Working towards seamless integration to enable real-time coordination between the virtual piano simulation and the EMS feedback provided by the wearable device.

STAGE 3

Final Prototyping

Control Mechanism:

Integrated the wearable device with a VR headset using Wi-Fi and ESP32, allowing real-time communication for finger control. Created a Peer to Peer network, enabling the VR headset to send instructions to the ESP32, controlling both servo motors and EMS.


VR Practice Mode:

Implemented a "Practice" mode in the VR scene, displaying a falling key visualization for users to follow the note sequence. Visual feedback included turning keys green for correct presses and red for incorrect ones.


Communication Protocol:

Established communication between VR headset and ESP32 using a web socket server. Unity's Native WebSockets library facilitated message exchange, sending note sequences from the VR scene to the wearable device.


Sequenced Learning:

Utilized Unity to assign specific piano keys for the note sequence. The VR headset sent the expected note names to the ESP32, guiding the user through the sequence. After completing the sequence, a stop command was sent, and the scene reset.


Arduino Code Logic:

On the Arduino side, the code stored the most recent note value and activated the play function. The play function triggered EMS at intensity level 1 and adjusted the servos, following the mapped finger sequence. A stop command reset servos and set EMS back to level 0, concluding the learning sequence.


This project successfully merged VR technology with a wearable device, providing an immersive and interactive platform to teach piano muscle memory through a combination of servo motor control and EMS feedback.

Fin.

How can we improve music learning experience in Virtual Reality?

The project aims to revolutionize piano learning by introducing a VR piano with a haptic glove add-on. This virtual experience simplifies the traditional challenges of learning piano, offering marked notes/chords on a virtual keyboard.


The haptic glove enhances the learning process by guiding finger movements based on note sequences, facilitating muscle memory for chords/songs. Targeted at beginners with limited resources, the system provides a cost-effective and space-efficient alternative to traditional pianos and lessons.

Demo

I believe in the enchanting power of good design, capable of weaving magic into every experience. Let's work together and make magic!


Feel free to drop an email at mrunal.dhaygude@gmail.com.


MRUNAL DHAYGUDE

Pianoverse: Immersive Piano Learning with Wearable Haptic Feedback Glove

UX DESIGN | RESEARCH | PROTOTYPING 

Team

Team of 5 students

Team

My Role

Team of 5 students

I collaborated with my team to effectively implement the prototype. My significant contributions were primarily focused on hardware prototyping and the design of the haptic glove.

Timeline

Aug’23 - Dec’23

Interactive Technologies in HCI

Course Role