Overview

Synesthesia Simulator was a virtual reality application that simulated three different variations of synesthesia, a syndrome that manifests as a crossover of multiple sensory experiences. I created this VR app using Unity, C#, and the Oculus Rift in May 2017.

Description

Synesthesia is a neurological condition in which stimulation of one sense pathway leads to automatic and simultaneous experiences in a second sense. Our project simulated three types of synesthesia for users unfamiliar with experiencing the condition. Since synesthesia affects multiple sensory organs, this makes it a natural fit with the sensory manipulation possible through virtual reality.

We created a synesthesia simulator that allowed users to experience three of the many types of synesthesia: chromesthesia, spatial-sequence, and grapheme-color. The user remained sitting with an Oculus Rift headset while making selections with an Xbox controller. To minimize vection and visually induced motion sickness we did not allow users to physically walk around the scenes. Ambient sounds played continuously throughout the application.

The user started in a room with three doorways and had the option to choose which sense they wanted to experience. If they looked around the room, they saw an explanation of synesthesia and a description of each experience placed on the walls. 

Chromesthesia is the association of sounds with colors, and the colors triggered by certain sounds are referred to as photisms. We placed the user in the center of a 360-degree video that depicts a beach scene with photisms connected to the sounds of the waves and the birds flying overhead. The pink lines changed shape, color, and intensity depending on the volume and position of the wave sounds. The yellow dots followed the cawing seagulls across the sky.

Spatial-sequence synesthesia is the perception of time as a physical manifestation in space. For this scene we took inspiration from researcher Holly Branigan’s synesthesia, which she described as a rough 3-D circle that surrounded her body whenever she thought about time. We created a circle of months surrounding the viewer where each month has a specific location in space. The current month was closest to the viewer, while future months became physically further and further away.

Grapheme-color synesthesia is the perception of numbers and letters as inherently colored. Each letter, number, or word is associated with a specific color; however, that color is different for each synesthete. We put the user inside a classroom and showed them a sequence of letters and numbers. The user can switch back and forth between colored and black text using their controller. We wanted users to see the contrast between their own experience and a synesthete’s experience when reading.

CS 498 SL: Virtual Reality was a special-topics course that provided students with a deep understanding of the fundamentals of VR as well as practical experience in developing VR experiences. As part of this course I studied computer graphics, tracking systems, light and optics, perceptual psychology, and human physiology to obtain a comprehensive understanding of virtual reality that is independent of current VR technology.

Collaborators

Doug Ebersol, Kevin Eicken, Kate Milleker

Screenshots