Listen to the Data - Literally

An article by Dina Genkina in IEEE Spectrum caught my attention: How Sound Could Make Sense of Big Data. It brought my attention to three things I wasn’t aware of: sonification, the Internet of Sound, and that we need to be thinking about 6G.

That's not quite sonification, but you get the idea. Credit: Microsoft Designer

She interviews Stephen Roddy, an engineer by training who lectures at University College Cork, in the – I kid you not -- School of English and Digital Humanities. His profile says he works “at the intersection of engineering and the sonic arts, developing auditory interfaces for complex data,” which helps explain why Dr. Genkina chose him.

If you haven’t heard of sonification, think of data visualization except instead of seeing data, you hear it. Data visualization has been a thing for some time now, with lots of people producing lots of lovely charts and graphs, but sonification is another way to translate data. It may not be not quite as mainstream as data visualization yet, but we’re finding more and more uses for it. Sonification is tied directly into the Internet of Sound (IoS).

The IEEE Communications Society says that IoS turns Internet of Things (IoT) devices into a “network of devices capable of sensing, acquiring, processing, actuating, and exchanging data serving the purpose of communicating sound-related information.” It further explains that it is the “union of two paradigms: the Internet of Musical Things and the Internet of Audio Things, which respectively address musical and non-musical domains in networked contexts.” There is an Internet of Sounds Research Network that claims 140 partner institutions in 30 counties.  

Credit: Internet of Sounds Research Netywork

As for 6G, most of us have made the transition to 5G, but lots of companies and research organizations are hard at work on 6G, which promises “to support applications like holographic communication, XR, autonomous driving, digital twin and AI native networks.” It is expected within the next decade, and it will further explode the amount of data and what we can do with it.

Professor Roddy told Dr. Genkina:

…sonification truly excels when dealing with highly complex, multidimensional, and noisy data where crucial patterns and relations are invisible to the naked eye. While the human eye operates like a serial processor—focusing on one small chunk of information at a time—the human ear acts as a parallel processor.

As a result, he says, “the ear can effortlessly identify minute variations, process multiple data streams simultaneously, and isolate important signals hidden deep within background noise.”

Professor Roddy points out the explosion in the global datasphere, and believes that the datasets have become “highly complex, heterogeneous, and filled with variables that interact in non-trivial ways,” often causing the data to become siloed. In response, he told Dr. Genkina, “Researchers are turning to sonification as a necessary tool to break through this data deluge and extract meaningful insights.”

Here are a few examples:

• Researchers are Georgia Tech’s School of Interactive Computing are using sonification and texture to convey information sea floor habitation in ways that blind and low vision people can more easily understands. Associate Professor Jessica Roberts explained: “Our job was to figure out how we can use sounds and touch to represent each of the four habitat types so our visitors can explore the ocean without being able to see it.”
• Researchers at the University of Kentucky are turning information about cells into music. Professor Luke Bradley recalls: “I was working on protein sequences and how they’re folded, and there’s patterns. And I would just kind of tap my pencil to the patterns of the amino acids, the different chemistry groups. I was wondering, could that be set to sound?” It turns out, they could, and it can convey useful information about the health of a cell. Professor Bradley says: “You can think of the cell as a symphony. And when everybody’s playing in tune and properly, it sounds great. But if you have one group that might have something off in tuning, it doesn’t sound good. And so that’s what’s happening by proxy, in the health of the cell.”
• Mark Temple, a professor at Western Sydney University, has been turning DNA into music for almost ten years. He now says: “My journey with DNA sonification illustrates how music and art can be powerful tools to communicate scientific concepts to a wide audience. It also underscores that creativity is alive and well in the sciences.” Others have followed in his footprints, such as using “genome music” to raise awareness of rare diseases.
• Researchers in Spain believe that sonification can take MRI data “to provide an intuitive and effective auditory representation of complex higher-order statistics,” which might allow for earlier detection of dementia, among other diagnostic applications. Similarly, Professor Roddy told Dr. Genkina about the Brain Stethoscope” project which converted an electroencephalogram (EEG) into sound, helping doctors more accurately detect seizures.
• There is a CosMonic sonification project aimed at broadening astronomy outreach using sonification “to create simple, acoustically accessible astronomical cases that are easy to understand.” In his interview, Professor Roddy cited how scientists translated gravitation waves into sounds; “you could literally hear the collision of two black holes that were billions of light-years away.” Solar wind data has also been converted into sounds.

Professor Roddy himself has used sonification to convert network data into useful audio, “clear, pleasant, voice-like sounds when the network was healthy, and shift into harsh, dissonant sounds when there are technical issues.”

With 6G, he predicts, “networks will be so complex that sonification will basically become necessary to make any sense of what’s going on… using sound as the primary tool to diagnose, navigate, and understand the massive, living data networks we have built.”

Maybe you like numbers. Maybe you need to read words. Maybe you learn best by seeing. Or maybe listening is how you perceive the world best. We live in a world of data, and that world is growing faster and more complex than we can grasp, so we’re going to need all the tools at our disposal to make sense of it. Sonification is one of those tools.