IMAGINARY SONIC STATES
MUSIC
-
TECHNOLOGY
-
VISUAL ART
-
ENGINEERING
-
MUSIC - TECHNOLOGY - VISUAL ART - ENGINEERING -
We are proud to present Imaginary Sonic States, a piece produced by Nayoung Lee Ph.D. Nayoung used real time data, collected from sensors on a ship, to train a time series prediction Artificial Intelligence (AI) model. This model produced predictions of the sensors’ future data, which then Nayoung converted into sound using a granular synthesiser and manipulated into a house music track.
“I had never worked on a song using an AI model before. However, as I use time series prediction AI models, I often thought about what it would be like to convert industrial sensor signals into sounds.”
Imaginary Sonic States highlights the integration of emerging technologies with the world of music production as we go deeper into the digital age. This fusion reflects an emerging balance where human creativity and efforts produce data for both technological advancement and artistic expression. It reminds us of Massive Attack’s 1988 album Mezzanine, which was translated into genetic code. The digital audio of the album was first converted into binary code of 0s and 1s. Each unique pairs of binary digits were then assigned to one of the 4 different nucleotides of DNA: ‘00’ to adenine, ‘01’ to cytosine, ‘10’ to guanine and ‘11’ to thymine, creating 901065 strands of DNA. These were attached to glass nanoparticles, encased in a layer of silica, and stored in a small bottle of water! Although this storage method is more complex and expensive compared to traditional methods, it allows the use of a widely accessible technique called polymerase chain reaction (PCR) to make millions of copies of DNA sections rapidly and inexpensively.
Nayoung is not only a talented DJ and producer, but also a researcher and engineer, currently working in a South Korean steel manufacturer and specialising in process optimisation, machine learning (ML) and computing. She also recently completed her Ph.D in energy systems and flow assurance at the Department of Naval Architecture and Ocean Engineering, Seoul National University. Her main research applied process modelling, stochastic time series forecasting, and physics-based AI on sensor network of offshore and onboard process systems including carbon capture, utilisation and storage (CCUS).
“The Imaginary Sonic States has two meanings. One is imagining non-sound data as sound, and the other is the result obtained by predicting the future, that is, imagining the future.”
A deeper dive into time series prediction AI models and the data that inspired Imaginary Sonic States:
A time series prediction model is a type of machine learning model that predicts future values, based on sequences of data points collected sequentially over time, known as time series data. By analysing patterns and trends within these time-dependant sequences, these models are commonly used for stock price forecasting, demand prediction, and anomaly detection. Recently, they have also been widely used in the manufacturing sector. Predictions can be made using various algorithms, with or without AI.
AI models that utilise deep learning for predictions are called Recurrent Neural Networks (RNNs), which are designed to process sequential data by maintaining a "memory" of previous inputs. Among RNNs, Long Short-Term Memory (LSTM) and Transformer models are frequently used, especially in natural language processing and speech recognition, with applications in companies like Google, Amazon and Facebook.
The raw data used to produce Imaginary Sonic States included fuel temperature, pressure, speed and vibrations recorded from sensors inside a ship in operation in real time. These data were used to train the model to output predictive data for these sensors which were then manipulated into the track using a granular synthesiser.
Cover art for Imaginary Sonic States by Nayoung.
This is a manipulated image using the overlay of the time series prediction results.
We also asked Nayoung:
What exactly a granular synthesiser is and how did it influenced this piece?
A granular synthesiser is a method of sound synthesis that breaks sounds into pieces and reorders them.
I had never worked on a song using an AI model before. However, as I use time series prediction AI models, I often thought about what it would be like to convert industrial sensor signals into sounds. Then, when I received a work proposal from STEAMUL8, I thought I should try using the AI models I use in research in my songwriting. It was actually a very fun task. In fact, I had a lot of trouble making the data sound like music, but I solved it by using a granular synthesiser, which is often used in the noise music genre.
What does the title of your piece signify?
The future was predicted using data values that were not originally sounds, and this was converted into sonic state. Imaginary Sonic States has two meanings. One is imagining non-sound data as sound, and the other is the result obtained by predicting the future, that is, imagining the future.
Welcome NAYOUNG!
I am learning a lot through this experience - a good opportunity to think about the relationship between my music making and my life as an engineer…thank you so much!
