Generate Sound Composition

Generate Sound Composition

Work created in 2025/6/21

This work is a cross-sensory digital composition that converts visual information into an auditory experience. It analyzes photos taken or images selected by the user and automatically generates a unique musical piece by converting their visual features into acoustic parameters. Combining modern computer vision technology and the Web Audio API (Tone.js), it explores the boundary between visual art and music.
At the core of the work is a transformation algorithm that quantifies visual elements such as color, brightness, contrast, and texture of images and maps them to acoustic parameters like pitch, rhythm, timbre, and spatial arrangement. The pixelated abstract composition displayed on the screen visualizes this transformation process and represents a reconstruction of the essential information extracted from the original image.
The minimal user interface eliminates operational complexity and maintains the purity of the experience. By providing only two simple input methods—a camera icon and a folder icon—and basic controls for play and stop, users can immerse themselves in the creative experience without technical barriers.
This work can be regarded as an attempt to artificially create synesthesia through digital technology. However, beyond mere technical implementation, it poses fundamental questions about the relationship between vision and hearing. While traditional music composition expresses the composer's intentions and emotions, in this work, the visual information of 'the other' in the form of images becomes the subject of music generation.
The pixelated abstract expression reflects the ontology of images in the digital age. Visual information, originally continuous and analog, is reconstructed as a collection of discrete data points and similarly transformed into discrete acoustic events. In this process, the context and meaning of the original image are lost, leaving only purely numerical and structural relationships.
The experience design of the work pursues a delicate balance between chance and necessity. Users actively select images, but the resulting music is unpredictable. This unpredictability is the core of the work, functioning as a device that defamiliarizes everyday visual experiences and invites users into a new auditory world.
Leveraging the characteristics of web technology, the design operates on a universal platform—the browser—without requiring special software or hardware, achieving accessibility and democratization of the art piece. This also represents a clear statement about the social role that digital art should play.
This work is an important experiment in the context of post-internet art, exploring new relationships between technology and sensory experience. Unlike traditional multimedia art, which aims for effects by juxtaposing multiple media, this work takes a more fundamental approach by completely converting one medium (visual) into another (auditory).
The real-time conversion from image analysis to music generation suggests new possibilities in the fields of AI art and generative art. While modeling the human perceptual system, it embodies the potential for perceptual expansion through technology by creating new sensory relationships that humans cannot experience through mechanical transformation processes.