I built Kandinsky as a means to both automate and make more objectively precise the translation of visual data into musical material. Kandinsky takes input from an any image, video or webcam source and is operated by assigning a set of any 12 colours to any 12 notes. It is preloaded with the historic colour scales of Newton, Castel, Scriabin etc but any of the 12 cells can be assigned any chromatic note (automatically through a bank of keys and modes) or manually – all in real time. Similarly each cell can be assigned a custom colour by sampling the image, or automapped to the colours in the image. A reader – static or sent on any number of trajectories will check for a stored colour (at a controllable musical subdivision) and if found will trigger that note.
The prototype and ancestor of Kandinsky – the unpronounceable Synaesthesizer – demonstrates this process below. Since MIDI data is extremely flexible, its output can be used to play any virtual instruments (in turn or simultaneously) and adapted in streams playing for example piano and an electronic drum kit simultaneously.
Kandinsky nuances this earlier design in a few ways. Its tolerance to colour recognition can be adjusted, and so too its velocity based on the proximity of the found to ‘target colour’. A similar colour produces a quiet note, but the exact colour will be loud. In addition, Kandinsky can output continuous RGB data on 3 different channels allowing the smooth controlling of volumes, filters and any adjustable musical parameter, breaking away from the discrete onset approach to sonification. An image can be converted into a smoothly shaped texture rather than a series of distinct notes as demonstrated below.
In addition, Kandinsky introduces ‘colour buses’, allowing for example a specific groups of colour to be split independently to different instruments and through alternative note choices. This means for example that a harp might play all the colours while the strings play – literally – the blues. Each cell can also be independently transposed by a number of intervals, or the whole key changed collectively, or the colours shuffled between cells. These little tweaks somehow coalesced to bring the device from a translation tool to what felt like a very intuitive and fluid instrument, providing a looking glass into the colour-world. I include a highly informal presentation of the very moment this transition happened, when from a Monet painting emerged a suitably lush and tender music that was waiting to be discovered.
A series of Kandinsky sketches unpacks the process with immediate conversions of a wide range of images.
