INFLeXions No. 4 - Transversal Fields of Experience (Dec. 2010)

Syn-aesthetics – Total Artwork or Difference Engine? Thinking Synthesis in Digital Audiovisuality by Anna Munster (University of New South Wales, Australia)
view PDF version

The Senses’ Last Riot

It’s a stinking hot day in mid-July in Venice, literally. The mosaic stench of refuse, algae and petrol fuelling the vaporettos rises visibly, up off the city’s tepid canals. I am here for the 52nd Biennale of the Arts, 2007, and, like many ‘art’ tourists, the parklands of the Giardinni, housing the participating nations’ pavilions and première artists, provide some air-conditioned respite. Here the footsore can disentangle that particular criss-crossing of sensory boundaries that unravels when navigating the Veneto. I head for the Russian pavilion: not only does it hint at an associated coldness but in publicity announcements I’ve already discovered that this year Russia’s theme is the relation between ‘the real world and technology’. Nothing colder. An eight-metre water tank greets me on entry, with a rhythm pulsating along its length. Wave is an installation, comprising a hydraulic pulsation travelling the length of the tank, driven by the screen-based breath of its artist, Alexander Pomomarev, emanating from a projection behind it. Suddenly, the sensation of cold, purified water is everywhere, gliding over my sweaty limbs. In another room Pomomarev’s Windshield Wipers, transposes mechanical motion into data, as actual windscreen wipers mounted against monitors wash away the displayed images. At the same time and relentlessly, each ‘view’ is replaced by an image of water, which is relentlessly replaced by a ‘view’, which is wiped away…

But it is not until I enter the immersive and darkened space of the screen-based installation housing the art collective AES+F’s Last Riot, that I really feel chilled.¹ Not chilled out, in the way that a mesmerising audiovisual sculptural performance at a festival or nightclub chills you into a different perceptual zone. Rather, I am chilled to the bone. The moving images in Last Riot fill three screens and 270 degrees of wall space with synthetically generated 3D backgrounds, objects and incredibly beautiful, young airbrushed models from a range of ethnic origins. Uncannily though every model looks the same. This vision is achieved via a fantastic combination of art direction, choreography and post-production imaging techniques blanching out any bodily difference. The models bend and contort across the computer-modelled landscape, which feels vaguely Nordic, possibly Siberian, and definitely post-apocalyptic. Cold, icy cold. The soundtrack, leaving no doubt as to the ethos meticulously invoked, is borrowed from Richard Wagner’s 1836 opera The Flying Dutchman.

The chill rising up through my bones comes from the orchestration of the aesthetic and political dimensions of this work, which mesh so brilliantly and evoke for me everything that is truly chilling about contemporary digital aesthetics. Here digital art’s cultural signals are cross-processed with a proto-fascist politics. This installation, so overwhelmingly immersive and epic, at once strips its audience of complex sensory engagement with its world. We are left to slouch against the wall and submit instead to its total spectacle. The world it evokes is one of violence without force. Its models are characters in battle with each other, using weapons borrowed from gaming; rockets perpetually launch in the background; the models loop through endless gestures of bloodless, yet relentless, conflict. This is anaesthesia – the nonsensing space of mainstream digital visual culture – a flat space that, as Last Riot intimates, is also synonymous with the contemporary politics of perpetual war.

Last Riot self-consciously plays with the notion of the Gesamtkunstwerk, in a parodic staging as an exemplar of Wagner’s ‘artwork of the future’ – the total artwork (Wagner, 1895). Its salute to the epic form, its aesthetics and display in the form of an immersive installation, merge the nineteenth century Wagnerian vision with contemporary mainstream digital audiovisual culture. For Wagner, the architectural space for the staging of future dramatic art should facilitate a complete exchange of the artist with the audience, such that one becomes (in the sense of changing places ontologically) a living, breathing instantiation of the other:

Such praise in favour of Romantic dramaturgy finds its full embodiment in the ‘nineties’ of the twentieth century, in the hyped up digital rendering of virtual reality.² Indeed Wagner has been very much at home in emergent digital culture and has been invoked genealogically in order to theorize the origins and future of multimedia in Randall Packer and Ken Jordan’s Multimedia: From Wagner to Virtual Reality (2001). Multimedia, too, would offer the dream of total experience (albeit upon the rather constrained stage of the computer desktop), by synthesizing the various elements of audio, visual, text and graphic files together with interactive gesture, all via the orchestration of digital code. Wagner becomes the genesis of a particular imaginary for the way digital code will operate, not only aesthetically but also as digital worldview. The total Wagnerian or multimedia artwork involves a practice of fusing all aspects of the aesthetic (or all sensory modalities) under the aegis of a kind of ‘meta-form’. As Packer and Jordan see it, opera was Wagner’s totalizing meta-form; the meta-form of the digital is the interactive interface in which call-response gestures literally orchestrate the fusing of all sensorial varying media (2001: xvii-iii).

But this connection of the total artwork to the meta-form of interactivity gives the operations of computation a particular status. It accomplishes more than simply giving the digital interface a binding or synthesising function. It sequesters the process of binding the varying elements. Interfacial synthesis becomes the pre-given ground upon which multi-mediated sound, image, text and so on magically melt, one into the other. The role of the computer in the Wagner-multimedia genealogy is to stage the immediate and transparent passing of all transcoded sensory elements into each other such that synthesis miraculously occurs. The coded environment breathes as the contemporary incarnation of a whole that has transcended its own senses.

In Last Riot, the inclusion of Wagner plays out as a critically reflexive tactic aimed squarely at the politics of codification and its concomitant mobilization of affect from Romantic to digital aesthetics. It marks out the fate of such a genealogical relationship rather than simply legitimating the traversal from Wagner to the digital spectacle as the inevitable march of art history. The total spectacle of the installation deliberately retraces the passage that aligns this digital aesthetics with the chilling totality that such synthesis performs. Wagner’s fusion of the arts allowed a passing of the heterogeneous into the homogeneous via the elevation of an epic musical form – tragic opera – to meta-form, becoming the vital breath that united elements, senses and functions. Deleuze and Guattari note the Romantic passage from the aesthetic to the political in the following way:

In this totalizing experience, small spaces for intensive thought and its extensive action are subsumed rather than being allowed to play themselves out. So, too, while offering relief from the heat, the digital spectacle of Last Riot confirms that the escape from the multimedia interface in digital aesthetics has only fused the varying sensory elements of sound and colour into a space, which freezes all movement.

At the heart of the macro-passage from the aesthetic to the political that Last Riot performs, there is a micro-passage from experience-affect to action-thought. The synthetic aesthetic of Last Riot is recursively produced across a number of planes. The most obvious can be found in the digital scenography, compositing beautiful live models into computationally synthetic backgrounds, blending all into a synthetic world. There is also a fusing of variable aesthetics through bringing together the digital image and palette with the Romantic compositions of Wagner. Synthesis is likewise performed across the senses via the particular spatialization of the installation in the Russian pavilion, which, in its screen tryptych, lands the public inside a hallowed architecture, somewhere between altar and IMAX. A body’s sense of this space in relation to all these syntheses is of being brought in to (immersion) the already synthesised; of inhabiting a pre-given mesh of digital and lens-based media, colour palette and epic audio. There’s little room to move.

What kind of conceptual space, what conceiving of space for thought, can arise in the middle of engaging with such synthetic art? How might the multi-sensorial experience become something other, where the elements of sound, colour, gesture and so forth are given lines that sometimes exchange, meld and yet nonetheless depart from each other in order to open up new worlds of the felt? And in the feeling, of thought? How might synthesizing continue to open up a space in which thought runs off along these lines as well, becoming eccentric, wild and fleeing? Thought, that is, about the exchange of sensations but also cognition generated in the very action of sensory exchange. Must the synthesis of heterogeneous elements – an operatic overture, cold colour palettes, a dark overwhelming space – collapse into a totalized digital world? A world, which is an immersive stage, upon which not the future artwork but the future of warfare is instead rehearsed and performed? What room in this to conceive-imagine how aesthetics and the cosmologies, toward which it gestures, might synthesize differently?

The question, and it is one engaging the efforts of many contemporary artists and thinkers, is how to affect that passage or, put differently, how to pass affectively between the aesthetic and the political so as to maintain the passage’s movement rather than collapse the movement of one into the other. Wagnerian affect is soaring, immediate and complete: a total passing of art into the public and of public life into the aesthetic. For Wagner this must occur through dramatic music – that form in which music connects theatre, language, architecture, dance and visuality. This total offering up of the arts in the dramaturgical Gesamtkunstwerk was to provide the opportunity for audiences to connect directly to all senses at once and hence, in Wagner’s view, to nature (1895: 188–193). Music in Wagnerian opera was intended to provide a mobilizing force for unification and it is in this that a particular kind of meta-synthesis is found. But the question of passing as a process of relation, of something passing into something else is passed over here in favour of immersion.

Last Riot consciously details the ways in which the passages between compositional elements amidst the arts and between the artwork and public space – the space of bodies in public congregating, moving, relating – are subsumed by a post-Wagnerian meta-synthesis. This is a syn-aesthetics of transparency, senseless flows, seamlessness. But Last Riot does not offer us any alternate passages for other aesthetico-political movement(s). It is an apocalyptic, epic hymn to the digital cooling of the senses. It leaves us cold as each sensory bloc is poured into another effortlessly; as we stand still, and without moving us. It is the senses’ last riot.

The Return of Synaesthesia

Or is it? Something riotous has been going on in the digital and electronic audiovisual domain over the past decade. Carsten Nicolai’s work across sound, vision, signal, noise and the ecologies of these provides different working methods and processes for synthesis and syn-aesthetics. In telefunken (2000), digital ‘signal’ criss-crosses media players: instead of an image signal from a video player, a CD player is hooked up to a television monitor. Audio tracks playing on a CD in a gallery space visually generate the movement, pulse and pace of white lines across television monitors. Nicolai calls this connection of CD to TV ‘erroneous’, giving us an insight into something else at work in digital syn-aesthetics (2002:78). For Nicolai, digital signal does not simply flow from one machine to another. Rather, the idea is to see what happens if an error in connectivity occurs. This is not simply the error as it appears in avant-garde art making in, for example, the Dadaist movement. It is the error as a fundamental problem encountered in the digital milieu – a milieu comprised of the forces, patterns and processes of signal generated in and out of code, passing in and out of electronic materialities. The error launches a bank of signal flows, which mesh and self-organize resolving themselves compositionally in their resultant auto and allopoetic processes. A digital ecology temporarily forms – the installation telefunken, consisting of cross-processed audio (CD) and image (televisual) signal that rests upon the mistaken synthetic conjunction of media players. Here synthesis is not so much a unitary ground as a processual meshwork.

In the work of Nicolai, Robin Fox, Ryoichi Kurakawa and others, in festivals such as the yearly Cimatics audiovisual extravaganza, in VJing and in the curation of digital art into exhibitions such as See this Sound, in Linz, the senses are getting a work around.³ Weaving its way through these various events and performances is a concept that intermittently pops up within aesthetics: synaesthesia. Thought about the synaesthetic has traversed both artistic composition and scientific approaches to human perception – artists and writers such as Rimbaud, Rimsky-Korsakov, Scriabin, Messaien and so on have been labeled as possible or actual neuro-synaesthetes (van Campen, 1997; Harrison and Baron-Cohen, 1997). Twentieth century artists such as Kandinsky and Cardew have suggested a necessary conjunction particularly of the visual and the sonic; that art made in one necessarily calls up the other sensory modality.

But there is a more general discussion of synaesthesia as a phenomenon of contemporary audiovisual spaces, as if, somehow, the digital and perception have become necessary compossibles. This hinges on the idea of a fundamentally connective analogue sometimes implicit, sometimes explicit between the neural and the digital. Although this connection opens up a number of interesting possibilities around the concept of plasticity, it also suggests a slide into the reified model of wiring. Prominent neurological conceptions of the synaesthetic and models of the movement and flows of data through software and information architecture such as the database converge. For example, Christopher Cox’s analysis of Nicolai’s telefunken draws together synaesthetic experience with digital/electronic signal:

Yet it is not just that the synaesthetic and the digital converge here. The process of audiovisual synthesis taking place in the installation is drawn into an epistemological and ontological space of the Kantian ‘a priori’. This seems to me to be architecturally problematic – a problem for the space digital aesthetics wants to compose. Accorded the status of a structural architecture, which would condition the space of perception-experience, synthesis occupies a similar place as the Gesamtkunstwerk. Must it? More on this later for indeed there may just be another way to think the ‘meta’ in meta-synthesis. In the meantime, it is worthwhile staying with this twisting of the neural and the digital together via more recent work on synaesthesia in both the perceptual (neurological) and aesthetic (cross-processing of digital signal in contemporary audiovisual art).

When it investigates the cause of actual synaesthetic experience (that is, involuntary consistent perceptions of colours with sounds and so on), neuroscience deploys concepts such as ‘cross-wiring’ in the brain (Ramachandran and Hubbard, 2001: 8–12). Signal carried through wires becomes the founding architecture upon which sensory experience is etched. As if micro-processes and micro-processing could be generated from similar electric-architectural unities.

Contemporary research in the neurosciences generally accepts synaesthesia as an anatomically based phenomenon of human perception located in a neurobiological architecture. Although there is variation in the ways in which synaesthesia manifests in perception – coloured-hearing, coloured-graphism, visual-smelling and so forth – neuroscientists agree that synaesthesia involves an involuntary and repeated invocation of one sensory modality by another in response to a perceptual stimulus (Cytowic, 1997: 23; Ramachandran and Hubbard, 2001: 4). Neurological research into synaesthesia can be ’sorted’ into two prevailing approaches: on the one hand, the idea that ordinary neural ‘pruning’ in human development fails to occur leaving in place an originary synaesthetic brain; and on the other, the idea that different sensory modalities and their functions are located in separated areas or modules of the brain, which are ‘cross-activated’ in synaesthetes.

There are two main competing neurological hypotheses for synaesthesia: Cross-Modal Transfer (CMT) and Neonatal Synaesthesia (NS). One derives from the other but makes more radical neurological claims. The CMT hypothesis is slightly older and was developed as a result of work by Meltzoff and Borton who posited that infants have the ability to recognize objects in more than one sensory modality (1979). So, for example, something that a baby has only touched can nonetheless be visually recognised by it. The process involved in this infantile experience involves the transfer of sensory ‘data’ across modes – haptic to visual. Visual recognition is here understood as something that must exist prior to intermodal processes. The process is possible because of the infantile brain’s cognitive ability to abstract representations from objects. It is this capacity for abstraction that points to where joining – the ’syn’ – of all the sensory modalities occurs. The CMT hypothesis rests upon the proposition that synaesthesia is primarily a function of inherent cognitive capacities for abstraction and representation in the human brain.

The NS hypothesis – more recent and supported by neurologists such as Baron-Cohen (1997), Maurer (1997) and Ramachandran and Hubbard (2001) – asserts, instead, that synaesthesia is a primary and originary state of infantile perception rather than cognition. Up until about the age of 4 months, the ’state’ of organization of the infant’s brain does not differentiate sensory input. Instead, the NS hypothesis claims, the neural architecture is cross-modal or ‘cross-wired’. Neurological development from this period on, then, initiates the process of ‘normal’ differentiation into separate sensory modalities. Some human brains do not fully differentiate, leaving original ‘cross-modal’ pathways active. In these cases, adult synaesthesia will persist and a person may experience the typical ’symptoms’ of involuntary call-up of colours in conjunction with hearing certain sounds. The NS hypothesis rests upon the notion that an originary totality of perception can be sought in an undifferentiated neonatal neurobiological architecture.

In the CMT model, cognitive abstraction makes sensory cross-modality possible; in the NS model complete nondifferentiation supports movement between sensory pathways. There is a fundamental symmetry, then, between the competing hypotheses even though they differ as to the developmental emergence and neuro-systemic location (cognition vs. perception) of synaesthesia. In other words, at the heart of both understandings of the synaesthetic lies the ‘ground’ of an originary unity of the individual brain - either fundamentally perceptual or cognitive. But it is precisely the unity that requires explanation. For in fact what we are seeking is an understanding precisely of the sensory, perceptual or cognitive activities of unification (or rather of joining) by investigating synaesthesia in the first place.

This brings us to the problems posed by classical models of ontology, which, as Gilbert Simondon pointed out, attempt to explain processes via their outcomes (1992). Hence the ‘unity’ of synaesthesia (emergent outcome) becomes an explanation for how it is that the senses join or cross. Synaesthesia, however, is processual –the conjoining of sound and colour is both conjunctive and an individuated percept. The question at hand is how to think perception as neither structure embedded in the mind nor the end product, the outcome, of a set of activities:

Perception in a living being, following Simondon, might be thought as a mediating process that is both a bringing together but, in this very conjoining, individuates. Perception conjoins the human’s sensory potentialities with its milieu, brings to it a world to which it is already parametrically tuned. Humans, for example, cannot see ‘infrared’ in the colour spectrum and can usually hear between 20 and 20,000 Hz. Perception thus depends upon a ‘pre-individuated’ set of human potentialities and limitations. But this bringing-into-relation of the human organism with a milieu is itself already relational and processual. This bringing-into-relation is the very work of perception as it actively adapts the relational architecture of the sensory-motor apparatus. This apparatus is less a machine or even system and more an internal resonance – an infolding of resonances that have meta-stabilized. In a human (and many other living organisms) this resonance is the sensorium as individuating continuum – a modulating relationality of all 5, 6, 7 or more of the senses to each other. This is what Brian Massumi has called the senses’ already existing ‘perfusion’, the intersensing or co-attraction of the senses as ‘almost’ there, a virtuality (2002: 156–7). This must always already have been the senses’ relational mode of existing in order for them to individuate (and which was itself already an individuation and the ‘pre-individual’ of the senses as plurality).

Now, this relational sensory perception-architecture does not ‘solve’ the neurological question of synaesthesia in terms of accounting for its neurobiological causes.⁴ But it does provide a kind of plasticity for thinking synaesthesia as process that both coheres and differentiates and which takes us away from the metaphor of ‘hard wiring’ that traverses both the neural and digital accounts of the synaesthetic. Perhaps the issue is the kind of architectures we use to think and make neurological and digital relations. Synaesthesia as both neurological condition and as digital aesthetic experience, then, needs to be unthought as a unified architecture of passages or pathways for signal flow. Perhaps it might then be thought anew as a relational architecture as Massumi has suggested (2002: 191–2). Here synthesis of image and sound, such as happens in the midst of a particularly freewheeling VJing set, composes an emergent syn-aesthetics recursively drawing upon and refreshing a field of varied continuities of lines of sensory expression that fan out between artist, art and audience:

As VJ, Mark Amerika alludes here to both mesh and potentiality as simultaneous pauses in the synaesthesia of contemporary audiovisual digital experience. If we are to deploy a digital syn-aesthetics, we must be willing to allow both the ‘syn’ and the ‘aesthesia’ resonant activities and architectures. But this may also necessitate giving up the idea of the artwork as total experience, where it seems to have shifted since losing its objecthood.⁵ New subjectivations must follow from this – not only for ‘the artist’, ‘the viewer’ but also for ‘art’. I will return to this when I attempt to understand current artistic experiments in cross-processing digital signal syn-aesthetically. What I want to suggest is that similarly we cannot approach the digital as exemplary of synaesthetic experience, if by this we mean that interfacing with digital art presents us with a totality of sensory engagement.

Synthesizers, Control and the Relational

Deleuze and Guattari posit the difference between Romantic and modern (contemporary) thought as a difference in approach to the question of synthesis (1988: 342–3). Romantic philosophy requires a formalizing synthetic identity for thought, which makes matter intelligible across all difference – organizing it as a continuity – the a priori synthesis. Modern/contemporary philosophy should elaborate thought’s materiality in order to harness forces that are not in themselves thinkable. Thought brought into relation by thinking, and by the thinking of thought with what is outside of itself; likewise for modern/contemporary visual sonic art/music. Visual art passes through the image in order to render other forces, the nonvisual, visible, perhaps, for example, as can be felt in the fleshy sounds of Francis Bacon’s painting. Sound art and modern music (Boulez, Cage and so on) connect the sonorous with its materiality and cosmology, which, according to Deleuze and Guattari are its nonsonorous molecularity. These are forces such as electromagnetism; forces that subtend sound but are not actually sonorous themselves.

Interestingly enough, there is both a machine and a process in the sonic realm, which, for Deleuze and Guattari, precisely achieves this non-Romantic mode of synthesis – the synthesizer:

It is the parametric operation of the synthesizer that uses one force – for example, the force of a wave being pushed through different ranges of voltage frequency – against the force of another element of sonic matter, which makes the synthesizer a relational apparatus. Synthesis thus shifts away from its function as the unifying ground it held in, for example, Kant’s judgements, toward a shifting terrain. This terrain is always in the process of being diagrammed as sonic and nonsonic elements consistently relate to each other in the making of what Deleuze and Guattari describe as a field of ‘generalized chromaticism’ (1988: 95).

According to Deleuze and Guattari, Kant’s synthesis occurs as a cognitive response to sensory information. It is the organizing capacity of cognition, which accords this sensory information ‘form’ as mental representation. But it would be a mistake, they suggest, to similarly accord the synthesizer – that mid-twentieth century musical invention – such an a priori function. It does not organize and re-present its various data inputs from some a priori space of ‘code’. Rather it allows the differing inputs (sensory elements) to form relationally via their various sonorous forces; that is, the attack and decay of a note or sequence of notes might be isolated, tweaked and used recursively or as a force upon another input. The synthesizer is not, then, a meta-apparatus, in the sense of being a transcendent organizer of sonic elements into form (either reproducing an individual sound or formally producing ‘music’). Rather the synthesizer is a new kind of instrumental assemblage: one comprised of the dynamics between sonorous and nonsonorous elements; sonic and infra-sonic forces. It is born out of an era of experimentation with sound as a materiality riven through with cosmic forces.

The sticking point is the digital. Deleuze and Guattari’s ‘discovery’ of the asignifying semiosis of the synthesizer arrives right on the cusp of the release of the digital synthesizer into the music market. Quite apart from where it ends up, unsurprisingly, in the New Romantic melodies of Duran Duran, Ultravox and Boy George during the 1980s, the synthesizer becomes an instrument for solving what is both a technical and political problem. This is already occurring during the 1960s but is not quite apparent at the level of musical and aesthetic production during that period. Most synthesizers in the 1960s and even up until the late 1970s are still large, customized and sit in a number of avant-garde recording studios such as those of the Cologne Recording Studio used by composers such as Stockhausen.⁶ Indeed, it is quite possible that by solving the problem in the specific way that it was solved, digital synthesis is able to rejoin its old neighbour, the synthetic a priori.

Modular systems dealing with the organization of electronic systems were on the rise during the 1960s. Together with hardware such as transistor devices, entire systems for electronic sound synthesis could be packaged in smaller more transportable units that were customized but mutually compatible (Manning, 2004: 101–5). Filters and oscillators could be put together to synthesize the one sound simultaneously. Previously sound synthesizers featured sliders and knobs where the electronic signal was processed at one point, synthesized and then passed down the signal line to the next point of synthesis. So, first oscillated, then filtered for example. However now the system could be designed so that an external set of voltage characteristics could control the signal outside its passing from one process to another. This allowed a secondary set of interconnections to produce the control information for any of the individual modules, creating the beginnings of a meta-data or ‘control’ layer of data, which acted upon the signal allowing it to be processed in a way specific to the forces of those actions (that is, the particular set of voltage characteristics).

So far we have, in effect, what passes as Deleuze and Guattari’s nonsonorous forces (voltage) affecting and modulating the sonic. And indeed the resulting sonority perhaps rendering audible something that is inaudible: changes in electrical current as it is conducted through the materiality of the transistor. Notably, voltage controlled synthesizers are analogue in terms of the signal and in terms of the ways in which an electronic signal acts upon another; that is continuously variable. But there are elements of the design and of the hardware that also make this kind of synthesizer a precursor to digital instruments – specifically its use of modular design and of transistor devices. Importantly, too, this very design of a ‘control line’ for passing signal through the system is cybernetic insofar as it sets up an architecture for one pass or feed of information to pass or feed into another layer of information. But it is more than cybernetic once the ‘control layer’ becomes digital, effectively allowing many more operations upon operations and, what is more, allowing these operations to shift from the status of ‘force’ to that of ‘execution’. In the digital synthesizer, this ‘control layer’ is not itself a force but rather becomes removed from the actions of forces, from forces’ affectivities (the way forces affect and are affected). In becoming executable (executive, perhaps) it turns algorithmic.

We could of course imagine algorithms as something different from operations that execute functions of code and as new condensations of forces. As Adrian MacKenzie has suggested, algorithms are complex condensations of code and sociality in which what is ‘ordered’ is already subjected to a certain epistemic setting into place, an ‘ordering’ of things by their place, by their time (2006: 44).

We should, then, think carefully about the interrelations of the digital synthesizer’s architecture, its relation to its own analogue voltage-controlled predecessor and to its milieu of information theory and culture. Specifically, it is to the particular culture of military research and development in digital computation that holds the key as to what happens to synthesis in the digital synthesizer. Paul Edwards has carefully outlined a setting in place of a triangle of command, communication and control governing the socio-technical ensemble in the emergence of digital computation during the post- WWII period of American military research. Digitality itself is not an obvious or necessary process here but is sought after because it delivers a certain militarily ordered zeitgeist that he describes as a rationally controlled ‘closed world’ (1996: 43-73). All elements of computational hardware, software, interface and system design come to us out of this socio-technical ensemble that has been and continues to be military research into information. Including the digital synthesizer. This is not to say that the analogue synthesizer’s components are not, too, adjuncts of what was once military electronic research. But the crucial difference lies with an architecture that re-places the force-materiality relation, still possible even in the voltage controlled synthesizer, with that of a form-matter relation, in which instructions are to be carried out upon sounds. In which sonorous matter is synthesized a priori out of the control line code.

Cross-processing Signal or the Analogue Compositionality of Digital Synthesis

Ryoichi Kurokawa’s digital cross-processing audiovisual sculptures and performances crop up in festivals, clubs, art galleries and music video. He is a VJ but one who is less interested in image for music than in experimenting with what we might call relational sculpture. The term ‘relational sculpture’ is my paraphrasing of the term ‘animated sculpture’, which can be found in Erin Manning’s analysis of David Spriggs’ work (2009: 143–50). Manning suggests that these ‘sculptural objects’ draw us in to the ‘seeing’ of the force of visual perception as we move around them, expecting, but not quite making out, bodily forms within their glass cases. The sculpture itself is not animated; rather it is the relation that the relational sculpture-viewer assemblage triggers between visual perception and movement that comprises the animation. An animation of the virtual in/of perception. Similar, perhaps, to what Olafur Elliason has said in relation to the aesthetic problematic that animates his work – the seeing of seeing or the sensing of sensing (2004).

In Kurokawa’s work questions of relationality are equally foregrounded. Along with a number of other artists who VJ or who produce audio visual performances where audio signals initiate, trigger and deform visuals in real time (and vice versa), Kurokawa’s work is frequently described in synaesthetic terms (Neissen, 2006). While operating as separate modes, the aural and visual aspects of the work appear to persistently elicit each other so as to produce a ‘synthesis’ of visual listening or aural visuality. In a video Kurokawa directed for electronic musician ditch’s track ‘mysterious hoze’ we sense this solicitation.⁷ But it is not one in which audio and visual modalities dissolve in an apotheosis of transparency. Instead the emphasis on the choreographic force and relation of audio to video signal creates an animated sculpture unfolding temporally. As if an electronic materiality, a cross-woven signal arising out of the interplay of sonic and visual forces, were also giving us a time of the electronic event. A kind of time-image digital signal – a ‘heautonomous’ cross-processing.⁸

Kurokawa initially builds a synchronicity between the techno beat of the soundtrack and the rhythms of the drawn blobs and shapes that pro- and contract in the screen space. The visuals seem to directly emerge and take form from the persistent beat of the dance track. But one minute and forty seconds into the sound, the visual drops off and out of time with the beat to become a quivering and tentative ring. Momentarily it lacks any pulsation driving its form-ation. We shift gears away from the synchronous unity of audio and visual modes to their disjunction, the departure of their edges from each other. Our aural attention shifts elsewhere within the driving techno-unity of the track to hear the understated entry of a jazz-inflected guitar riff. Would we hear this variation as anything other than simply melodic if an electronic heautonomous synaesthetics had not entered the mix? Perhaps an informed listener would be able to discern the multifarious universes of reference that always inflect contemporary electronic dance music. But what Kurokawa brings visually to the sonic pulse here is a departure from the inevitability of beat, which might dominate if the visuals simply reinforced the rhythm. Instead the audio signal’s intervals open up by generating a flickering relation with the image. The ear is visually oriented towards the sonic differentiation held virtually by the music.⁹

Kurokawa has passed between the synthesis of audio and visuals to open up the conjunctive potential of the electronic. No longer is ‘mysterious hoze’ only a driving dance track but one comprised of other universes of reference such as jazz (which of course dance music is but which can become drowned out by a dominant beat). Visual and aural are no longer synchronously mapped but become intensive relational universes mapping and unmapping each other. In Kurokawa’s live audiovisual performance/sculptures, too, there is a building together and a falling apart of sonic and visual intensities. Rather than arising from the pre-given of synaesthetic signal, what is worked at is a sounding out of the molecular consistency at which sensory modalities might become conjunctive.

Cross-signal processing of digital signal – for example, the use of digital sonic signal to trigger, produce and modulate transformations and formations of visual signal in practices such as VJing – is an area that has already attracted some attention as an example of a different kind of syn-aesthetics. Mitchell Whitelaw has argued that the transcoding of sound and image in the work of contemporary Australian artists Robyn Fox and Andrew Gadow can be understood in terms of cross-modal binding (Whitelaw, 2008: 259–276). Here the sound-image produced in these digital audiovisual environments might be thought as a cross-modal ’object’, which both points to the underlying digital signal and to a domain of correlation between modalities. Correlations (co-relations) Whitelaw suggests are part and parcel of perceptual experience – edges and limits in a given perception that suddenly make it shift from sensation to meaning. Sher Doruff has also suggested that neural synaesthesia does not need to function as ground for the digital and, we can add, nor should ‘signal’ across wires legitimate the neurology of synaesthesia (2007). Instead, she puts forward the idea of a transdisciplinary synaesthetic practice. A practice of inter-composing bodies, signals, machines where sensory modalities are not the starting points of relation and fusion is not the necessary outcome of their co-mingling. Instead, an image-sound sensed is a contraction of the practice, emerging out of the resonances set off by digital aesthetic generative architectures for sound, gesture, proprioception, image, electronic signal etc. Mark Amerika puts it another way, suggesting that VJing as a lifestyle practice of writing the image into existence involves processing (or transferring) the energies of sensation and perception before we cognitively organize them:

Interestingly enough the programming environments for much cross-signal audiovisual processing in software such as MAX/MSPJitter deploys a ‘patch bay’ diagram in which signal flow is directed in and out of processes such as filtering.¹⁰ A number of VJing packages such as GridPro alternatively use a kind of array interface in which signals are ‘sent’ places via circuitous routes in order to interact with each other. So although there are plenty of algorithms working away crunching the signal and executing effects what seems to have been left behind in the cross-processing environment is the architecture of the control line. This is not to say that there is not a lot of data instructing other data what to do. Is this the re-appearance of a meta-form in the guise of meta-data?

Perhaps…and yet meta-data is also simply data about data.¹¹ Or, put another way, data exerting a singularly informatic force upon other data – in-forming data about data. This ‘other data’ can likewise become meta-data and so on. In fact, this does happen in a cross-signal programming environment when a number of parameters (or ‘patches’, which are small and discrete code modules) all wrangle for their place in a sequence of programming events. It is often not clear in a live audiovisual cross-signal processing situation what signal, what data is telling what other data what to do. Setting up a number of sequential patches can nonetheless result in recursions that cause or stall the working together of data, resulting in ‘erroneous’ or unexpected interactions and the invention of new image-images, image-sounds, sound-sounds and sound-images. Signal’s micro-movement becomes compositional and not necessarily at the hands of the subject position of the composer/artist. From an array of micro-passes, signal flutters, stutters and modifies signal. As Whitelaw suggests, signal in cross-processed audiovisual aesthetic ‘objects’ is not a case of simple transmission of information from A to B:

It seems, then, that in these types of signal cross-processing events we revisit the analogue voltage-controlled synthesizer where continuous variables or flux patterns between voltage and sonorities contract into emergent sonic sensations. But we have shifted design away from a separate control line –the voltage-controlled synthesizer’s command-control cybernetic heritage. Instead the ways in which audiovisual artists and VJs are using in digital cross-processing, modules are constantly shifting around, never acquiring ground, in fact. In cross-signal processing audiovisual events, especially in live and somewhat aleatory circumstances, digital synthesis loses its tendency toward the synthetic a priori. Sensation that finds lines of expression through cross-signal processing is no longer causal nor is it a fixed phenomenon. Rather it becomes visual sonification, sonic visualization, diagramming a resonating, moving architecture. Not structural but relational. Not synthesized but conjunctive. Something that builds rather than is built. A digital syn-aesthetics finding its compositionality in analogue mode.

Notes

1 For a full description and set of images from the work see, AES+F’s website at http://www.aes-group.org/last_riot.asp

2 See for examples articles such as Meredith Bricken’s ‘Virtual Reality: No interface to Design’, Cyberspace: First Steps M. Benedikt ed, Cambridge, Massachussetts: MIT Press, 1991, pp. 363–382. This kind of article espousing VR utopianism emphasised the experiential transparency between computer- generated environments and their audiences, especially of a kind of exchange between onscreen actors and avatars and interactants. The rhetoric of presence likewise dominated the discussion of these kinds of simulations and the interaction with users or audiences.

3 The Cimatics festival has been held for 7 years and is supported by Cimatics Platform, an international organization based in Brussels supporting and exhibiting audiovisual art. It started as a VJing Festival in 2003 but has expanded to include an array of auidovisual sculptural and kinetic events, masterclasses and performances. Its website can be accessed at: http://www.cimatics.com. See this Sound is an exhibition, which historically examines the interrelations of sound and image via electronic and digital media arts. It examines these from
Anna Munster “Syn-aesthetics – total artwork or difference engine? Thinking 88 synthesis in digital audiovisuality.”
the varying viewpoints of media, perception and culture and from a range of theoretical frameworks in these areas. It ran from August 28, 2009 to January 10, 2010 at the Lentos Museum in Linz, Austria.

4 It should be clear that the aim of this essay is not to provide an explanation for synaesthesia as a neurological phenomenon. However, this does raise the issue of why it is that synaesthesia as thought neurologically requires a causal explanation as such. If, for example, an ontogentic approach were to be adopted by neurologists (following Simondon), synaesthesia might simply sit within a spectrum or continuum of human perception as a potential mode through which perception individuates. Synaesthesia would already be part of the ontogenetic field of human perception; one of its potentialities. To a certain extent, the concept of neuroplasticity (ie that the brain is in a constant relation of neurobiologcal formation with its milieu), which is now becoming more accepted within the neurosciences, holds out the possibility that a more ontogenetically inclined mode of thinking might permeate this field. If the brain is process rather than entity – suggested by the neuroplastic notion that neurobiological change can and does consistently occur – then this might also suggest that perception too is no longer an ‘outcome’ caused by certain wirings of the neural pathways. For a good and accessible overview of neuroplasticity, see Norman Doidge, The Brain That Changes Itself: Stories of Personal Triumph from the Frontiers of Brain Science, New York: Penguin, 2007.

5 I am here referring to Western art theoretical debates of late modernism in which Michael Fried positioned Minimalism as a new art ‘position’ to break with both representationalist and modernist/medium-specific concerns especially as evidenced in mid-twentieth century painting. For Fried, Minimalist sculpture, in particular, was concerned with the wholeness and indivisibility of the object (See Michael Fried ‘Art and Objecthood’, Art and Objecthood: Essays and Reviews, Chicago: University of Chicago Press, 1998, 148–172). In turn, this issue about the objecthood of the Minimalist work of art was also countered and debated by the equally strong Conceptual Art movement of the 1950s and 1960s. Lucy Lippard’s Six Years: The Dematerialization of the Art Object (Berkley and LA: University of California Press, 1997) remains a key text in this debate, where she politicizes the assertion of objecthood in Minimalism and (white, middle class, chiefly male) American art criticism of the period.

However, the debate moved into somewhat different territory during the 1990s with the emergence of both ‘environments’ that were art works, such as virtual reality art, the dominance of video installation at biennales and festivals and the art criticism of Nicolas Bourriard’s ‘relational aesthetics’ (See Nicolas Bourriard, Relational Aesthetics, Paris: Les Presse Du Reel, 1998). Bourriard emphasizes notions such as participation and interaction as key aspects of twenty-first century art: neither object nor concept is crucial, instead the ‘experience’ counts.
Anna Munster “Syn-aesthetics – total artwork or difference engine? Thinking 89 synthesis in digital audiovisuality.” In many ways, it is this aspect of the digital environment that returns the digital artwork to Wagner’s Gesamkunstwerk.

6 The Studio for Electronic Music was founded by Hans Hartmann, the director of West German Radio in Cologne in 1953. Members of the Groupe pour Musique Concrète, which included Stockhausen, used it to experiment with electronic sound using various analogue and early digital synthesisers. See, Karlheinz Stockhausen, ‘Electronic and Instrumental Music’ Audio Culture: Readings in Modern Music, C. Cox and D. Warner eds, New York: Continuum, 2004.
Deleuze briefly writes about the difference between analogue and digital synthesizers, arguing that whereas analogue synthesizers are ‘modular’, digital synthesizers are ‘integral’ (See Gilles Deleuze, Francis Bacon: The Logic of Sensation London: Continuum, 2003, 116). What is integral about the digital synthesizer is the way in which it integrates all operations upon sound and by sound, making it pass through binary codification. I do not think Deleuze’s distinction between the two kinds of synthesizers actually holds historically as it is the analogue voltage controlled synthesizer that introduces a ‘control’ line operation upon its modules. Likewise, digital synthesizers use a modular design. It is the case that a cybernetic model of codification is at work in both the analogue voltage controlled synthesizers and early digital synthesizers. This is what I have tried to indicate in this essay. Deleuze further argues in this text that it is modulation (which can be seen in the example of the subtractive synthesis of frequencies in analogue synthesizers) that characterizes the analogue. It should however be noted that in a much later text, Deleuze argues that modulation is characteristic of control societies, which characteristically are also codified digitally. I think he later sees that it is not the digital nor analogue per se that are at stake but the question of the command-control model that is immanent to the logic of first- order cybernetics (See Gilles Deleuze, ‘Postscript on the Societies of Control’ OCTOBER 59, Winter 1992: 3-7). My thanks to Troy Rhoades for directing me to Deleuze’s remarks on analogue and digital synthesizers.

7 The music video ‘mysterious hoze’ by ditch (Rioychi Kurokawa, dir. 2007) can be accessed on YouTube at: http://www.youtube.com/user/opdisc#p/a/u/1/_eqBT6YbXBs.

8 The term ‘heautonomy’ is borrowed from Deleuze’s work on the time-image in avant-garde cinema, in which he uses it to refer to the disjunction achieved between the acoustic and the visual in the films, for example, of Marguerite Duras (Gilles Deleuze, Cinema 2: The time-image, London: Continuum, 2005: 247ff). Duras’ cuts between sound and image deterritorialize the self-identity of the sonic and the optical, which in classical and narrative-based cinema tends towards the affirmation of a wholeness between the two. In VJing or music videos created by electronic audio-visual artists such as Kurokawa, the Anna Munster “Syn-aesthetics – total artwork or difference engine? Thinking 90 synthesis in digital audiovisuality.” Disjunction is not so much a ‘shock’ as it is in avant-garde cinema as a modulation of the coherence that a repetitive bass track sets up.

9 I am making something of an assumption in my analysis of this video. From what I know of the way in which Kurokawa works (gleaned from online interviews and watching documentation of his performances), he uses both audio and video signal in conjunction with each other as generative forces upon each other. So, for example, he may produce an audio track, which he will then use to cross-process video material. He might also then take this material and feed it back to affect the audio signal. My assumption in the analysis of this music video is that given he has been provided with the audio by another artist, he would have used the audio signal generatively to produce the pulses and intensities we find in the visuals. For further information about Kurokawa’s working methods, see Kurokawa, Rioyichi and Bertram Niesson, ‘Rioyichi Kurokawa: Perspective and Nature [Interview]’, Digimag 17 (September 2006), http://www.digicult.it/digimag/article.asp?id=627.

10 For general information about the MAX/MSPJitter environment, see the Cycling’74 website at http://cycling74.com/products/maxmspjitter/. In particular the page on ‘control’ is interesting. As will be seen from a typical segment of a MAX ‘patch’ on this page, although the software proclaims its ease of control, this is because, as it says, ‘everything is connected to everything.’ See the following page on the website: http://cycling74.com/products/maxmspjitter/control/. Effectively, then, control is less about a command line algorithm in this environment and more about the capacity for anything to become a controller, that is, for any module of the customized environment to modulate any other module.

11 The concept of metadata has been around for about 40 years and was first coined and then trademarked as Metadata by Jack E. Myers in 1973. In general, computer and information scientists and librarians agree that it is data about other data and that it is linked associatively to that other data both temporally and spatially, as for example with ‘tags’ that provide keyword semantic descriptions of web content. In general computational terms, metadata handles and locates this ‘other data’ in a more efficient way so that the bulk of the other data does not have to be processed. Its position is not entirely clear, though, as other data can become metadata too and from the point of view of processing, metadata is just data! For a good overview of the emergence of the term, see Bellevue Linux Users Group. ‘Metadata Definition’ The Linux Information Project, 2006, http://www.linfo.org/metadata.html.

Bibliography

Amerika, Mark. 2007. META/DATA: A Digital Poetics. Cambridge, MA: MIT Press.

van Campen, Cretien. 1997. ‘Synaesthesia and Artistic Experimentation’, PSYCHE. 3.6 (November). http://journalpsyche.org/ojs-2.2/index.php/psyche/article/view/2358/2290.

Cox, Christopher. 2002. ‘From synapse to signal’, Autopilot [catalogue of Carsten Nicolai’s work]. Berlin: Die Gestalten-Verlag GmbH. 14–17.

Deleuze, Gilles. 2005. Cinema 2: The Time-Image. Trans. Hugh Tomlinson and Robert Galeta. London: Continuum.

Deleuze, Gilles. 2005. Francis Bacon: The Logic of Sensation. Trans. Daniel W. Smith. London: Continuum.

Deleuze, Gilles and Felix Guattari. 1988. A Thousand Plateaus. Trans. Brian Massumi. London: The Althone Press.

Doruff, Sher. 2007. ‘Extreme intervals and Sensory Fusions: the Hinge’. Mutamorphosis: Challenging Arts and Sciences [conference proceedings]. Prague. (November 8-10). http://mutamorphosis.wordpress.com/2009/03/01/

extreme-intervals-and-sensory-fusions-the-hinge/.

Edwards, Paul E. 1996. The Closed World. Cambridge, MA: MIT Press.

Elliasson, Olafur and Chris Gilbert. 2004. ‘Interview with Olafur Elliasson’. BOMB, 88. http://www.bombsite.com/issues/88/articles/2651.

Harrison, John. E. and Simon Baron-Cohen. 1997. ‘ Synaesthesia: An Introduction”. Synaesthesia: Classic and Contemporary Readings. Eds. S. Baron-Cohen and J. E Harrison. Oxford: Blackwell. 3-16.

Kurokawa, Rioyichi, dir. 2007. ‘mysterious hoze’ music video, music by ditch, op disc label.

http://www.youtube.com/user/opdisc#p/a/u/1/_eqBT6YbXBs.

MacKenzie, Adrian. 2006. Cutting Code: Software and Sociality. New York: Peter Lang.

Manning, Erin. 2009. Relationscapes: Movement, Art, Philosophy. Cambridge, MA: MIT Press.

Manning, Peter. 2004. Electronic and Computer Music. 3rd edition. Oxford: Oxford University Press.

Massumi, Brian. 2002. ‘Buildings, Biograms and the Body Topologic’. Parables for the Virtual: Movement, Affect, Sensation. Durham: Duke University Press. 177–207.

Meltzoff, Andrew N. and Richard W. Borton. 1979. ‘Intermodal matching by human neonates’. Nature, 282, 5737 (22 November). 403–4.

Nicolai, Carsten. Telefunken. 2000. installation: cd player, cd, sony hiblack trinitron tv, dimension variable.

Nicolai, Carsten. and Hans Ulrich Obrist. 2002. ‘Hans Ulrich Obrist in conversation with Carsten Nicolai’., Autopilot [catalogue of Carsten Nicolai’s work]. Berlin: Die Gestalten-Verlag GmbH. 74–82.

Niessen, Bertram. 2006. ‘Ryoichi Kurokawa, Perspective and Nature’ Digimag 17 (September).

http://www.digicult.it/archivio/digimag_17eng/articoli/audiovideo_bertramniessen.html.

Packer, Randall and Ken Jordan. 2001. Multimedia: from Wagner to Virtual Reality. New York: WW. Norton and Co.

Ramachandran, Vilaynur, S. and Edward. M. Hubbard. 2001. ‘Synaesthesia — A Window Into Perception, Thought and Language’. Journal of Consciousness Studies. 8.12: 3–34

Simondon, Gilbert. 1992. ’The Genesis of the Individual’. Incorporations. New York: Zone Books. 297–319.

Strawn, John. 1985. ‘Overview’ Foundations of Computer Music. Eds. C. Roads, and J. Strawn. Cambridge, MA: MIT Press.

Stockhausen, Karlheinz. 2004. ‘Electronic and Instrumental Music’. Audio Culture: Readings in Modern Music. Eds. C. Cox and D. Warner. New York: Continuum. 370–80.

Wagner, Richard. 2001 (originally published in English 1895). ‘The Art-Work of the Future’. Richard Wagner’s Prose Works, vol. 1. Trans. W. Ashton Ellis. The Wagner Library. Ed. P. Swinkel.

http://users.belgacom.net/wagnerlibrary/prose/index.htm.

Whitelaw, Mitchell. 2008. ‘Syneasthesia and Cross-Modality in Contemporary

Audiovisuals’. Senses & Society. 3.3: 259–276.