“The different reader, decoder, translator, interpreter and diagnostician” – T.P.

A medium does not mean, but is. Following McLuhan’s cliché aphorism, let’s not take media for it’s ablative function; that of media-as-vessle “by means of which” a *thing* is mediated. Instead of understanding media as about the world, the proposition, following McLuhan, is to consider media as the world itself. The rain is the falling, the wind is the blowing. It is then redundant to say “the wind is blowing” because of course it is. Not redundant would be if you tell me how it is blowing, or what the blowing does.

Data sonification is the process by which data is sonified; comes to be sound. The process can take on a number of specific models and methods. However, the parameters for the definition of data sonification are clear. For one, data is the input and sound signals the output. Moreover, the output is to reflect objective properties or relations from the input, the transformation is to be systematic, reproducible and intentional. Ok. Considering these delimitations, it is clear that the written program; the code which carries out the transformation from the input to output, can perform in a wide range of ways. Data sonification is a semiotic system because it is “made of two independent worlds that are connected by the conventional rules of a code” (Barbieri 207). Next, our question would be, what kind of semiotic system is data sonification.

Dr. Thomas Herrmann showed us the different types of data sonification on a scale from analogic to symbolic translation. An example of symbolic data sonification would be the sound of crumpling paper you hear when you empty the trash on your computer. Analogic translation interests me more. Data in that case comes to correspond to sound; according to parameters etc. the data is ‘sounding out’. Even though translation occurs, meaning doesn’t necessarily arise from the output of pure analogic data sonification, as the process doesn’t re-code the output into a signifying system.

In general, when one would sonify data, the intention would be for the listener to learn something; to ‘read’ sensorially; extra- or non-linguistically. Dr. Thomas Hermann had a cohort of examples of near-future or already implemented technologies and scenarios wherein understanding through sound can be applied in the real world. All of these basically came down to monitoring systems. Whether it is the passenger of a self-driving car, surgeon or broker; the complex object they are peripherally trying to monitor can co-exist with them through sound. Our human sense of hearing is evolved to be directly linked to our somatic responses in ways that linguistic comprehension lags behind. Not only can we hear 360, while doing other tasks, but we are also incredibly sensitive to change in sound or sonic sign of a predator sneaking up on us. To me, the difference between a programmed sonic icon, which would tell me when the stoplight is green or when the gasoline in my car is low, is not that much different from the airport voice-over reminding that the danger level is code-orange or the voice at the beginning of escalators telling you to watch your step. Though perhaps generating via different systems, and not all as dynamic or responsive, the semiotic foundation is the same. Whether literacy of the codes are learned empirically or even studied, what underlies the systems is logos.

Analogic data sonification on the other hand, raises semiotics through sound to a much more interesting plane. First of all, I’m reminded by Valery Vermeulen and a person in the audience during Dr. Thomas Hermann’s sessions complaining that direct sonification doesn’t result in interestingly expressive sounds (to make a song with). It reminded me of different types of divinatory systems, one more analogic than the other, and the difficulty at times to read them.

The reader, the decoder, the translator, the interpreter and the diagnostician are not at all times one and the same.

Image taken by T.P.

Let’s (shortly) try to delve into the conditions of possibility of a semiology of sound by itself. By this, I do not mean the phonetic aspect of the linguistic unit (alongside semantics and syntax), but rather the material of sound itself. The vibratory, non-language material that is presupposed by language systems and heterogenous to it. Under what conditions can ‘sonic images’ and signs (in other words; sonic information) be understood to signify? The structural conditions of traditional linguistics cannot account for considering such un-bound material as signifiable. We have learned that the sign contains both the expression and the content; as a two-fold of signifier and signified. Deleuze, drawing of Danish linguist Hjelmslev’s semiotics, defined a relationship of designation distinct from linguistic signification.

Deleuze famously accounted for a taxonomy of the cinematic sign in his two books on cinema: The Movement Image (1983) and The Time Image (1985). Deleuze defined the image not only as visible, but also legible; an object of sensation which simultaneously can give rise to a thought or idea. The cinematic image gets its ‘aboutness’ by being intentionally framed (and thereby, as a part, stands in relation to the Whole, which it is a part of). If we apply this to sound, we see that perhaps the sound of a train passing by doesn’t designate in the same way as the result of data sonification, which according to Dr. Thomas Hermanns classification must be intentional. Deleuze – following Bergson’s equation of image and matter in Matter and Memory (1896) and updating Peirce’s triadic sign with Hjelmslev’s semiotics – proposed to consider pre-linguistic designation as “pure semiotics”. If signifying concerns extracting knowledge from material, asignifing pure semiotics is one which reveals meaning by putting it into relation. This extreme summary doesn’t do the complexity and rigor of Deleuze’s method of justice, of course. However, important for the story at hand is that by means of intentional framing, pre-linguistic material can be understood as significatory only by means of relational knowledge.

I remember Valery Vermeulen, in his lecture last December at Transmission in Motion, likened music not to geometry; the mathematics of points, lines and ratios, but rather to algebra, because of algebra’s symbolic nature. Considering the mathematics of symbolic correspondences, he said, allowed a view upon music’s ability/function to create ‘a vibe’ by means of its relations. I would like to know more about this and feel I must ask him to elaborate ON this more. Anyways, this has caused us to complicate one mention of the symbolic, that of the method of data sonification, with another, that of algebra relating to music. One note to get us out of a confusing mess would be to emphasize the difference between music which created ‘vibe’ and sound which expresses data. The symbolic, as I understand it, is a structure of logic based on arbitrary relationships. This enables us to extract, which at times could be useful. Making meaning by calling meaning. The analog means continuous variation, ratios and correspondences. I’m thinking of yes reflection, no interpretation.

References

• Barbieri, M. On the Origin of Language. Biosemiotics 3, 201–223 (2010). https://doi.org/10.1007/s12304-010-9088-7