Posts Tagged: ‘multimodality’

For a couple of years now, ever since I ‘seriously’ started engaged my dissertation research, I’ve been forming up this idea of sensory postcards as a methodology for doing everyday ethnography – but also, I guess, sensory postcards as a DIY new media practice that is facilitated by the ubiquity and mediation of mobile smart technologies. I even wrote this little thing for the Ethnography Matters blog. What I want to suggest is that by taking pictures, collecting environmental data and creating and sharing videos and recordings online, end users are participating in a kind of methodological approach to re-mediating experience and environmental surroundings. The only difference between that and a citizen-science or citizen-journalist initiative is that the same activity (of capturing multimodally) has a specific organization and structure, aimed intentionally at a public outcome. This post has actually marinated in my draft folder for a long time and I’d like to just let it go for now because there is a lot lot more to it, and I won’t fit it all in one entry, but one has to start somewhere. This blog is in fact already a collection of different ways of doing ‘new media ethnographies’ or ‘mobile ethnographies’ of the everyday: using visual, locative, measurement-based, and aural materials; putting them together in a variety of ways. The one limitation I have placed on my practice has always been – whatever can be accomplished on the device alone. Nothing leaves the device to be dissected and remediated on the computer as I’m truly interested in how mobile devices can be used, and how designers in fact respond to the on-the-ground use of these devices, so I see it as essential to continuously push the limits and communicate publicly about these experiments. Below is a small collage I made using several different apps: Over, which allows poster-font annotating of photos, SpeakingPhoto, which takes a static picture and overlays 10 to 30 seconds of sound recording over it, and again, SpeakingPhoto which allows collaging – stringing together of different ‘aural postcards’ into a slideshow. What I find interesting in making a slideshow is that it not only strings together individual entries into a narrative, but it also readily highlights the contrast between different sonic environments, by virtue of sharply transitioning from one to another.

This is going to be a picture-less, sound-less postcard, but nevertheless I feel compelled to comment because it was such a unique, simple, yet savory experience. I was really charmed by the simplicity of multi-modality – an open train or real tracks, in the brisk cold night, huddled with strangers going through a narrow passageway with scenes from various fairy tales unfolding off to the sides. A combination of real actors, props, detailed fabrications of scenes from fairy tales just enough to evoke memory of each one, really gave the train ride a sense of presence. In some ways it made me aware (and hopefully others) how used I am to the flatness of experience provided by media – whether it’s my TV or my computer, or even my beloved iProducts.

Auditorily, the delightful part was that each fairy tale scene was announced by sounds first, before it became visually present. But let me backtrack. The train itself has cheapo variety park speakers built in and played music constantly. Let’s see if I can describe it. It was a cartoonish melody but it wasn’t contemporary or popular, and not too childish. Had a fairy-tale character, perhaps reminded me of old vinyl records I listened to of dramatized fairly tales, or perhaps other people were reminded of their Disneyland experiences and so on. In any case, even tho it appeared to be ‘generic’ in a sense, it was actually a custom mix, because it mixed in elements – melody, voice and sound effects that represented in a refreshingly subtle way the fairy tale scene that was just about to materialize out of the darkness ahead. A particular highlight for me was the sound effect of hundreds of mice scattering and screaming in that pattern mice do, sound intensifying as we came up on a fabricated scene on the side of the tracks of a fake corpse covered in at least 50-ish plastic mice/rats.

Reflecting on this experience, again, I just can’t say enough how refreshing it is for me to re-conceptualize this notion of ‘multi-modal’ display/interface that gets thrown around a lot in the interaction design community, and for that matter, in education(al technology). The way this train ride was multimodal, with the darkness of the forest around us, eery branches and leaves, fog-covered pond, water reflecting the moon hiding monsters below, cold biting night air, analogue metallic clanking of wheels on rails with its distinguishable crackling when direction changes, the low-fi sound melody and effects played via speakers, plus the immediate sounds of the live actors on the various fairy tale sets. It was a truly multi-modal experience with a presence that the most sophisticated piece of technology hasn’t been able to get close to. The sheer depth of – that’s how I can really think about – the depth, dimensional effect of different sensual experiences layered together, sight, sound, touch, temperature, smell, emotion….Makes me realize – duly so – how limited the conceptualizations of even the most contemporary forms of multimedia, multimodal, tangible technology are.