Research Imagininghttps://researchimaginings.com
Travels in Design Research
Thu, 21 Mar 2019 15:54:14 +0000 en
hourly
1 http://wordpress.com/https://secure.gravatar.com/blavatar/220a1ca66bcfa6d2b4310e21192318f5?s=96&d=https%3A%2F%2Fs0.wp.com%2Fi%2Fbuttonw-com.pngResearch Imagininghttps://researchimaginings.com
Designing for scalehttps://researchimaginings.com/2018/11/08/designing-for-scale/
https://researchimaginings.com/2018/11/08/designing-for-scale/#respondThu, 08 Nov 2018 17:02:21 +0000http://researchimaginings.com/?p=599Continue reading →]]>I ran a workshop at the RCA last week on the topic of scale. I’ve been thinking about this for a while in the context of training sets for machine learning algorithms and the cognitive limits of trying to deal with the sheer number of data entities required. This has some profound consequences, not least the increasing political, commercial and cultural power that collecting and owning large data sets confers. A power in dramatic contrast to its costs – we have given Facebook its data gratis. The inequality of this ability to act in the marketplace of data driven effects flows directly from the difficulty of dealing with large data sets. Only very large companies, or those small ones who have attracted significant investment, are able to leverage the advantage of massive data. This means in turn that they they are the only ones able to offer sufficient incentive to the very few data scientists with appropriate skills. Pursuing the argument further, we can observe that the specialist knowledge required to automate the configuration of large data sets means the people most affected (that is, we neo-liberal subjects) are locked out of the dominant discourses around data driven experiences, and remain their passive consumers – posting comments to Instagram, listening through Spotify, updating LinkedIn profiles – with little thought to how our actions populate databases on a scale it is very difficult to comprehend.

A chance comment from a colleague related to how architects project their imaginative abilities up, from scale model to built environment prompted me to ask the question; what can we learn from how scale is handled in art and design that might be applicable to the field of massive data? There are a whole range of strategies that come to mind, from the pixillated paintings of Chuck Close, to Joseph Cornell’s boxes, Claes Oldenburg’s sculptures, and Tadao Ando’s models. Students were asked to go to a West London shopping centre, observe the space around them and annotate a grid with the various scales of human activity around them. The outcomes from this exercise included a variety of strategies.

Evan Reinhold devised a spherical grid on which to annotate the various scales of sound in Westfield, from intimate conversations, phone calls and footsteps to the music that plays throughout the whole building.

Claudia Soler Bernardini made a circular plot of the various light levels in Westfield. An inner ring shows the immediate personal context, the middle ring out to an approximate distance of ten metres, the final ring shows light levels over the whole centre.

Henri Holtz showed how people move through the space, showing where people enter the from and where they end up. Her grid focused on the distance people travelled and the trajectory they took across an arrangement of concentric circles representing a specific area of the building

Eriko Takeno made a mask to hold up in front of her face, on which she noted the sound sources around her, deliberately unaware of the distance and scale she was experiencing them at. The resulting notations are concealed behind a translucent paper layer to indicate the way they were only partially perceived.

The next step of the task was to make a more developed representation specifically of the relationships between different scales of data recorded in the space.

Kumi Oda used the reciprocal strategies of compression and expansion to represent sound waves. A highly compressed aggregation of sounds has an expanded counterpart that can also be viewed in multiple axes. Applying this to data we can imagine that compression and expansion, currently handled in say zipping of large files, could be a far more interactive and decisive experience. The ability to scale the data manually using sliders or range adjustment would also afford a choice of resolution.

Eriko used the metaphor of layers of transparency and opacity to explain the relative scales of data she experienced in the space.

The strategy used here involves moving the subject relative to the object. Extrapolating this to a dataset we can imagine spatialising the data in a virtual environment and moving through the subsequent dimensional representation so that we could inspect multiple scales in a single view by assigning opacity values to various parts of the data that were related in different ways.

Henri Holtz’s concentric drawings ended up in the form of a multiply folded and nested paper form. The strategy here shows that by folding concentric rings in a particular way such that they can be expanded easily and then nesting them, a huge amount of data can be presented at increasing scales of resolution or magnification. Folded forms can be unfolded to different levels of detail. This design evoked the way the brain is folded, the folding of protein, and the complex folding of DNA. This is an effective way of showing different scales of the same data at the same time.

Claudia ended up with a hemispherical geodesic dome structure within which she imagines other smaller structures would be nested. This implied regularised stepping down of scales decreasing (or increasing depending on the direction of travel). Again a good way of handling the navigational challenges of large data sets.

Fabio Fidanza and Jack Smith created a way of abstracting scale to a set of intersecting intervals. By hiding and exposing the data in a rhythmic pattern they allowed for visual complexity mapping. This would be a useful indexing method for large data sets and would respond well to a sonic or filmic treatment. A rhythmical microfiche system with unique identifying signatures for interacting with varying degrees of complexity.

The imposition of constraints is described by Nigel Cross to be a decisive way that designers conceive the world, one he says that is very different to an engineering mindset. The constraints seen in the results of this workshop, the strategies of folding/nesting, expanding/contracting, intersecting/projecting were seen to be immediate ways designers dealt with problems of scale. One aspect was that these qualities presented as dyads, reciprocal characteristics of a single system, perhaps the basis for a more developed theory of how scale is dealt with in art and design.

In response to the call for sensitising concepts in the form of designerly abstractions suggested by Yang and others as a way to make machine learning technologies available as a design material to user experience designers I ran a workshop at London design agency Venture 3. The aims of the workshop were to develop ways of thinking about machine learning that will help the team there to work in new ways and respond creatively and ethically to the challenges and opportunities of machine learning.

Participants started by listing some of the positive and negative aspects of digital systems driven by machine learning. The intention here was to ask the question: What is going on? i.e. what work does machine learning perform in digital systems. These were done from the perspective of users, rather than designers or developers. Participants were asked to write positive and negative attributes onto sets of cards. Positive attributes of machine learning in digital systems included the removal of human error, the potential for an increase in leisure time, more emphasis put on human abilities, personalised medical diagnosis, and an infinite amount of content. Negative aspects included; an inability to opt out, the threat to jobs, a reduction in human contact, loss of physical experience, and the monopolistic dominance of companies rich in data. Participants reported that their analysis took the form of an axis with efficiency and autonomy as its opposing values, saying ‘every time you make a move to make your life more efficient there’s a necessary sacrifice of autonomy’. Others pointed out that machine learning driven systems are content agnostic and derive value from interactions; ‘the user has more negatives and the platform gains more positives from the existence of machine learning’ as users deliver their preferences and choices for free to the platform it uses machine learning to tailor its service ever more accurately to other users.

Some participants chose a particular consumer system, such as Uber, to focus their analysis, mentioning the phenomenon of the misplaced surge pricing that took place in the aftermath of the Manchester Arena bomb attack, and the Greyball scandal in which Uber used algorithmic processing to avoid regulatory inspection. Other observations related to a potential future AI-deficit ‘what does a life outside AI look like? Will there ever be an opt out?’ and how it may be paradoxically satisfying to see machine learning making mistakes and not fulfilling the all-seeing omniscient system it is often sold as. From the perspective of design these observations raise the possibility for building in a certain amount of error, for a designed opt out, the need for shared experiences to counterbalance hyper-personalisation, and a nuanced understanding of how personal identities are subverted or reinforced over time.

Phase two of the workshop involved participants making representations of their ideas of the effects of machine learning – designerly abstractions using physical materials. As framing concepts we used the categories of transparency, unpredictability, opacity evolution, learning, and shared control identified by Holmquist as the main factors in designing for AI. The aim for this phase was to get some of the ideas off the page and into tangible, substantial forms around which a new set of understandings could coalesce. Design consists of making things, at Venture 3 this means the creation of systems, campaigns, identities, and interfaces. Increasingly the materials these products will be made of will include machine learning, how can it be made tractable for designers?

Identity emerged as a major theme of this phase with one group observing that the siloed subjectivising effect of machine learning could be counterbalanced through data merging; ‘what we should be doing is combining your data with lots of other versions of yourself… and maybe you could have almost alter egos of yourself’. The defining metaphor here is the bubble, as suggested by Pariser (2012). Participants proposed ‘bubbles could expand or decrease… different things fluctuate based on your experiences and your interactions with the AI and with other people’. The effect of this would be that ‘you cannot distinguish yourself anymore because the complexity of the information that you’re giving to the bubbles doesn’t allow them to clearly identify you as a single person’, an engineered counter-strategy to extreme personalisation. ‘The version of you that exists is relative depending on all the other things that are in and around you. So there is no one you, there are twenty versions of you’.

The need to account for the complexity of entanglement in systems that depend on machine learning was also expressed in the designerly abstractions produced in this phase. Thinking through making was evident here as participants worked to articulate their thoughts in physical representations. The way that a single piece of data, such as name or date of birth, is processed so that it produces a particular result in, say, a digital profile was challenged by the idea that multiple pieces of data (bearing in mind a single tweet contains 32 metadata items) add up to more than the sum of their parts. ‘Me telling them that my name is Lily and I’m 23 essentially means that what I though were two yellow and red pieces of information make an orange piece of information’. The metaphor of colour mixing is used here to explore the notion of algorithmic complexity. ‘because it’s all built on assumption and generalisations… although the pictures might be clearer, it actually becomes cloudier’. This approach raises the possibility that designers should work to subvert the illusion of efficiency and accuracy of machine learning. ‘You have these entities that enter into a relationship with each other, which seems like a much more sophisticated thing to communicate than… “this is who we think you are”‘.

The final phase of the workshop required participants to imagine speculative digital products in which their observations and thoughts about machine learning could be implemented. The first of these was a layered costume, ‘a shroud of who you are… something deeper about who you want to be today… it evokes a sense that your data is embodied around you, it’s not a distant cloud thing.’ The metaphor of bodies, layers, and textiles is at work here representing how data-driven descriptions are interdependent and can have very real tangible effects on people. It is noticeable in this project that a spiritual dimension is expressed as surrounding the data self.

Another project of this phase focused on the design of a machine learning augmented judicial system. The initial suggestion was that machine learning could help to remove human bias in a jury by accounting for a wide range of precedents and legal details. This idea evolved into the realisation that the more complex problem was considering the many types of circumstantial evidence, contextual factors, and emotional consequences and the possibility that machine learning could be useful. The group then suggested that “removing the jury and using this super holistic lawmaker” may help with trying to create laws that address behaviours that are not yet crimes. Examples such as upskirting and revenge pornography where the moral and legal situation seems clear but for which the judicial system has not yet developed legislation. The algorithmic lawmaker ‘creates precedence for new laws… by using machine learning to cross reference similar cases’. The system would do this by using vocal analysis to recognise shame, embarrassment or signs of depression in witness statements.

At the end of the day we reflected on the findings and methods used, particularly from the perspective of using tangible materials to create the designerly abstractions Yang calls for. These are intended to be sensitising concepts for designers and act as boundary objects in the development of a shared language between user experience designers and data scientists. Participants said the methods were useful for revealing hidden characteristics “prototyping quickly uncovers problems that you maybe hadn’t thought of sooner”. One of the benefits of the phased approach was speed “You’re just getting things down in a really quick way – out of the sketch pad too”. While there was a direct mapping between metaphorical and the physical “You do literally look at the problem from different angles and that makes it different, you start to look at the relationships”. So making physical representations closes the gap between abstract and concrete while allowing complexity to emerge. Participants acknowledged a cognitive dimension to the exercise ‘it just felt really different, you feel like you’re using a different part of the brain’. Making things at low resolution was useful; “it removes the barrier for people like myself who are less in the know (about machine learning)… it helps you be a little bit more open about how you talk about it”. So there is a real benefit in terms of communicating with physical abstractions.

Resolution of representations also emerged as an important factor. ‘If we’re creating products for people who aren’t creatives or designers it has to be even more simple to understand.. I think it helps in being quite primitive with our prototyping’. The physical nature of the materials allowed participants to bypass usual design reasoning; “It’s not about making a perfect outcome, it’s about communicating the core of what I think”. “It doesn’t have to be right it just has to communicate right now!” The limitations of materials and time were also liberating for participants; “I can only hope to get something that kind of communicates a thought and that’s quite freeing”.

For one participant using playful physical materials had a positive emotional effect. “using different childlike materials to talk about (machine learning) does… make you feel like (designing for it) could be a bit more joyful” and allowed for reflection on working practices. This ended with the question “What criteria would we as a company use to make AI joyful rather than efficient?”

In conclusion, workshop outcomes had a range of effects that could be summarised as foregrounding complexity, countering illusions, revealing hidden effects, and challenging design practices. I will be exploring these further in upcoming workshops.

]]>https://researchimaginings.com/2018/09/07/sensitive-abstract/feed/0image 4johnfassIMG_6582gPresentation1Designing machine learninghttps://researchimaginings.com/2018/08/29/designerly-abstractions/
https://researchimaginings.com/2018/08/29/designerly-abstractions/#respondWed, 29 Aug 2018 16:35:20 +0000http://researchimaginings.wordpress.com/?p=571Continue reading →]]>The relationship between user experience (UX) designers and machine learning (ML) data scientists has emerged as a site for research since 2017. Central to recent findings is the limited ability of UX designers to conceive of new ways to use ML (Yang et al. 2018). This is due to a number of factors. Firstly, human intelligence is very different to machine intelligence. Instead of using a heuristic or associative model, ML uses statistical inference to produce outputs that can often seem nonsensical or confusing (Yang, 2018). Secondly, the types of data that UX and ML depend on can be mutually incompatible. UX Designers have developed an extensive set of research techniques to produce qualitative insight into how people experience digital systems. ML data scientists use mathematically derived automation to deliver quantitative findings (Girardin and Lathia, 2017). Thirdly, ML technologies are difficult to understand. UX designers have found it hard to understand the limits of ML and how to apply it appropriately. Finally, the type of designs facilitated by ML technologies can be unfamiliar to UX designers. This is because ML driven systems evolve according to human behaviours and are constantly updated as models are fed new streams of data (Girardin and Lathia, 2017).

The call from HCI researchers and design researchers in this context (Yang et al. 2018) has been for ‘sensitising concepts’ intended to help make ML available to UX practitioners as a new design material. Sensitising concepts reach beyond their immediate material manifestation to sensitise designers to the possibilities of the suggested design resource. Sensitising concepts expand the field of practice of
a particular design domain by demonstrating how new materials encountered within that domain may be used. Sensitising concepts are embodied in ‘designerly abstractions’ which free designers from having to fully grasp the technical constraints of ML technologies (after all data scientists are rarely expected to understand even the most basic conventions and practices of user experience design), instead allowing them to explore alternative forms. They act as boundary objects between UX design and ML data science fostering new ideas and bridging the gap between design possibility and technical capability (Yang et al. 2018) working to make ML available as a design resource. This appeal for boundary objects also acts as a call to mobilise designers in the field of ML, and artificial intelligence more generally, positioning design as an intermediary between data science on one hand and the regulatory or legalistic readings of the field on the other.

Examples of these abstractions include responses from research participants that suggest ML results in personalised experiences, evolving relationships, and uncertain outcomes. The emphasis overwhelmingly follows a transactional model of exchange through which ML technologies are seen as providing a new aspect of service in exchange for the data that enable those new services to develop. There has been to date little reflection in the research on the possible negative social effects or unforeseen consequences of an increase in human experiences that are determined by ML algorithms. It seems necessary to include the ethical and moral aspects of designing for ML technologies in the sensitising concepts and designerly abstractions they are embodied in. These may include observations that ML driven systems reproduce inequality by reinforcing the biases inherent in the training data, or that they bring about a loss of control and transparency. This last point finds validation in the recent House of Lords report AI in the UK: ready, willing and able? which finds explainability to be a desirable characteristic of future ML systems.

]]>https://researchimaginings.com/2018/08/29/designerly-abstractions/feed/0IMG_6356johnfassDesign and AI/MLhttps://researchimaginings.com/2018/06/17/design-and-ai-ml/
https://researchimaginings.com/2018/06/17/design-and-ai-ml/#respondSun, 17 Jun 2018 11:58:25 +0000http://researchimaginings.wordpress.com/?p=568Continue reading →]]>The accumulation of enormous quantitative data sets, via digital social media and other systems, paired with recent developments in neural networks and increases in computing power has delivered unexpectedly rapid improvements in what artificial intelligence technologies have been able to achieve (Holmquist, 2017). This means the influence of algorithmic decision making and machine learning on digital products, from social media to financial management and healthcare, has increased significantly. As these systems start to pervade everyday life, they present a challenge to human understanding. We risk developing highly influential technologies of such complexity and opacity that they surpass our abilities to shape them into forces for the common good.

The consequences for culture and society are profound. Firstly, the ethical implications of personal data that is captured in a public digital place and used to train an algorithm, designed by a private corporation for unknown purposes shielded by commercial secrecy involves a dramatic imbalance of power. Secondly, the invisibility and opacity of machine learning technologies means access to the means of production is limited to the few people trained and skilled in creating them. Finally, the conscious or automatic manipulation of flows of information via digital products has been shown to be a danger to democratic processes and information equity.

Involving designers in the development of practices that will help understanding and explanation what is going on in the interfaces and interactions of digital products that depend on artificially intelligent systems means making the case for design in this context. How can designers of digital products make the workings of artificial intelligence more apparent to users? What practices and methods can help designers of digital products to reveal the workings of artificial intelligence? How can a set of practical design principles help to counter some of the negative effects of cognitive technologies in digital products?

Design research has not paid detailed attention to this topic in this way, although it has been covered in studies related to the ethics and politics of machine learning (de Bruin and Floridi, 2017, Mittelstadt et al, 2016) and in theoretical approaches to interaction design (van Allen and Marenko, 2016).

Designers have sought to be deterministic, defensive or opportunistic in the face of AI technologies, describing new applications for machine learning technologies, arguing for the preservation of human voices in the design process, or describing how to deploy AI more fully in the design of digital products. The technologically deterministic approach is seen in design research that emphasises technical solutions to AI communication problems (Feldman et al, 2017) or AI integration with new types of hardware (Vidaurre et al, 2011). The defensive reaction to AI technologies in design responds to the threat of human designers being superseded by computers (Teixeira, 2017). This strand of thinking focuses on what human designers can do that AI is suggested to be incapable of. The opportunistic reading of AI in design works in two ways. Firstly, by identifying ways AI technologies can improve user experience by increasing personalization, or analysing huge amounts of user data. Secondly, by providing usability guidelines for how to use AI in design (Holbrook and Lovejoy, 2017, van Hoof, 2016) this thinking proposes a set of skills designers will need to respond to the emerging age of AI in design.None of these approaches to the topic of AI and design consider the ethical, moral, and political implications for designers involved in creating AI driven systems. They do not attempt to explain how AI is used in their designs nor how the AI may be influencing the choices people can make, or are subject to. People are understood as subjects of the technology, for whom designers optimise the user experience by harnessing the power of machine learning.

Instead I suggest in my research pragmatic responses to the argument put forward by Holmquist (2017). He suggests ways designers can make the behaviour of artificial intelligence understandable. For example, designing for transparency means showing in a design what the AI is doing at any given time, designing for opacity recognises that the intricate workings of AI driven systems may be beyond immediate understanding. Designing for unpredictability takes this further to emphasise how the nature of machine learning means it is subject to error and uncertainty.

In an interview published in 2009 by New Materialism, Karen Barad responds to the idea that the dualist tradition in cultural theory, i.e. the historical strand of thought engaged in considering relations between mind and body, has given way to a ‘new’ materialism (also referred to as the ontological turn or post- humanism) that emphasises the complexity of these relations and considers them as ‘travelling fluxes of nature and culture, matter and mind’ and that through these fluxes ‘active theory formation’ is possible.

A link is made here with Barad’s agential realism, which proposes that matter is always inextricably intertwined with discourse. Of course this is not a new idea and it has roots in Foucault’s dispositif and especially in Judith Butler’s discursive-linguistic concept of performativity. A key underlying concept of agential realism is intra-action, which Barad calls ‘an ongoing open process of mattering through which ‘mattering’ itself acquires meaning and form in the realization of different agential possibilities’. This is in opposition to interaction, which presupposes the existence and relational aspect of already mattered (that is, objectified) phenomena. Through intra-actions the entanglement of matter with the material-discursive strand in cultural theory is given new theoretical instruments with which to dismantle the old dualisms and transcendences.

Following Haraway, Barad then deploys a metaphor from physics, diffraction, which is used in a generative sense to frame a non-dualistic and consciously ethical analysis. Barad proposes that the practice of diffraction involves ‘reading diffractively for patterns of differences that make a difference’ (in a clear reference to Bateson) and is a way of being ‘suggestive, creative, visionary’. A diffractive methodology means reading insights through one another, attentively reading for differences, and is ‘a metaphor for another kind of critical consciousness’ (Haraway). Barad explains this metaphor through the physics of optics, contrasting geometrical optics, which is unconcerned by the nature of light, and physical optics, which considers the light passing through the measurement apparatus, and the measurement apparatus itself, to be interdependently entangled.

The diffraction metaphor is further developed by Barad with reference to quantum entanglement, part of a much wider theoretical project to weave the natural sciences into the humanities. Indeed much of the work diffraction is intended to do is produce ‘new patterns of thinking-being’ by reading texts from contrasting traditions through each other, generatively producing new insights from the resulting intervolvement. Barad also uses the famous double slit experiment to show how wave-particle duality demonstrates intra-action between electrons and the apparatus used to measure their distribution. The phenomenon arises in its enactment, in fact the phenomenon is the enactment, measurement produces properties and boundaries. This is the difference between an ontological and an epistemological reading of diffraction.

Diffraction is at once a practice, a methodology, a route to analysis, and an ethical commitment. One question for me here (following the principles of pragmatism) is; how can we use these ideas? what use can an onto-epistemological mattering be in design? How should we implement agential realism in design practices? what kinds of suggestive, creative, and visionary designs might be possible?

Diffractive design would draw on other knowledge domains to produce new insights. This is not at all a new idea in design and is already seen in design’s productive engagement with, say, synthetic biology or space science. One criticism of this kind of work is that it avoids methodological entanglement, reserving a special place for itself in the sphere of representation. Non-denominational diffractive design would abandon its exceptionalism and be willing to read itself through, say, chemistry or palaeontology, rather than through semiotics or ethnography. This is not to say that these disciplines would be looking for reflections of each other, but that the diffraction pattern of their mutual interference would uncover their previously existing entanglement.

Diffractive design would attend to the intra-actions of its own separate disciplines. In other words textile design would come into meaningful (by which I mean generative) entanglement with interface design, vehicle design with typographic design. This would produce what Barad calls ‘inventive provocations’ by consciously designing for the entangling agencies (the meanings, habits, materials, and discourses) of different creative domains. From diffractive design then would come a new kind of designer, she would configure material-discursive situations across various traditions to generate new entanglements, rather than new products or objects.

Diffractive design would also acknowledge the apparatus (knowledges, experiences, educations, clients, fundings and institutions) through which it comes into being. In fact it would necessarily be inextricably intertwined with these things. Agential realism would suggest that this is a matter of responding to ‘the particularities of the power imbalances’ in design. These are seen as always contingent and always enacted through practice. Designs that responded directly to the set of conditions that produced them would not just be reflective, they would also be transmutative of those conditions, and observable in systems of meaning and matter.

Diffracting design would recognise itself as an apparatus. One through which designers are caused to act in certain ways, and through which non designers are subjected to those acts. Design as an apparatus would draw attention to its own internal workings as a creative act; the generation of ideas, setting of briefs, listing of references, researching into usabilities, putting into tangible form and evaluating outcomes etc. That is not to say that the confines of the apparatus are necessarily definable but as Barad suggests ‘boundaries, properties, and meanings are differentially enacted through the intra-activity of mattering’. In other words it is through praxis that design as an apparatus becomes recognisable, or at the very least perceivable.

Diffracting design would work through the diffraction of its field of operation. By this I mean to use diffracting as a verb rather than adjective. To diffract something in this context means in some way to divide it into a specific selection of its constituent parts that may subsequently emerge as a form of patterning, or as a spatial and temporal signature. Design performs this diffraction through the material realisation of forms. Materials are considered to be in a relational arrangement with the makers, contexts knowledge, and actions that they come into contact with. This is perhaps where Barad’s ‘matters of fact and matters of concern’ take on a more reified aspect. The theoretical backdrop to Barad’s reading of cultural theory through physics has its roots in feminist science and technology studies. These tend to prioritise bodies over substances, emphasising ‘bodily production’, ‘engagements among body parts’, ‘the marks left on bodies’ to pick just a few from the interview. Barad’s description of matter is also clearly embodied ‘Matter feels, converses, suffers, desires, yearns, and remembers’. How then should design theorise its use of materials and mattering, which may range from code to concrete?

Diffracting matter in design, means diffracting design itself. To this end Haraway advises against the ‘god trick’ of positioning ourselves outside of, and separate from the mattering of the world around us. Instead we are inevitably, and thus ethically, intertwined with it. Barad mobilises the metaphor of entanglement from quantum physics to describe this relation. In design this would imply that we think of ourselves as comprising a design material, connecting again to the embodiment motif, or perhaps that ways of shaping and arranging materials by hand or by mind involves an ethical (by which I mean response-able as Barad says) entanglement with making things. Being, acting, and knowing (the onto-ethico-epistemology) meet in matter.

Much of the language used to describe the concepts of the new materialism is circular and can be contradictory. For example, the dual meanings of the word matter i.e.as verb and noun is consistently evoked, often in a playful poesis. Two parts of the theory stand out for me as potentially problematic. Firstly, much rides on the connection between metaphors from quantum physics and concepts of relationality from cultural theory. The outer reaches of quantum entanglement being possibly unfamiliar territory for readers in the humanities the suspicion remains that the entanglement metaphor is taken at face value and is over-instrumentalised in the argument. Secondly, as others have noted the automatic assumption of an ethico-ontology as a precondition for being and knowing, (that is, ethics does not have to be placed, articulated, or committed to, it just is) may let us off the hook too readily.

Finally, I don’t pretend to understand all of this by any means but wanted to set out my thoughts regarding Barad’s work in relation to design. This post is published in the spirit of open inquiry and questioning.

I ran a session on cultural probes with Caroline Claisse last week with some interesting outcomes. My part was to trace the historical development and positioning of cultural probes as a research method in the field of designing for technology and HCI, the session threw a few things into focus for my own research, especially as I’ve been writing a draft of my methodology chapter this week and so reflecting on how methods and epistemological viewpoints combine into coherent, or at least explainable, methodologies.

Cultural probes were developed here at the RCA in 1997 in the context of a masters program in interaction design run by Bill Gaver. The probes method is a very art-and-design oriented response to the prevailing methods in HCI at the time which were dominated by the cognitive tradition on one hand, and the engineering tradition on the other. These two camps together attempted to propose that traditional methods from social science, such as ethnography, and quantitative methods from engineering, such as human factors testing, could deliver some form of truthful insight about people that designers could then go and put into form. Instead, cultural probes are intended to be open-ended, ambiguous, and inspirational. They are not supposed to deploy traditional understandings of objective research methods, and are certainly not a form of requirements gathering. In other words, there is not likely to be any kind of exit interview where participants talk in detail about what they have done in a way easily captured by designers. Gaver et. al describe them as a bit like space probes – returning fragmentary data at infrequent intervals from unknown places.

Perhaps the hardest part to grasp is the idea that cultural probes are not intended to be goal oriented. They are deliberately provocative and can be opaque and enigmatic – not usually qualities associated with research methods. If probes are used to generate a design solution in a very explicit and directed manner then usually lots of other supporting arguments and methods are used. This implies the observations, interviews, video recordings, and questionnaires common in the conducting of qualitative research. Much of these supporting methods perform objectivity without any meaningful understanding of the hermeneutic tradition. This is where the clash with the truth-finding scientific methods of engineering and cognitive science is most apparent. The role of the researcher as conscious interpretive presence common to the ethnographic tradition is dropped in favour of the notion of an instrumental researcher uncovering hidden truths about the world.

In their influential paper, Boehner et al. argue that probes have often been used as an off the shelf method in technology design. The form of the probes (camera, diary, maps, postcards) is present but the spirit (open ended, subversive, playful) is not. Research findings end up funnelling responses into datasets, or worst of all statistical tables. The results end up being quantitative instead of qualitative. Probes are experimental, evocative, surprising, they open the design space, they do not define it. The idea of a designer as someone whose rationalising scientific instrumentality imposes function on a chaotic world is not the one embodied in the probes’ methodology. So for my own work with browser history comics, social network modelling, and experiencing meta data the methods are probe like in that they are playful creative activities, the interpretive framework is important, I’m interested in the artefacts around which peoples’ stories emerge. They are however much less like probes because; I am asking some precise questions in order to gain understanding about specific lived phenomena, I am not designing for a separate output inspired by responses, Instead I will integrate and involve participant responses into practical design work, and I have been talking to people in detail about what they did and why. My work sits somewhere between open ended ambiguity, and directed generative design research.

Caroline and I gave the students a difficult task in the workshop which in retrospect we could have introduced slightly differently. Designing an ambiguous set of materials to elicit inspiring responses in a playful and creatively rewarding way is not easy! See Caro’s work in this area for how it should be done. The students came up with some fascinating solutions;

This is the ‘looking for a room’ Homing pigeon by Riah Naief, Will Fairbrother, and Wasabii. You take it with you when looking round potential places to rent and it reports on the general condition of the place. Produced in response to the brief – design a cultural probe inspired by what people think about looking for somewhere to live in London.

Responding to the same brief, this map of London uses 3D glasses, stickers, and coloured plastic film to annotate map of London as you search for somewhere to live. Maps have been used in Cultural probes since Gaver et al’s original kit in 1997 and I really like the different modes of annotation provided here.

A cultural probe exploring the social meaning of cooking for family members produced this folded menu-like leaflet. Tasks involve drawing people, design a menu based on personality and recording responses as people eat together. Although the form of this probe is very simple, easy to produce and accessible the tasks it requires would produce some richly textured information.

This probe is intended to uncover some responses related to the playing of computer games in public. It is pitched more explicitly as a set of instructions related to game playing and includes the delightful task of explaining which computer game avatar you would go for a drink with and why. Again, simple, playful, accessible.

Finally, Lorenzo Pradelli made this elegant probe to elicit data about cycling in London. Although not strictly a cultural probe since it is a quantitative data gathering instrument, it is a great design. The questionnaire sheets are wrapped around the bike light batteries meaning they do not work. The cyclist opens them up in frustration and finds the paper scrolls asking for written information.

Given that the time available to come up with these designs was no more than 90 minutes, the student teams managed to produce some great ideas. Any one of these could be developed into a novel research instrument, rich with possibility for creative inquiry.

We invited the brilliant Elisa Bellotti from The Mitchell Centre in Manchester to talk to us about social network analysis. Their work is all about showing social relationships in various types of representation conducive to mathematical analysis. Computational sociology involves analysing large scale networks in various domains such as social networks and health, or social networks and consumption. My own work in this area focuses on how people choose to represent their own social relationships in various constrained categories. Elisa explained how these are called ego-networks since they are centred around an individual. It turns out in social network analysis, and computational sociology in general that representation isn’t really a concern. The shape of networks, and their topological diversity and consistencies are important only for the kind of mathematical analysis they afford. In other words there is no real value seen in network representation as a document in itself and what it might reveal about the person who made it. I guess this reflects the difference between working in a school of art and design, where we think constantly and rigorously about the politics of representation, and working in the sociology department of a large university. Nevertheless, I did find the lack of analytical attention to representational structure surprising.

A semiotics of social network graphs could include, for example, a notion of physical distance mapped to cognitive distance from the centre, or how entities are arranged respective to each other – radial, sequential, scattered etc. I would see this as a human centred approach to network analysis. Since, as a positivist scientific discipline, social network analysis must attempt to conceptualise broadly applicable categories from specific observations, and render those observations susceptible to mathematical resolution, it has less time for design, meaning, human centredness, or participative methods since they are not really required. Most significantly for me, how the making of a personal network representation works to externalise an internal model of relationships and the accompanying potential for individual meaning making, is not something a social network analyst would pay attention to. The role of design in that process is limited to how a basic set of diagrammatic categories such as triads and dyads contributes to a reading of the network structure. The aim of computational sociology in the context of, say, networks and healthcare is to reveal things like where in the network resources should be concentrated. The network is a medium for deeper analytical attention to how social networks work in a variety of domains rather than an opportunity for the creation of personal meaning on its own. My work is phenomenological in the sense it tries to get at how people experience digital social networks for themselves. Social network analysis has a very different set of priorities.

A big issue for me is the resolution of the network representation. I have found that network models on their own are relatively information poor. Without accompanying interviews and other supporting analytical data, the physical models do not show how the process of making them has contributed to knowledge construction or meaning making. Listening to Elisa talk about the large scale multi-dimensional networks she works on makes me speculate that low resolution graphs (those with relatively few nodes and edges) may be information poor in general. More likely however, is that the information I’m after is not apparent in the model alone. My research question here is How can design activity work to externalise mental models of digital social networks? In other words, it’s the process of creating a physical model along with the resulting representation that delivers insight, rather than than the model alone. This is a process-oriented system as opposed to a goal or product-oriented system. As an aside I have found that representations conform to the so-called Dunbar number. This is an idea from computational anthropologist Robin Dunbar that the human brain has evolved to manage no more that 12-15 close personal relationships. This can be measured by counting nodes.

Representativeness is another issue. It could be that the inner mental models of personal social networks don’t take the form of conventional network arrangements at all, and the material constraints of my design force people to conform to a network shape. Perhaps the dominant spatial metaphor shown in network diagrams influences the way we conceive of our own relationships and left to our own devices we might more naturally show relationships as an ordered list, or composition of hazily defined clusters. In any case, my research question isn’t what representations most accurately show how people conceive of their own networks but what the role for design and making is in externalising those mental images. To that end, future work would explore the affect of different types of material representation on externalisation. I have found that participants take to the task very readily without much guidance or instruction, and that the process of doing so is rewarding in a personal creative sense.

One of my colleagues Dan Lockton brought up the suggestion of causal network representations. This suggests augmenting existing network models with layers of causal indication. i.e. I have observed this phenomenon and it is in relation with these other things, and I think these factors have caused this phenomenon. Some form of annotation would achieve the same result but could get unmanageably complex from a coding perspective. Again perhaps one for future work. Overall, the place for a design-oriented perspective on social network representations seems to be fruitful territory, one with plenty of scope for invention, human-centred surprises, and practical design exploration.

For last week’s work in progress show, working with Shobhan Shah, I put together an interactive installation that was designed to be a digital/physical video player. The basic idea is that the nodes in a network become buttons that trigger video footage of someone talking about the particular relationship represented by that node. So as you press on the buttons wether in a specific sequence or not you get the story of a specific social network. This gets at some important elements of my research – that social networks embody the very human qualities of personal relationships and that one important way of understanding these relationships is by recounting or telling stories about them. In order to view the video it needed to be back projected – you can’t stick physical pins into a monitor. It was important to me that the proportions of the network substrate were identical to the original. I sourced some frosted perspex and pointed a Pico projector at it to find out what the minimum projection distance would be to achieve the image size required. This distance then dictated the height of the resulting plinth on top of which the perspex sheet sat. The next challenge was to drill holes in the perspex sheet in the same configuration as the original tile, stick buttons behind the holes and mount cork plugs robust enough to hold a coloured push pin in place but still allow for the detent on the button beneath.

Of course we soon discovered that back projecting an image onto a sheet of plastic has focus issues. The projected light passes through the lower surface but is diffused by the frost in the centre to result in a soft image on the front surface. We never discovered the workaround to this problem but my colleague Michaela French came up with a brilliant solution. For her own exhibit she projected onto small sheets of transparent perspex that had only one surface frosted by laser cutting a shallow engrave over the entire surface. Great thinking through materials and a real why-didn’t-we-think-of-that? moment. All our buttons had to be connected to an Arduino at floor level 1 metre away in order to interface with a mac mini hidden in the base of the plinth. The issue then became how to stop the wires from creating sharp shadows on the image. After much fiddling and sticking eventually we got all the wires out of the way and stretching away internally down to the base of the plinth. The final setup was then a projector mounted at floor level, pointing upwards to a projection surface which had back mounted buttons onto which brass tubes containing cork plug were glued. These mounted into the perspex such that they were flush with the upper surface. Of course back projected images are reversed so we had to fix that and make sure the entire surface was covered by the projected image not leaving any black borders or misaligned videos.

This tangling with physical materials, what Ingold calls correspondence, involved dealing with MDF panels, perspex sheets, push pins, tiny buttons, brass tubes, cork sheets, and ultra-fine filament copper wires, all arranged in a finely balanced configuration of elements intended to deliver a specific experience. The processes associated with these physical materials -stripping and wrapping wires, sawing, cutting, drilling, glueing, clamping, painting, and soldering all unfolded in a careful sequence of planned actions. In some way the materials are inseparable from the processes necessary to shape them into their final forms. Many other and different processes are of course possible, but the nature of our design meant a very detailed and particular ordering of materials and processes. I’m calling this contextual gestalt, meaning the putting together of various materials in a series of fixed physical and compositional relationships with each other, intended to result in a pre-conceived expressive placing.

A very particular form of material synthesis is created when in active correspondence with digital materials, In this example the digital materials consisted of; HD video edited to carefully trimmed snippets, their associated soundtracks, a processing sketch to control the relationships between pins, buttons and wires, and the operating code of the Arduino micro-processor to which the wires, and resistors were attached. Digital video is notoriously difficult to work with. Diverse compression algorithms have wildly different effects on playback and sound quality, reversing the footage to be back projected involved exporting them to a completely new application, file format standards vary across applications, and computer playback depends on the graphics processor unit of the target device. While they are cheap prototyping tools, and allow for creative experimentation on a revolutionary scale Arduino micro-processors are also prone to failure, can often feature power supply problems and not well suited to long lasting public interactions. Luckily our circuit was very simple, the videos very short and the exhibit only on show for 5 days.

The overall synthesis of digital materials with physical materials involves very different design and making skills and it is rare to find people equally proficient. While I am comfortable in a physical workshop, working with manipulable tools, I did rely on Shobhan for the Processing and Arduino elements. In fact many of these types of digital/physical works are made in teams of people with complementary skills. The real challenge in this field is integrating the smooth, nano-second speed, invisible processes, and dematerialised interactions of the digital with the messy tasks of glueing, sawing, clamping and sticking of the physical world. Add in the unpredictable and ultimately chaotic nature of human behaviour and the resulting opportunity for surprise, delight, failure, and revelation through design can be rewarding.

I’ve been making a prototype this week for exhibition in a ‘work in progress’ exhibition and thinking about the moral dimensions of material engagement with research artefacts from a few different angles.

My intention has been to integrate the physical social network tiles with the videos I made of participants talking about them. So the idea is that the physical network, and the story of the network are presented in a single experience through an integrated object. By back projecting the video onto a network tile whose nodes are physical buttons that play the various videos, I hoped to make a direct connection between network nodes and network stories. I set out to make a completely new object since my feeling was that adapting, transforming or otherwise changing the original network representation created by a research participant would be ethically unacceptable. My research centres around the use of design as mediative – that is design is used to elicit responses from various interested people – the end result is all their work, often intimate and personal, usually revealing of personal relationships at varying degrees of fidelity. Even though the network tiles were returned to me for safe keeping and analysis, research ethics would imply participant responses should be treated as original works in themselves, my role in configuring the materials into a relevant form notwithstanding. One way through this dilemma would be either to ask for further consent to intervene in people’s completed representations or to involve them in the making of the final piece.

Creating and presenting representations of what participants do in co-creation settings is seen as ethically problematic. James Clifford makes the point that once participants start to be seen as co-authors with researcher as ‘interpreting observer’ a hierarchically organised representation must be the result. Gibson-Graham says design representations are have a performative function – they allow the researcher to perform both the role of author and the role of expert creator and analyst. The ethical requirement is that the design researcher takes account of these textured relations. The politics of representation, so well covered in art and design schools in the undergraduate lessons on semiotics, come into play here. What set of social relations are embodied by the representations we construct? How accurately do they reflect the situations of their original making? What new set of institutional or artefactual relations are set in motion by the representation? Simonsen and Robertson suggest it is necessary for the design researcher working with people to take account of all these factors when making representations of what participants have done. I would of course agree with that but not to the extent of precluding the making of expressive representations at all. Rogers has written about the chokehold ethical concerns imposed from different areas (eg medicine) can have on design research, with designers avoiding investigation into some areas i.e. assistive technology, due to rigid ethical requirements devised in a pre-technological era.

One weakness of the ethical arguments in the participatory design literature is that they all assume a similar dynamic. The doing of participatory design is considered to be aligned towards work and to be oriented towards the development of technologies that improve working life in some way. Since what I am doing is examining the role design in the externalisation of digital experiences, much of the literature seems misdirected. For example there are clear ethical concerns about inclusivity when designing say, an app for children in a hospital. This focus on goal-directed outcomes is perhaps a reflection of the roots of participatory design in the union-driven movement of 1970s scandinavia. Problematic methods shown by Star and Strauss in this context include the creation of non existent characters such as personas, decontextualising the representations by exploring scenarios, and abstracting indicators. The problem here is how to generalise towards a design that might fulfill wider needs than simply those captured from participants. My research is not about providing a design appropriate to the carrying out of a particular task, nor is it confined to workplace requirements. Instead, my focus is the development of a series of tools useful for thinking about digital experience. To that end the research is deeply moral and political in that its purpose is to provide an experiential window into otherwise inaccessible or opaque knowledge. I am doing this not by explaining, describing, or recounting but by designing a set of circumstances – institutional, environmental, and material – conducive to reflection targeted in a certain direction.

A final ethical concern related to working with participant data is how it is analysed. It is morally acceptable to impose a set of criteria and categories on the data that may not be there? Researchers have to reach some kind of theoretically or statistically generalisable conclusions, particularly if they wish to achieve a PhD! This involves normalising the contingent behaviour apparent in participant responses. It’s clear that the resulting representations can have the same effect. If the design I create flattens clearly expressed priorities, or overlooks others in pursuit of clarity or creativity then the result is a misrepresentation. Star and Bowker address this problem with the idea of the ‘residual category’, meaning everything left over when the data has been organised. ‘None of the above’ or ‘not otherwise specified’ may be the places where the richest description is found. Designers should, they suggest, examine this material more carefully.

As I continue to develop what will be a series of designs; some representative constructions of the research process, and others more speculative, these ethical concerns will inform my notions of reflective practice and my own role in shaping the research.